3:21AM – The migration is as good as completed, with the final two mailboxes transferring and all other services operational. 

1:51AM – I spoke too soon… make it three finished and the fourth underway.

1:46AM - Two of the seven IMAP accounts have been transferred so far and we’re onto the third. Mail services have been operating for almost two hours without any issues on the new mail server. In fact, initial (and very rough) calculations show a speed increase of almost 25x, which has me quite satisfied.

Unless you own one of the remaining five mailboxes then as far as you’re concerned everything is done. If you do own one of these five outstanding accounts then you’re mail is working fine, but a few message might be missing on the new server, until I finish this migration and it’s a mirrored copy of your old set up.

12:01AM – Then it struck on me… we don’t need to use the --delete2 flag for these mailboxes that use IMAP! I can change the DNS now and leave those seven mailboxes syncing over the next few hours.

11:03PM – The migration is going as well as planned. The migration script has moved over 250 mailboxes, without issue. We’re waiting for the final seven mailboxes to transfer, though these are the largest of all our customer mailboxes. When these last few are done we can change the DNS and let the mail flow again. We’re definitely on track for our 3:00AM deadline.

9:39PM - Our email migration has commenced. Mail delivery has been suspended and mail is spooling on our IronPort.

Outlook 2002

From the menu select Tools, E-Mail Accounts. View or change existing email-accounts. Next. Change “Server Information”, “Incoming mail server (IMAP)” or “Incoming mail server (POP3)” to mail.thefrontiergroup.net.au

Outlook 2007

From the menu select Tools, Account Settings. E-mail tab, highlight the account. Click “Change” button. Set the “Incoming mail server” to mail.thefrontiergroup.net.au

Outlook Express

From the menu select Tools, Accounts. Highlight the account. Click “Properties” button. Change “Incoming mail (POP3, IMAP or HTTP server)” to mail.thefrontiergroup.net.au

Apple Mail

From the menu select Mail, Preferences, Accounts. Select the account and change “Incoming Mail Server” to be mail.thefrontiergroup.net.au

• We’re migrating to a Zimbra server which is the same platform that Yahoo! Mail uses. It has a much nicer web interface and will allow us to scale our mail systems in the future.
• There should be no service disruption. Mail cannot be lost based on the migration plan we have. At worst it’ll be delivered to your old mailbox when you check your new mailbox, and we can rectify that.
• Your outgoing mail server (SMTP) does not change. This is very important. The Frontier Group does not send mail on your behalf.
• Your incoming mail server for all Frontier Group accounts should (already) be mail.thefrontiergroup.net.au
• hotrod.thefrontiergroup.net.au is being deprecated and any accounts that reference this server should be updated to mail.thefrontiergroup.net.au immediately.

Who does this impact?

Only customers that are collecting email directly from The Frontier Group’s mail servers will be affected. If you use a local Microsoft Exchange server or have the email for your domain forwarded from The Frontier Group to another mail server (like your internet service provider) then this will not affect you.

NOTE: This does not affect your SMTP server setting. It may only affect your incoming mail (POP3 or IMAP) setting. Do not make any changes to your SMTP settings or you will be unable to send email.

What’s going on?

Currently email comes into our network via our IronPort anti-spam and anti-virus appliance. That email is then delivered to a Frontier Group mail server or is sent offsite to a third party. In the case that your mail is delivered to a Frontier Group mail server, you may need to update the account details within your email client to accommodate the new server.

Do I need to change anything?

Possibly. Now is a good time to ensure you are collecting your email from mail.thefrontiergroup.net.au because mail services on other servers will be shut down in the coming week.

If you are collecting your mail from another server, like hotrod.thefrontiergroup.net.au adjust the server entry to be mail.thefrontiergroup.net.au.

If you are collecting mail from a third party, like a Microsoft Exchange server or your ISP then you don’t need to adjust anything.

The process

We have configured a new mail server to mirror the current configuration, which means that your domains, accounts, passwords, aliases and distribution lists have been converted and migrated. Over the coming week we will be synchronizing your mailbox to ensure that your old mail is available on the new Zimbra server.

Just before we “throw the switch” we will send out another email indicating that the change is happening immediately. This will be at midnight on a weekday.

Then we will be updating the DNS records so that mail.thefrontiergroup.net.au points to the new mail server.

Lastly we’ll send an email indicating that the change over has been completed. If you receive that email then you’ll know everything is working fine.

Because we’ve set your new account up with the same username and password, the first time you check your mail after we make the change it should be from the new mail server. You shouldn’t have to change a thing.

You may need to enter your password again. Our staff can reset your password if you have forgotten it.

Your immediate next steps

If you have an IT department or use the services of an IT consultant, forward this information to them and ask that they check you are configured correctly to handle this change. That way, when we enact these changes you won’t experience any disruptions.

Our staff are able to assist you with any queries you may have regarding this. You can contact them via (08) 9322 3055 or email support@thefrontiergroup.com.au.

I hear you ask “how is that a negative side effect?” It meant that I put off the investigation into bench marking and tuning our server platforms simply because the move to newer hardware gave us significant gains. I had planned on doing some configuration tuning and system refinement, but the sheer increase in processing power meant that I didn’t bother with it back then.

Today, however, I decided to investigate what the addition of a PHP opcode cache would provide, and I was very happy with the results.

I came across an issue with running APC, the opcode cache developed by Rasmus and other core PHP developers, alongside the Zend Optimizer. The two are incompatible so you’ll have to play favourites and pick one or the other.

Installation of APC is easy on Ubuntu. There’s many guides online, but I found this one to be the most succinct. You end up installing the following packages (note: I didn’t have build-essential initially and PECL couldn’t build the package because I was missing make):

mlambie@prime:~$  sudo aptitude install php-pear php5-dev apache2-prefork-dev build-essential

The installation of APC is easy via PECL:

mlambie@prime:~$  sudo pecl install apc

Then enable the module by creating an apc.ini file:

mlambie@prime:~$ cat /etc/php5/apache2/conf.d/apc.ini
extension = apc.so
apc.enabled = 1
apc.shm_size = 48
apc.include_once_override = 1
apc.mmap_file_mask = /tmp/apc.XXXXXX

I ran some benchmarks using Apache Bench. The results were very encouraging. I saw a reduction in the time per request fall from 91ms to 37ms (250% improvement). The volume of requests per second increased from 11/sec to 27/sec (245% improvement). Lastly, 98% of all requests are served within 639ms instead of 1165ms (182% improvement).

Each actual apache process saw a significant memory reduction too, from 26MB down to 20MB (almost a 25% footprint saving). When you’ve got multiple processes running the reclaimed memory adds up quite quickly.

I found the figures staggering, and the improvements are actual, real things that are visible when interacting with the sites and applications hosted on prime.

It’s fair to say that we more then doubled our performance, from a single server, simply by adding PHP opcode cache. If you’re not running one on your server, you might want to consider why.

echo <password> | sudo -S clear && sudo aptitude update && sudo aptitude dist-upgrade && sudo aptitude clean'"

I don’t like that my sudo password is displayed on the screen. I could get around this by manually editing /etc/sudoers to allow for password-less aptitude. Alternatively, perhaps I could encrypt my password inside the Applescript and send it, pre-encrypted, to sudo. They’re options I guess.

You’ll notice that the first thing I do is clear the screen, but when there’s a second or so lag it means my password is bare for all to see. I’ll consider that when I run the script.

Below is an Applescript snippet which shows you how to open a dialog box and take some simple text input:

set my_password to display dialog "Please enter your password:" ¬
	with title "Password" ¬
	with icon caution ¬
	default answer "" ¬
	buttons {"Cancel", "OK"} default button 2 ¬
	giving up after 295 ¬
	with hidden answer
if length of (text returned of my_password) is not 0 then
	display dialog "Running the application!" buttons ["OK"] default button 1
else
	display dialog "You didn't enter a sudo password!" buttons ["OK"] default button 1
end if

Having spent a bit of time with Ruby lately, I don’t like the syntax of Applescript very much, though it gets the job done.

I use Terminal (or is it called Terminal.app?) on Mac OS X as my terminal emulator, and I’ve messed around with using Applescript to open multiple tabs before, so I figured this was a problem worthy of a Sunday-morning.

What I envisaged was iterating over a list of servers and having Applescript manage a Terminal instance, opening new tabs where appropriate, and executing the upgrade command in each tab for each server. Sounds simple enough.

And it was. I used Martin Ström’s excellent article as a base for my script, and ended up with the following:

set tab_count to 0
set servers to {"astrotrain", "bumblebee", "grimlock", "ironhide", ¬
	"jazz", "laserbeak", "mirage", "prime", ¬
	"prowl", "ratchet", "ravage", "rumble", ¬
	"soundwave", "wheeljack"}
-- Update the package list, dist-upgrade and remove the downloaded
-- packages so they're not included in the backups
set dist_upgrade to ¬
	" 'sudo aptitude update &&
	  sudo aptitude dist-upgrade &&
	  sudo aptitude clean'"

-- Make our settings globally available
global tab_count, servers, dist_upgrade

-- Mainline
on main()
	tell application "Terminal"
		activate
		repeat with server in servers
			set cmd to "ssh -t " & server & dist_upgrade & " && exit"
			my open_tab(cmd)
		end repeat
	end tell
end main

on open_tab(cmd)
	tell application "Terminal" to activate
	my create_new_window_or_tab()
	tell application "Terminal" to ¬
		do script with command (cmd) in last tab of window 1
end open_tab

on create_new_window_or_tab()
	if tab_count ≤ 0 then
		tell application "Terminal" to do script ""
		set tab_count to tab_count + 1
	else
		tell application "System Events" to ¬
			tell process "Terminal" to ¬
				keystroke "t" using command down
	end if
end create_new_window_or_tab

-- Run the mainline
main()

Apologies for the highlighting – the plugin doesn’t understand Applescript.