Migrating Emails from one cPanel account to another

My previous cPanel migrations followed the procedure of recreating email accounts. This is going to be onerous with many accounts or impossible if I do not know the passwords for each account. In this post I will outline what I did to migrate the mail accounts without creating them manually.

In short, to migrating email accounts, I copied over the following folders:

  • /etc/ – this contains the account details and passwords
  • /mail/ – this contains the mailbox and emails

Thereafter I deal with any hardcoded paths. I found that these only exist in two places:

  • /etc/@pwcache – all password cache files
  • /etc/passwd

This means I need to do a search and replace to fix those paths.

If Autoresponders and Forwarders are involved, these will need to be dealt with copied over as well. I’ll outline the steps below.

Overview

Now here is the overview of the steps – followed by details

  1. Audit what needs to be moved
  2. Testing
  3. Prepare DNS records for switchover
  4. Copy over /etc/ and /mail/ and fix paths
  5. DNS changes at the old host
  6. Name Server changes at the registrar
  7. Copy over any mail that went to old server during the transition

Seamless transition?

The goal is so that all users do not notice anything and continue to work. In my situation, I have the following:

Protocol: All users use POP to retrieve mail. I think IMAP users should transition seamlessly as well but have not tested it.

Server names: Mail clients are configured to use mail.mydomain.com, where mydomain is the domain I am moving over. When the domain’s address changes, the mail client will retrieve mail from the new server with no changes. If the email client is configured to retrieve mail from a server name assigned by the service provider, such as when connecting through SSL, then this has to be changed on the clients end after the move.

SSL – Currently my users use unencrypted POP/IMAP access. If SSL or TLS is turned on, there may popup certificate warnings, and confuse users etc.. Most cPanel setups have SSL configured for the server-assigned domain name. Also I can only set up SSL certificates on the new server after the move so certificate changes are certain.

In my testing with the above conditions met, the most I should need to do is tell the users to restart the email client or their computer, at worst, flushing their DNS cache might be needed.

Other Precautions

SPF Records: During the transition, to prevent rejection of mail, I make a small addition to the SPF records. In the old server’s SPF record, I add the new server’s IP address, and in the new server’s SPF record, I add the old server’s IP address.

Email Routing: On the old server, I set the Email routing to be ‘local mail exchange’. Default is Auto, which might cause it to reject mail if it detects that some other server is supposed to be the real server. In case mail gets delivered here during the transition and the mail server sees that DNS has changed and now thinks it is not allowed to receive mail here. Maybe this isn’t necessary. Just being overly cautious.

Pre-switchover deliveries?: Can mail get delivered to the new server before DNS switchover? Likely not, but if a sender happens to already be using the new server we are migrating to, and is using that as an SMTP server, it might get delivered there. We might want to set up a catch all account to cover this. In any case, we can track if this has happened with the Track Delivery app in cPanel.

A note about File Manager

One caveat – when I was zipping the etc folder using cPanel’s file manager, a test email account did not get recreated, I realized that the new zip file did not contain one of the password cache files for a test account. Two things happened. One, for the password cache file to be created, the account needed to be accessed or mail to be sent. I did that with Horde (or any email client). Now that the password cache is in place, I zipped the folder and uploaded it. No Dice. The file still did not exist. I realized that I needed to refresh the cPanel File Manager for it to detect new files that were added before re-zipping the file. Files can be missing from if you do not refresh file manager before zipping a folder!

Steps

Here are steps in greater detail:

Audit what needs to be moved

  • Mail Accounts
  • Email Forwarders
  • Autoresponders
  • Catch-all email setup
  • Domain Aliases that forward mail from one to the other
  • DNS records – confirm that the email accounts are indeed hosted in the cpanel account, not somewhere else.
  • SPF records, Server names and MX records that need to be replicated on the new server
  • What TTL are the DNS records set at. Numbers like 14400 seconds(4 hours) or  86400 (24 hours) are common.

Lower TTL values in DNS records

For all DNS records, I changed the TTL to a low number. I use 300 seconds (5 min) or 600 seconds (10 min).

Some people just switch the name servers at the registry and wait.  This takes longer. Another problem is that the NS records are not available for editing so the TTL remains at 1 day or longer. So I do the server switch at the DNS zone first, by changing the A and MX records. So it is easier to switch back if there was a problem.

I do the Name Server change after the A and MX records are already pointed to the new server and have already ensured all is well.

The Big Move: Copy over /etc/ and /mail/ and fix paths

Here is the most time consuming part, and the part that needs to be done as quickly as possible:

Mail Accounts and Mail folders

Use cPanel’s Backup to FTP the entire setup to the new server. In my case this takes 10-20 min for a 3 GB cPanel backup file.

Unzip the etc and mail folders and place them in the correct paths

When one changes hosts, the username usually changes. Download all the @pwdcache files and Passwd file and change the paths and reupload.

Autoresponders

If you want to preserve autoresponders, there is a folder called .autorespond in the root folder. extract this and place in the home directory.

Use an FTP program as the cPanel file manager does not show hidden dot folders.

Forwarders

To preserver forwarders: Use cPanel’s Backup panel to download the forwarders as a .gz file. There are paths that need to be fixed here as well. Rezip with the exact file name use cPanel to restore the forwarders. The reason we do this is that these forwarders place files above our home directory where we cannot access them as a shared account user. Do the same if there are Email Filters.

Testing

Here are some tests I did to validate that my procedure worked:

Testing Creation of Email Accounts:

After copying over the etc and mail folders, going to Mail Accounts to see if the mail account appears

Testing Mail Delivery after account creation, before DNS change:

To test mail delivery, pick one of the accounts that was created on the new server and use “Access Webmail” to send an email to itself or to another account.

Testing Mail Client Switchover

Host File Method: Configure mail client to connect to a test account on the old server. After copying the account to the new server, use the above webmail method to send mail to the new account.

Modify the computer’s host file to simulate a DNS switch over.[link to procedure].

Do an NS lookup to check that your computer is indeed looking at the new server. Use mail client to connect to the new server and download mail. See if it works.

Testing Copying of New Mail

Copy over mail files into the folders named ‘new’ – and see if the test account sees the mail that you manually moved over.

Perform DNS changes – MX records

With all the above done, I am ready to switchover.

New Server: Look at the DNS records and note the IP address.

Old Server: The only change I needed to do is change the A record of the domain so it points to the new server’s IP. The MX record already points to the domain so no changes need to be done there. In 10 minutes all mail will be going to the new server.

Now that it is switched over, I send a barrage of test emails to see where they go….

Name Server Changes

Once this is confirmed working and dust settled, I’ll do the name server change at leisure, which has a longer TTL of 86400 seconds (1 day).

Copy over any mail that went to old server during the transition

After the DNS switch, I monitor Mail Delivery using cPanel’s Track Delivery app to show all incoming mail. Turn on Show Successes. I do it on both servers.

Any mail that gets delivered to the old server will be logged here. I can then go into the users mail folder and download them to move over to the new server, placing them in the ‘new’ folders of the respective owners.

Cleanup

Once no more mail goes to the old server and any mail has been transferred, I’ll request the host to delete the old cPanel account to prevent any problems with duplicate setups.

Disclaimer

Test and test and test. Any procedure here may be dependent on your exact setup and version of cPanel so may not work, this is only a reference for ideas. This currently works in my version of cPanel (58-68). Let me know if you have other ideas to add to this.