Proton Server Replacement

**NexDog** · 11-21-2009, 10:50 PM

In spite of our recent maintenance on Proton the server has continued to be unstable. So I decided to just replace the server and we built a new Core2Quad server and were due to start the replacement process today. However, two hours ago the server crashed and refuses to boot up. We spent some time on it but our experiences tell us it could take many hours to rescue a server from such a state. As the new server is already online it just makes sense to replace it now, just unfortunate it has to be done while the old server is offline.

The old server has a RAID array and the new server has a RAID array and 2 arrays won't into into the new server. So what we are having to do is mount all the data onto one drive - this process takes a few hours. Then we place the single drive with all the data into the new server and copy it across. We'll copy over website and database data first and get sites and services up. Then we'll copy over mail last but mail will still work. With this method the server should be up and running, serving sites and mail within 6-8 hours but that is only a rough estimate at this time.

At least it's Saturday night/Sunday so hopefully this disruption won't be too much of an inconvenience.

Laurence
HN Support

**NexDog** · 11-22-2009, 04:26 AM

Our first attempt to migrate the RAID array to a single drive failed with a corrupt partition that wasn't recognised by the operating system. About 90 minutes ago the process was restarted and should be finished at any moment. If it fails we will have to a bare metal restore from our backup server which could take some time. So settle in for an extended downtime period - although we will do all we can to get everything up and running soon.

**NexDog** · 11-22-2009, 09:07 AM

Second attempt was successful. The copying of files is underway. There is alot of data and many things need to be installed. We hope to have the server up in some working order within 8 hours. Many thanks for everyone's patience and understanding. You have all been great and that makes this task a lot easier for all of us here.

**NexDog** · 11-22-2009, 09:45 PM

This whole replacement process is taking longer than expected. We are meeting various obstacles and problems that have to be solved before we move on to the next step. We have some partition issues that required re-partitioning the server as the partition names on old and new server didn't match up. Some IP routing issues, CF installation issues and Windows Update issues etc. But the process is ongoing and we expect to have the server open for business by Monday morning US EST if not sooner.

**NexDog** · 11-23-2009, 05:26 AM

Data copying is still underway but only about 65% done. There appears to be some domains with many thousands of small files that are taking a long time to copy. We might bring the server up and leave these copying in the background but with all services running copying will become very slow. Whatever the case the server should be up before noon US EST but as you can see, ETAs are just best guess scenarios at a certain point in time.

**NexDog** · 11-23-2009, 09:55 AM

Everything is just about done. We are just copying database data now and after that have a few reconfiguration tools to run and then we should be good to go. We think we should have the server up in 2 hours if all goes to plan.

Thanks for your understanding and patience.

MileHigh · 11-23-2009, 04:34 PM

Can we have an update please? It's been hours since the last one, and days since it was stable.

**NexDog** · 11-23-2009, 06:56 PM

I said at the beginning I don't like deadlines and this is why.

I left last night (my time) with all the database and web data copied so I thought it was almost up and I was suprised to find it still down. Apparently some of the config folders for domains are large and coping those was taking time. I told the techs to skip the remaining ones and just bring the server up. About 50 domains will still be down but these can be brought up manually after the server is up. The config folders will have to be copied over while the server is up and those domains manually reconfigured. So when the server comes up, if you have an urgent domain that is still down please contact support to get your domain moved to the front of the queue.

Thanks for hanging in there.

**NexDog** · 11-23-2009, 10:06 PM

Right now we are running the Plesk reconfigurator tool. This tool goes through each domain and corrects all the permissions and ownerships. When this is done we can run mchk which will get mail up. We'll still be missing 50ish domains when teh server comes up as I mentioned before ut right now we are just trying to get everything up as soon as possible.

**NexDog** · 11-24-2009, 09:58 AM

The copying of the config files that were missed is ongoing. It doesn't show any progress bar so it's impossible to know how far it has progressed or what is left. We did some test manual copying of domain folders in this main folder and domains that were copied and reconfigured did come up so this is definitely the last step. I know some of you have super urgent domains that need to be up so please email the domain names to winsupport@hostnexus.com and we will put them to the front of the queue by manually copying (if not already done) and reconfiguring. Please understand that we'll get 100 requests now so we'll try and get to them as soon as possible. But whatever the case the server really does have to be up extremely soon. It is copying at 4mbps which is the equivalent of 1gb every 4 minutes and it has been copying for ages. So hang in there, definitely almost there but feel free to contact support to try and get your urgent domains up sooner.