Proton server crashed and fails to boot. We have decided to just replace the server early. For more information please see this thread. Proton clients also notified via the server’s mail list.
- Domain Price Increase For Domain Resellers
- Epic Fraud Attempt – from Charlie Sheen!
- Merry Christmas from HostNexus
- Cloud Hosting Coming Soon!
- Please Help Us To Help Your Email
- Hurricane Irene May Hit Data Center
- Discontinuing Nameservers
- Speeding Up Your SQL Server Connections
- Domain Reseller Pricing
- CFMAIL Changes
- Laurence (aka NexDog): He’d...
- Collectonian: I wonder if the real...
- Paul Salmon: I am surprised with the...
- Rudy: Ofcourse its easy to unfollow...
- Laurence (aka NexDog): The mail...
- Rudy: Thank you for this post. But...
- Wendy: Good post. Do you ever wonder...
- Peter: Thanks for sharing this. I...
- Eric Kirk: Hurricane Irene hit as...
- Quickroute: Here’s another...
Recent Status Updates
Categories
Stay up-to-date with the goings on at HostNexus. Subscribe via RSS or Email!
The HostNexus Blog covers company announcements and system updates as well as general ramblings about Web Development, SEO, Design, Hosting and anything of interest to our clients and readers.
Jgethers
November 23rd
There is NO WAY it should take 2 DAYS to install and recover a server!!! If you were starting over there would be no reason to mess with a RAID array to get it mirrored or whatever it was you were doing! Fresh install on RAID 5 or 10, and restore the data. What about our CF DSN’s? Are those going to exist. This is not 99.9% uptime.
Laurence (aka NexDog)
November 23rd
Hi John,
I explained exactly what was happening every step of the way. The new server had RAID, the old server had RAID, the old server wouldn’t boot so the only way to attack this was to copy the data over. However, the new server (or any standard server) would not allow two RAID arrays to fit inside it. Therefore we had to migrate the RAID array to a single drive. This unfortunately took some time. Then we found the new server had a different partition structure to the old one so data paths would have been messed up. We lost more time waiting for the data center to correct this with Partition Magic tools. We’ve been working as fast as possible but many of these delays have been out of our hands.
Good news is that everything is just about done. We are just copying database data now and after that have a few reconfiguration tools to run and then we should be good to go. It’s been 36 hours by the way. 12 would have been normal but I’ve also seen it take 72 hours with complications. So not great but at least it happened over the weekend!
Thanks for your understanding and patience.
Laurence
Laurence (aka NexDog)
November 23rd
Forgot to say – CF DSNs are contained in a file called neo-query.xml so they should be intact.
MileHigh
November 23rd
Mail still down, websites still down, clients are about to walk. What is the status??? This has been an incredibly long outage, and I’m getting threats of clients walking because of “lack of urgency”!!
Laurence (aka NexDog)
November 23rd
Hmm, posted updates on the mail list every few hours and the forum. Plus a small update above in response to John.
Akhilesh
November 23rd
Can someone please tell us if we can ever expect the server to come up.
There is no update whatsoever since the last 7 hours and the time mentioned in the deadline was 12 EST.
let us know is there is any update.
Jgethers
November 23rd
If the system was on a tape backup, there would have been no reason to migrate the array. Were those drives striped? I build networks and know that it can be difficult, but I have recover FSMO roles and or setup new domains in way less time. I have about 20 clients who have no email, due to DNS being down and even more with web outages. I have given them the deadlines that you had given us and those are long gone. This month is free right….
Garry
November 23rd
At least one month free…. I have already lost one client, who pays a monthly fee to me…..
Laurence (aka NexDog)
November 23rd
Yes, of course this month is free to all clients. Just send your request to acconts@hostnexus.com and they will refund this month’s hosting bill.
Chris Sizemore
November 23rd
I have 30 clients without email now after 40+ hours. We have maintained and migrated servers many times where I previously worked including RAID arrays and database content. This is going extremely long and the responses are extremely unprofessional. Can we please get this finished ASAP? Usual total server migration including RAID and data is usually 12 or so hours and given the underlying software, should be much faster. I don’t understand except to outwardly say that either you are understaffed or without the resources necessary to do the job efficiently and completely. There are many things that could keep these failures from happening in such a complete way including mirrored server environments, separate email services and DNS services on different servers with redundancy and more.
Chris Sizemore
November 23rd
Is there any way to get mail services up right away?
Laurence (aka NexDog)
November 23rd
Chris, right now we are running the Plesk reconfigurator tool. This tool goes through each domain and corrects all the permissions and ownerships. When this is done we can run mchk which will get mail up.
Laurence (aka NexDog)
November 23rd
None of our systems are on tape backup. No-one uses tape backup anymore. That’s 90s technology. Seriously, no host uses tape. The server has 500 domains with data that needs to be copied so it simply takes time. It’s also a lot slower on Windows. I said at the beginning I don’t like deadlines and this is why.
I left last night (my time) with all the database and web data copied so I thought it was almost up and I was suprised to find it still down. Apparently some of the config folders for domains are large and coping those was taking time. I told the techs to skip the remaining ones and just bring the server up. About 50 domains will still be down but these can be brought up manually after the server is up. The config folders will have to be copied over while the server is up and those domains manually reconfigured. So when the server comes up, if you have an urgent domain that is still down please contact support to get your domain moved to the front of the queue.
Jgethers
November 23rd
Ultrium drives using USCSI320 or something on a fibre channel is quick. Here’s the thing Laurence… People count on your company to provide us with good hosting services. Proton has had issues for quite a long time. Instead of waiting for it to die, you should have had a server all ready to go that you could just move the domains before it crashed. This is seriously horrible. If these domains are not up tonight, I can guarantee you that I will lose business over it. That is not a good thing for anyone. Even having DNS is better than what has taken place over the last 3 days. I have had servers go down and I NEVER leave until I know service is restored. I am sure that you are using something like FTP backup, which is slow. I don’t mean to be a prick, but I purchase space from you so I didnt need to manage another dedicated server.
Laurence (aka NexDog)
November 23rd
John, we provide a standard service. RAID’d servers with remote backup system. I’ve explained a few times what exactly is happening and nowhere did I say anthing aout FTP backups. The RAID array was DD’d to a single drive, that drive was inserted and mounted in the new server and is being copied off that.
Also I mentioned a few times that we had a scheduled server replacement and the new server was online but the old server died 48 hours before the replacement would have finished. We were actually due to start it literally as it died.
No-one here is leaving. We’ve had 2 techs on in Windows all the time. If it could e done sooner, please believe me, it would get done sooner. We aren’t sat here twiddling our thumbs.
Akhilesh
November 24th
We understand that no one is leaving and you all are working very hard for our benefits. But I am sure that you are able to understand our problems,
We have already lost 5 customers till now and are on the verge of loosing
another 5 who are ready to move their portals out of our service, not because we are going through this problem, but because we are un-answerable to them and do not have any status updates for them.
Please do not try and tell us that you are trying as we already are aware of the same, but give us a fixed timeframe which you should have after more than 72 hours of working on the issue and with the kind of experience and commitments that a company like Host Nexus has…
Anyone hearing!!!!!!!!!!!!!!
johans
November 23rd
Any update on this? Not good as I am not getting important emails.
johans
November 24th
Well no response – this happened a few years back when the server failed and there was no backup hardware available and as I recall Nexus (Lawrence?) stated they had learnt their lesson after that – guess not.
Laurence (aka NexDog)
November 24th
Johan, my other replies were posted after yours as an update. Sorry I didn’t reply to you directly. We are trying to get a good output on the reconfigurator which is failing at the moment.
We have a backup of Proton but we aren’t even using that. But I’m not going to keep repeating myself on the procedure we are doing.
Laurence (aka NexDog)
November 24th
Akhilesh – we hear you. I know it’s no consolation but in the future please think about separating your main domain from that of your clients so if there are ever issues you are contactable. It’s for this reason we have HostNexus on a dedicated server in a completely different data center to that of our clients.
Akhilesh
November 24th
Is there anyway that this status can be posted online on this blog for everyone to see and know with an estimated time for getting the domains up and running
Thanks,
Akhilesh..
Laurence (aka NexDog)
November 24th
Well, I’m keeping everyone updated via teh server’s mail list but if your mail is on Proton I guess you aren’t getting my messages. I’ll post messages here too though.
I can’t give an ETA except to say it will be up at some point and no data is lost.
Akhilesh
November 24th
if not an expected ETA in terms of hours, can we still give an ETA in terms of number of days or something like that so that we can inform our customers about the same.
Laurence (aka NexDog)
November 24th
Okay, so the Reconfigurator simply won’t complete without finishing off the last 50 domains so that is what we are doing now. I felt very pressured to get the server online in some form and this has now caused a huge delay of 12 hours.
Best case scenario right now is 2-3 hours data copying and 2-3 hours for final configuration. Hopefully the home stretch.
Garry
November 24th
Any update? I am assuming that something went wrong again?
Jgethers
November 24th
Garry
November 24th
My apologies on last post….. I didn’t hard refresh browser and missed latest update…. I know your doing your best, but I’m getting very anxious.
Can you please post times (and time zone) with Blog posts….. because as you point out, I don’t receive your e-mails.
Laurence (aka NexDog)
November 24th
The copying of the config files that were missed is ongoing. It doesn’t show any progress bar so it’s impossible to know how far it has progressed or what is left. We did some test manual copying of domain folders in this main folder and domains that were copied and reconfigured did come up so this is definitely the last step. I know some of you have super urgent domains that need to be up so please email the domain names to winsupport@hostnexus.com and we will put them to the front of the queue by manually copying (if not already done) and reconfiguring. Please understand that we’ll get 100 requests now so we’ll try and get to them as soon as possible. But whatever the case the server really does have to be up extremely soon. It is copying at 4mbps which is the equivalent of 1gb every 4 minutes and it has been copying for ages. So hang in there, definitely almost there but feel free to contact support to try and get your urgent domains up sooner.
Andor
November 24th
It is nice that the option to receive mail works already.
But as we enter the fourth day of the server being down (the fifth day if we count in 7 days weeks…), it would be nice if there could by now be any indication on when the first websites will be up again. Or when we will be able to send mails again.
So, an update would be nice, as the last one is from five hours ago.
Garry
November 24th
update please
jgethers
November 24th
My website permissions STILL are invalid… you need to run the plesk reconfigurator again on those domains. Email is up but NO websites…
4 days down… WTF? Short week here in the US!!!
Garry
November 24th
at least you have e-mail…… what is the status? 12 hours since last update.
jgethers
November 24th
Just talked with a tech… Still sounds like there have been issues with Plesk reconfiguration. This scares me. Laurence, that new box should have RAID 5 with a hot spare at least… RAID 1 doesnt provide much fault tolerance.
Garry
November 24th
perhaps it’s time to migrate me to a new server…….. preferably one of the newer ones.
jgethers
November 24th
Proton is the newest one I am guessing.. they just built it
Garry
November 25th
clarification…… a newer one that works….
Laurence (aka NexDog)
November 25th
The copying finished and the reconfigurator ran but ended with many errors. It has reactivated some domains but not all. We are checking everything and reactivating manually whatever is left.
Andor
November 25th
Any indication how long it will take? Any chance we could get at least one site up today? :-S
Would be nice if we could keep the consecutive downtime under a week.
Steve
November 25th
Come on. Why are we not back? Can we not get mirrored on another server etc? Have two major clients who hate me. Bad enough having the ex-wife. Guys, need answers and solutions NOW.
Laurence (aka NexDog)
November 25th
I feel your pain. I’m also dealing with lots of hate right now but I totally understand where he anger is coming from.
Laurence (aka NexDog)
November 25th
We’ve done server mirrors and replacements in quarter of the time, this is not something we aren’t used to. Yet this server continues to throw new errors and new issues at us for no apparent reason.
The copying took ages, the Reconfigurator failed and domains are being redone manually. This takes a long time (10 mins per domain) which is terrible as that means at least 48 hours and that isn’t acceptable. We now have Plesk support in the server trying to figure out why the Reconfigurator isn’t working. If anyone can fix it, it is certainly them so I’m quietly hopeful that they can help fix this.
We replace servers all the time and even with all services online we can do it in less than 48 hours so this whole issue is totally not normal.
jgethers
November 25th
If the Plesk configurator is failing, set the permissions manually. Http(s)docs needs to propogate to child objects…. I hav done this many times… a ton of domains, but if you have 2 people on it we would not be down… My support ticket system is down as well… THAT IS NOT GOOD!
Andor
November 25th
A hint for those of us whose sending e-mail stopped working: if you have your SMTP port set to 26, which was supported and sometimes recommended by the old Proton, change it to the default port 25. Only port 25 works on the new Proton.
Laurence (aka NexDog)
November 25th
As of now we have over half the server’s domains up. If your domains are still down please contact support. We are still working with Plesk Support for a global fix though. The problem is coming from corrupt files in the restore process. Hopefully a global fix will be available soon, meanwhile please contact us to get you urgent domains up.
Laurence (aka NexDog)
November 25th
Most domains should be up now. There are about 50 left with certain issues and we are fixing those manually. Even Plesk support faced some challenges with helping to fix this issue.
Please check all your domains and if there is anything wrong please contact support and we’ll get them fixed very quickly.
In 8 eight years this has been without a doubt the hardest server restore I have ever seen. It shouldn’t have been this bad but sometimes things just don’t work the way they are supposed to. I hope everyone can recover from this quickly and I am sincerely sorry for this extended outage. Good news is that this is now a new server so everyone should see a nice performance boost and good stability.
Thanks again for your patience and understanding.
Andor
November 25th
Half a day ago you sent the message that more than half of the domains are up. But we are many hours later than that and I have only 10% of my domains up! I thought I was pretty generous by suggesting you could place more than half of my domains to the end of the queue, as long as you placed the important once somewhat to the front. But it seems almost all of my domains are in the end of the queue now.
Last thing I heard was a a message that I “would be updated very soon”, but that was twelve hours ago or so, and since then it seemed you all just went home. No news, and more importantly, no domains up! Come on!
Now I am starting to believe that you are actually TRYING to ruin people’s business! What in the world can I tell my clients after a week of downtime? No news from you guys, no progress on putting up the sites, what the hell is going on there?
Andor
November 26th
Hi Laurence,
Maybe you could wait saying you hope everybody could recover from this quickly when the problem is actually SOLVED?
Exactly 12 hours ago I sent a list with my 13 most important domains that need to be up. This at your recommendation on this list. But only 2 of them have been brought up since.
So right now I’m very far away from even thinking about recovering!
Laurence (aka NexDog)
November 26th
Andor – there’s only 5 Proton related tickets in the queue. Everyone needs to check everything is working and we will fix any outstanding domains on a case-by-case basis. My apologies if your domains were some of last to come up.
As what you tell your clients I suggest you do what I do and have done and that is keep everyone appraised of the situation multiple times per day via different mediums. I have been emailing on the server’s mail list, posting here in the blog and posting updates in the forum.
Laurence (aka NexDog)
November 26th
The techs say just one of your domains isn’t working currently. We are waiting on Plesk support to help us with this one.
Andor
November 26th
Laurence – you are correct, but mostly because your response comes some eight hours after my comment. Indeed just one of my domains is still down right now, and it’s not an important one.
So yeah, about now I can finally start the recovery process, mentally and physically
Garry
November 30th
Laurence,
I have tried to go through more private channels than to express this in your blog, but I haven’t received a response from your accounts e-mail since I sent something three days ago.
I find that my partial refund for this month is simply not satisfactory. In this same blog you already committed to one month free, and I only received a partial refund, never mind that you committed that refund only halfway into this latest episode.
As in my e-mail to your “accounts”, I mentioned that I lost one client, and had to provide two months free hosting to all others….. this is a major hit to me, and I find your commitment to me as a client since 2004 very lacking.
I was on the cusp of recommending your services to some people looking for hosting, and since this happened, I am steering them as far away from your services as possible.
Overall I was very disappointed in the extreme down-time….. and now with your limited efforts to offer anything substantial to your clients for what I consider to be a gross error on your part, leaves me aghast.
What can you do to solve this, because I am not very happy with HostNexus.
Laurence (aka NexDog)
November 30th
Garry,
I promised one month free and that is also in our uptime guarantee so not sure why Accounts only gave you a partial refund. They are WRONG and I’m pretty angry they messed this up. Show them this comment and make sure they refund the full month. Sorry about this.