[RESOLVED] Network Maintenance December 23rd 7pm US EST

We have some pretty major major network maintenance coming up on the 23rd at 7pm EST. The “too long, didn’t read” version is that we have to move all our IPs (over 8,000) to a different set of routers at the data center and everything is going to be down for maybe an hour but we hope just a few minutes.

And now for the more complex explanation of why and how. As many of you might know, HN doesn’t own a data center. We have private racks in our own section of the facility. All these racks have switches which the servers connect to. These switches are then connected to a single AGG router which is then connected to shared core routers. The AGG router is becoming extremely overloaded and we are very concerned that this could become a single point of failure. So we need to eliminate this router in the network. There is also another issue that is somwhat more complex. We do a lot of server replacements as we have so many servers and technology gets old. What we like to do is bring a new server online, transfer data and then reroute the IPs to the new server so there are no IP or nameserver changes for clients. The replacements are almost seamless and painless for clients on the server. The most ideal network layout for any data center is to bind IPs to a server and then route any extra ranges to the original IP’s gateway. But when we do server replacements we have to static route all IPs to a server. This is what is causing the current load issues and the data center really wants us off their core equipment if we want to continue doing this type of server replacement (which we do, very much).

So the solution is to dual link our switches direct to new core routers which will be ours and separate from other routers in the data center thus eliminating the troublesome AGG router. This give us failover links to failover routers and further isolates us from everything else in the data center which is a very good thing. In a sense we will be running a mini data center inside a data center which will be great as we continue to grow. So December 23rd 7pm EST 30-60 minutes downtime - all servers. The IP and traffic reroute shoud be automatic as we just move the vlans where all IPs are assigned but just in case your server is an anomaly and on an a different vlan (statistically almost impossible) please do check your server is responding. We will of course manually verify each server too but there are many so will take a few hours to get through them all. Should be a smooth ride though!