Network Outage

**NexDog** · 01-07-2010, 03:46 AM

Currently there is an outage at our data center. All I know is that everything in the building is down and network engineers are working to resolve the issue. Hopefully will not be too long!

**NexDog** · 01-07-2010, 06:20 AM

The outage was over shortly after my post. I have been told by the data technicians that I need to wait till later in the day for an official announcement explaining what happened. When I know more I'll just post on the front page of the support section of our site and here rather than mail again. Thanks for your attention and happy new year to you all.

**NexDog** · 01-07-2010, 09:04 AM

We just had another 30 minute outage and the network when up is generally horrible with high packet loss. Hoping network engineers can nail this one quickly.

Dfishel · 01-10-2010, 01:14 AM

Did they ever tell you what happened? As a techie I always like to know.

**NexDog** · 01-10-2010, 09:55 PM

Yep:

Quote:

This morning at approximately 9AM EST, we experienced a failure in one of our core routers which cause some network outages for approximately 30 minutes. Initially we found a faulty supervisor card in the router to be the problem, this lead us to replace the card immediately, and restore the router back onto the network. The issue then arose shortly after returning traffic back over the router, that the new supervisor card also failed. During this second failure, traffic that was passing over that router was essentially black holed and not delivering traffic properly via BGP sessions over the other routers as it should have been. In other words, failover routers were not seeing this traffic as a true failure, thus it did not shift traffic properly as it should have. We have since removed that router from our network, and will be replacing it entirely as the issue is more of an upstream problem of the supervisor card causing the crystal inside the card to fail. We apologize for any inconvenience that this outage may have caused to you, and we will update you on the replacement of the router when it will be deployed. We thank you for your patience and understanding in this matter and appreciate your continued support and patronage.