RAID Drive Replacement

**NexDog** · 08-05-2009, 09:29 PM

Quote:

20090805200803 - Controller 0
ERROR - (0x0F:0x0002): Unit degraded: Unit #0

The failed drive in the array will be replaced at 5am tonight (August 6th) US EST. Please allow 30 minutes for this maintenance window.

**NexDog** · 08-06-2009, 06:33 AM

Not good news unfortunately. After the drive swap, the raid card did not recognize any drive. We swapped the raid card and same issue occured. No hard drives detected.

After putting the original drive back in, it shows only one hard drive. As we attempted to boot the server, a loud clicking sound began echoing from the hard drive. The server no longer boots and it will require an OS reload.

We are still seeing what can be done but if data is unsalvagable we'll need to do a full restore from the backup server which does take time.

I'll keep you all updated.

mere25 · 08-06-2009, 09:05 AM

how much longer to fix it? i have so many clients complaining about their sites that they have been down for over 6 hours or maybe even more

mere25 · 08-06-2009, 09:06 AM

didn't you have the same problem on this server last year when it was down for 3 days?

**NexDog** · 08-06-2009, 09:50 AM

Hard to say. Restore is in progress. Probably looking at least another 6 hours.

Hang in there....

aztc · 08-06-2009, 09:58 AM

Enough with THETA! I should maybe just unplug my phone and stop checking email today.

aztc · 08-06-2009, 10:01 AM

I think I saw a Help Wated sign at the gas station the other day, guess I'll use this extra time to apply....

aztc · 08-06-2009, 11:32 AM

Ok, I have calmed down -

I know from past experiences with HN that these folks will bend over backward to restore Theta and they have always come through. It is really no big deal to be down for a few hours, I see even twitter is offline, so good luck guys, I still trust you and will simply relax and take care of other business for a few hours. No worries, HN is on it!

aztc · 08-06-2009, 11:39 AM

Twitter DOS attack: Denial of Service Attack Knocks Twitter Offline (Updated) | Epicenter | Wired.com

**NexDog** · 08-06-2009, 12:59 PM

See, crap happens to us all.

This is taking alot longer than previously estimated. The data center hasn't been on the ball as much as they should and only now have we hooked Theta into the private backup network we have there. So still looking at 6 hours from now. It's 2am for me here so I need to crash but techs can give you updates via tickets. My apologies again for this prolonged downtime.

All the data is there. I've been very careful to have our admins continually check backup data integrity as the software can often error out. But they've always been on top of it so I'm confident a decent restore can be done. My eyes are closing but hopefully when they open again in 6 hours it will be all good news.

All you guys on Theta are super-old HN clients. You know the drill and I'm sure you're all experts at handling your client base. Just give them all updates and don't disappear and we'll all breeze through this.