File System Check Monday 23rd March 3am EST

**NexDog** · 03-22-2009, 07:01 AM

Muon is throwing some file system errors that is preventing our system from completing valid backups to the remote backup server. Therefore I am scheduling a file system check (fsck) for tomorrow in the early hours of the morning at 3am. This will unfortunately require 1-3 hours downtime. Hopefully the fsck will repair the errors and Muon will be business as usual.

**NexDog** · 03-23-2009, 02:15 AM

Server coming down now.

**NexDog** · 03-23-2009, 06:37 AM

All done. The fsck did repair some inode errors so good stuff.

**NexDog** · 03-24-2009, 02:37 AM

Muon still having issues after the fsck yesterday so we need to do some hardware checks and replacements. This has to be done now to ensure things don't escalate badly on this server.

Sorry for the lack of notice but this cannot be delayed!

**NexDog** · 03-24-2009, 04:13 AM

RAID card has been replaced. Hopefully this was the underlying cause and will fix the issues.

**NexDog** · 03-25-2009, 02:49 AM

Every day Muon's /home partition is going into read-only mode due to problems with the file system. We are doing another fsck on it now but it is looking increasingly like we will need to do a hard drive replacement for this server. So sites are down while this current fsck is taking place. My apologies for the inconvenience.

**NexDog** · 03-26-2009, 02:18 AM

Bad news I'm afraid. The primary drive has partitions in read-only mode yet again. We are going to try one more thing and if it isn't fixed tomorrow we will be doing a primary drive replacement and data restore that will take around 4 hours downtime so prepare yourselves for that just in case.

skulbite · 03-30-2009, 08:44 PM

Hi Lawrence,

How's Muon now?

**NexDog** · 03-30-2009, 09:07 PM

I need to check with Matei. He didn't say anything so I assume it's okay.

WebDevBB · 04-07-2009, 08:49 PM

Anything new?