Muon has been acting up lately and throwing up some file system errors. We planned to fsck this weekend but the box crashed and demanded a fsck (File System Check) now. So we are letting it run.
It shouldn’t be long as there isn’t a huge amount of data on the server. Will post more when I know more.
http://www.hostnexus.com/forum/muon-issues-announcements/12594-muon-file-system-check.html
UPDATE: It’s still going through the motions. Unfortunately a fsck gives no progress bar so we have no ETA.
UPDATE: Nothing much to report. The fsck is just doing its thing and there’s no way to tell when it will end.
UPDATE: We had to cancel the fsck as it was 5 hours and showed no sign of finishing. We are now deciding on whether to replace the server (which was planned anyway) or schedule another fsck at the weekend.
UPDATE 9th July 8.15am: Right now the /home partition on Muon is in a read-only state. This means that nothing can write to it which means databases and email. Only static pages will work which obviously isn’t very useful. We are attempting the fsck once more. This it’s being done whilst the server is online so we can monitor its progress. New server is on order at the data center and hopefully it will be ready in 24 hours. Meanwhile we are hoping this fsck will pass in a few hours and the server will be in a working condition until it can be replaced.
It’s frustrating, I know. But I thank you all for your patience.
UPDATE: 11am: The fsck has finished and the file system has repaired itself. Due to how corrupted the file system was it appears that half the data is also corrupt and non-recoverable. But not to worry, we have complete and recent backups. In fact a backup just finished 10 hours ago. So we are currently restoring everything from the backup server. We are restoring mail first, then we’ll do databases and after that we’ll do what’s left. Hopefully in 2 hours we’ll have everything back to normal.
I’m also getting a new Intel Core i7-920 server (4 CPU Cores with HT @ 2.66Ghz) and 6GB Memory online to replace this old and weary server which should be online within 24 hours and then we’ll do the migration. Then in a few weeks we’ll do the Plesk 9 upgrade and also a ColdFusion 9 upgrade.
UPDATE: 1pm: Mail and databases now fully restored. Just have a few static files to resync now. Home stretch!
