After I don’t know how many hours of surgery/disaster recovery/uncontrollable swearing/whatever you want to call it, MOST of hackerforhire.org is back up and running. Email is still flaky, but that’s of no concern to anyone other than me. That’s the short
version of what happened … this is the long one.
On Friday of last week, Sara and I went to see X-men 3 (good movie, I highly recommend it). On our way back, we noticed that there were some pretty heavy storms … that or a ravenous 40′ ogre with dysentery and a really bad mood disorder just went trouncing through the Reading area of Cincinnati laying waste to the land. Either way, the result was the same: Power out. The whole way back, every neighborhood had it’s power out. I was relieved to see that ours was still on being as some of our friends were going to be out for almost 2 days.
When we got in, Heather said the power had flickered but hadn’t gone out. For my server, that is possibly the worst thing that can happen. Why? Because my server is the only one in the world I know of that can be half-way rebooted. Yes, you read that correct, half-way rebooted. What’s halfway rebooted you ask? Basically, the system “reboots” the drive but the processor and video card stay functional. This leads to as what can only be described as a “bad thing.” Sorry to use harsh words, but it’s a very frustrating experience.
Right … so basically my drive had a seizure and Ext3, wonderful, faithful, I-Node truncating, recovery-journal destroying, Super-block mangling, Backup-super-block randomly hiding, Backup-super-block overwriting Ext3 ground to a screeching halt. The best part about Ext3 is that the “auto-fix” … and I use that term o-so lightly … basically truncates unknown I-nodes and them moves them to /lost+found and renames them with their I-node number so you have no frickin’ idea what the file was before hand. Even better it does it with directories too! Now, I know that it’s my own damn fault for running the stupid file-system, but man … Windows seems to have it better.
Fortunately, the “good” part my design was keeping all of my data contained on another drive that doesn’t vomit on itself at the first sign of a power fluctuation; however, with Gentoo keeping a lot of it’s critical package information in one of those data directories … it makes stitching the OS back together pretty hard because it thinks that packages are or aren’t installed and with whatever options it may or may not have had … longer story short, not a fun time and someone needs to write a better DR paper for Gentoo because I’m not doing it.
Everything is backup and running now … all the critical stuff anyway (minus mailing lists) so feel free to resume your reading enjoyment at Hacker for Hire (all 2 or 5 of you:-)).
Oh yeah, and to that no talent ass clown in #gentoo … thanks for being an “expert” on disaster recovery and telling me how I could just use it the way it was and it would work. You frickin’ lying bastard!!! It doesn’t work like that! I had errors flying out the ass trying to emerge my system because it “thought” libraries were installed when they weren’t as well as portage “uninstalling” packages that weren’t even installed. You can go to the seventh sub-cellar of Hell’s out house and chew a turd you damn jerk! Next time, tell people how much of a pain it is to merge their old config/world files and then rebuild everything on the system to get things back to a stable state! Oh yeah … I hate you … you seventh sub-cellar of Hell’s out house dwelling, turd chewing jerk!