After the final update was completed I booted up the VMs only to find the two Windows Servers came up no problems, and none of the Linux servers booted. None of them. They started up, black screen and white cursor in the top left hand corner and that was it. No boot no errors, just gone.
I ran through some basic repair work and had no joy. I changed the GrubConf.py from if arg.strip() == "${saved_entry}": to if arg.strip() == "${saved_entry}" or arg.strip() == "${next_entry}":. Still no luck. I re-installed Grub - no joy. I booted from the Ubuntu server CD and tried to run rescue the broken install. On several of the VM's I couldn't even get the system to run a shell in the / file system (/dev/xvda1 typically on these setups). It was incredibly frustrating and after three weeks of banging my head against this, I've still not managed to get this all going.
Luckily I had backups of most of the servers and so I haven't lost everything but I have lost some critical data. The message here, dear friends, is multiple:
- test the updates one at a time to make sure everything starts up. Yes it will take longer and it's a bit more frustrating, but believe me, you want to know if things are going to shit themselves.
- backups backups backups - are not important. RESTORES! are the important part. Believe me, I cried into my beer when I couldn't get one of these machines back up and going - it just wasn't going to happen. I lost a wiki with data that wasn't anywhere else and that's caused me no end of grief. The extra work it's caused is phenomenal and I feel very bad about it indeed.
I've reached out to the collective wisdom of SAGE-AU (www.sage-au.org.au) but sadly no luck yet. I'm open to suggestions!
No comments:
Post a Comment