Sunday, October 8, 2006

A Scary Story (a wee bit early)

This is one scary story that can start right out with the climax - the hard drive lurking in your computer just waiting for an opportune time to die. I speak from much recent experience.

First to start to show signs of trouble was the drive in my office Mac. I had known for a long time that it had some bad sectors that we were avoiding, but in recent weeks it had stumbled onto them a few times and let me know that it was time to take that last long walk. However, repairs on my own computer tend to wait for a dull moment, and there are not many of those, so it was left to fate to decide when this drive would spin no more.

Before it had a chance to go, a drive on one of the servers I maintain decided to take the leap. Unfortunately, that server's job is to perform backups of all the other hard drives in the office, and so it not only was a bit more difficult to recover from, but when my own drive decided to keel over the very next day I had no backup handy to restore from.

At the same time as this drama was unfolding at work I was fiddling at home with an iPod whose hard drive was too tired to come out and play. Luckily the data on an iPod is just a copy of what is in iTunes, but it was still going to be a tad expensive to repair (at least I didn't have to cover the costs of the office failures).

So, my plan upon arriving at work on Monday morning was to get the backup system running and get my own drive restored. However, before I could even get to the office one of our faculty called with a sick laptop - the disease turned out to be a catastrophic hard drive failure.
Hers was especially unfortunate, because even though I could not have retrieved her data right away from my comatose backup system, her data was not even backed up on my system as her laptop tends to live its life at her home where my server can not see it. We sent her drive off to a data recovery service, we'll have to see what she thinks her data is worth. The last quote I got from this service was $1900 for the contents (20 large database files) of one hard drive (after the educational discount).

By Tuesday another hard drive in a PC at the office died. This one at least will be recovered via the resuscitated backup system in a day or so.

The week ended with another drive that was getting a bit balky, and so I quick replaced it before it had a chance to totally die on me.

Things are looking up. My backup system is back on its feet, and so I will be able to restore my own drive's contents. I was able to cajole the drive in the iPod back into service for a while, so it may yet recover, and I am making plans for folks to back up their drives that stay at home and out of reach of my automated system.

As with any good story, this one has a clear moral: Hard drives are evil and want nothing more than to take your data with when their holiday from Hell is over. If you don't have some other copy of your files you will soon not have your files at all, and the Devil charges a LOT to get them back (if he feels like dealing at all).


wunelle said...

Boy, talk about plagues and pestilence! So far (knock on wood) my whopping four drives are behaving nicely. My oldest is in the G4 Powerbook, and it's coming up on its four year anniversary. The others are within a year.

Amazing that you should have so many in such a short time. I'm assuming that your backup server was a failure of the server itself and not of a hard drive per se, since those back up to tape, no?

How old was the iPod? I use mine quite a bit in the car, but mostly it gets jostled around in my backpack. I wonder how long it will last. I guess batteries can be replaced by a professional.

Anonymous said...

I have several computers at work running 24/7, including one that is still running os 9.2 on an old machine that locks up when put to sleep, so I keep the hard drives spinning. Haven't had a failure in 8 years (In fact, I've never had a failure). I suppose it's only a matter of time. Luckily, my business is small enough to back up to DVD-RAM every evening without too much hassle. Still haven't figured out how to back up my two terabyte drive without buying another two terabyte drive, though;)

Alex Random

Jeffy said...

wunelle - The backup server did actually suffer a hard drive failure. The backups do get written to tapes, but a large catalog of the contents of the tapes is written on disk. Without the catalog you can't find any of the contents of the tapes. Luckily, the tapes can be read and the catalogs reconstructed, it just takes around 30 hours per catalog. I have two of them reconstructed, and two more to go.

Given how many hard drives I have to observe (roughly 500 computers with at least one per) it is probably not that surprising to have this many go in such a short time. If a drive will run on average for 10 years then having about 50 failures a year sounds about right. I'll have some weeks with none, and some with a couple, and a few nasty weeks with several. I just have to be sure to take notice of and appreciate those special weeks when I have none.

The iPod is about 2 years old. My anecdotal experience is that portable hard drives fare about the same as desktop drives, so I wouldn't expect the iPod to fail any quicker than any other drive. At least when it fails you are not out any irreplaceable data, just the cost of a drive (around $100 in this case).

Alex - Sounds like you've had pretty good luck so far, but keep those DVDs spinning - you are probably due. Your 2TB drive should be no problem - you can get the entire contents on just under 450 DVDs ;-)