A technician in Alaska accidentally wiped out the electronic records ofa $39 billion fund.
According to the newsstory, during maintenance he reformatted both the main and backuphard drives. For some reason the backup tapes were unreadable.
According to the web site no one was blamed for the $39b error. Thecommissioner said: “Everybody felt very bad about it and we all learneda lesson. There was no witch hunt.”
I would have fired those responsible for backups. There is no excusetoday for losing data. The technology is widely available and cheap.
Ironically they were able to recreate the data from the ultimatebackup source: 300 cardboard boxes containing paper records. Some mayremember that England created a digital version of the Domesday Bookduring the 80s. Just a decade later the laserdiscs were obsolete andunreadable. The original 921 year-old manuscript is still in goodcondition.
Paper may be the ultimate backup media, but it sure is hard to use.Some sorts of data, like software, may never exist on the printed page.And paper is no guarantee of preservation considering how manylibraries have been destroyed by fire.
It's not just libraries. In the last couple of months threecompanies have contacted me with tales of massive fire damage. One willprobably fold. Insurance covered the equipment losses but all of theirdata is gone. The backup tapes were consumed in the inferno.
Hurricane Katrina should have taught us about keeping off-sitebackups way off-site since a single event can take out an entire city.
We engineers don't seem to be much better than Alaska's government.In 1999 the FAA lost critical software for controlling flights aroundO'Hare when a disgruntled programmer deleted all of the non-backed-upcode from his computer.
In August of 2003 the Electronic Frontier Foundation announced theirFTP site had been hacked, and they had no backups.
A 2002 survey in SD Times showed that 40% of developers don't use aversion control system. Without a VCS it's impossible to do disciplinedbackups. The VCS stores source code and other files in a centraldatabase that gets backed up daily (one hopes) by the IT folks.
The data is probably more valuable than any other asset mostcompanies own. Why do we tolerate sloppy backups?
Jack G. Ganssle is a lecturer and consultant on embeddeddevelopment issues. He conducts seminars on embedded systems and helpscompanies with their embedded challenges. Contact him at . His website is
t's good to see that you talk about data backup as a mean to recover from human error and not from disk failures. Many people use RAID systems as “replacement” for backups. RAID only offers instant recovery for HD failures, not for human errors like in your example. The ultimate solution is:
– Run RAID for no downtime running system
– Backup all your data
– Backup offsite as well
Surprisingly, this is quite affordable nowadays, even for home systems.
– Julian Requejo
The paperless office is well … a myth! Our usage of paper, since the 'computer revolution' has gone up exponentially! In fact, there is even a book with this title!
Paper is an extraordinary medium. Lightweight and flexible, it is high-resolution, supports thousands of typefaces, can present both black-and-white and color illustrations, and its high contrast makes it very easy to read.
Plus… if we didn't have paper; then how else can we make those nifty little office airplanes to throw in the shredder?
– Ken Wada
A good backup policy/strategy is to backup using an automated system to an off site facilty. A good time might be daily at the noon lunch hour when most employees are away from their PC's.
One related topic regarding backups. Having a good battery backup for your PC during power outages is important. I have one for my PC that will give me slightly over 1 hour of running time in case the power goes out. You don't want to be caught in the middle of a system backup and have your PC shut down!
Just a side note… I recall a case when I was defragging my HD and the power went out for about 15 minutes… good thing I had the battery backup to keep my PC going.
– Steve King
But then the sentence “We engineers don't seem to be much better than Alaska's government”; How many of us would you fire?
Oh, by the way, the mistake did not cost the state of Alaska 39 billion dollars; that sort of sloppiness is part of what leads to these kind of mistakes. Discipline and accuracy is valuable in backups and reporting, too.
– Ed Ezzell
Always close-the-loop by retrieving backups periodically. Don't assume an automated backup system is fool-proof.
– Martin Allen
If possible the data back up should me maintained not only at the company place, but also at a a place remote to the company site. That will ensure redundant data backup.
– Shridhar Pophali
All of the World Trade Center data was (and probably still is) backed-up regularly in an entirely different state, so I was told by someone working in the data storage industry at the time. I guess they know all about witch hunts.
– Stephen Marple
Also make sure that you test your backups periodically to verify that you can actually restore data from them.
In terms of archival storage how long can you depend on a tape? a CD-ROM?, a DVD? What media works best. Do you trust your corporate data to the “free after rebate” CD media from the local office supply?
– Tim McDonough
Ed, Jack did not write that the error cost Alaska $39B, he wrote that the records were wiped out.
Jack, engineers carry out wonderful risk analyses on everything but themselves. We are the eternal optimists!
– George Whitefish
Backup is not a project.
It's a process.
FAA issue. Possible lives at risk … someone(s) should be fired!
– Tim Flynn
Most old backup tapes are not that reliable. We are a small organisation. About one eigth of the backup tapes did not write properly. Probably one quater of the tapes did not read a year later. If our computer room burnt down we could not recover anything without extreme luck as the driver software was not available on the internet and could not be found after a year but on those backup tapes. We did not lose data fortunately.
We moved to big hard disks as backup as the tapes took 24 hours to restore and this was unacceptable to the users who were on the same floor and kept knocking on the door.
– Leigh Brown
Remember 9/11's backup restoration nightmare; turned out that clients that chose IBM for the corporate-wide computing solutions were relatively unscathed whereas others were not so fortunate.
– Matt Staben
I'm of two minds about backups (must be redundent processing and data storage on my part).
Backing up your source and documentation is an obvious step, but how many people back up their development environments? PC's get replaced, compilers and OS's get upgraded. How useful is the source if you can't get it to compile without having to port it to a new environment!
We use a VCS system for source and documentation files, but in addition I keep an old HP Kayak workstation with a removable hard drive bay just for legacy software systems. I grabbed all the old small drives that nobody wants anymore and dedicate a hard drive for each major project with all of the tools required for to build and test that project on the same hard drive.
This gives me the gold standard of backups. I know for a fact that I am always able to pick up where I left off if required to fix a bug or update an older software version. I aslo hid a couple of the old Kayak workstations in case my main machine dies (IT was going to pitch them).
Hard drives are dirt cheap and if treated right are more reliable than just about any other long term storage system available. Especially when you store them powered down in a water and fire resistant safe.
– Phil Ouellette