Alaska Department of Revenue vaporizes $38 billion account
By Darren Murph 
posted Mar 22nd 2007 5:21AM
posted Mar 22nd 2007 5:21AM
The Last Frontier is no stranger to computer failure on a noteworthy scale, but the latest mishap far exceeds the severity of yet another e-voting failure. A quick-fingered technician at the Alaska Department of Revenue reformatted a hard drive while handling "routine maintenance work" that contained an account worth $38 billion -- yeah, with a B. To make matters exponentially worse, he / she also deleted the backup drive for reasons unbeknownst to mere men, and we can probably assume that at least a few individuals in the department suffered a near-heart attack when they found the backup tapes completely unreadable. The only remaining proof of the oil-funded account was in 300 boxes of paperwork, which had to be digitized yet again by staff members working incredibly long hours completing work that had just been done a few months earlier. Incredibly, no one was reportedly punished for the incident, and while the recovery efforts were actually finished in just six painstaking weeks, the damage inflicted by a few careless keystrokes totaled $220,700 in excess labor costs. Ouch.[Via Fark]
Am I the only one who has ever heard of On-Track... A simple reformat is no sweat to recover from.
yeah right, an accident... he accidentally deleted the hard drive...oh and then he accidentally deleted the back up.. well, i guess its easier than destroying whole buildings ..
If they held accounts details for that amount of money on one disk and one backup tape, they deserve to lose it.
Well, probably it's not just the labor cost. Assuming one can claim to have to pay later due to the admin latency, interest over such an amount can be reward enough for the payer to fund some IT staff to care a bit less than usual. And IT staff are probably never instructed to care about the amounts on tape/disk or they could get too nervous in the first place..! Security by obscurity helps (a tiny) little here; what if the techies would have known they have such huge sums at hand to tinker with ..?
Six weeks interest for 38 bil is 142500000$
...holy crap I'm poor...
PJK @ Mar 22nd 2007 6:10AM
If they held accounts details for that amount of money on one disk and one backup tape, they deserve to lose it.
EXACTLY! Such fools! My lil 10 year old cousin Jasmine knows how to make multiple copies of her work, why can't they? SAD!!
"Da da da da DAMN!!" - Fabolous
Well, "barrowing" 38 billion for even 5 weeks to play a little on wall street can be enough for a few to retire on.
I wonder whose account it was. I wonder if that oil money has something to do with Dick or his buddies.
No, it's Bush's fault.
SOMEONE is about to lose to his/her job...
if anyone actually believes this was an accident they are very very gullible.
Uh, did anyone actually READ the article? It wasn't the account information, it was just applicant information for the yearly dividend payout. The actual fund itself wasn't impacted, except for the costs associated with recovering the data.
*scratches head*
Hmm... Where did I put that 38 billion dollar hard drive? I swear it was right here. Oh well, I'll just use the backup. Wait, where's the backup? Hey Bill! I thought that 38 billion dollar backup was on here but I can't find it.
Bill: Oh. Whoopsy-daisy. Guess I erased it. I also erased this hard drive. Anything else you want me to erase, boss? I seem to be pretty good at it.
-Yeah...I don't quite believe any person or buisiness could be that stupid or careless with 38 billion. Did that 220 grand to fix it come out of the 38 billion account? Sounds fishy. And No One got fired?! People where I work get fired for only being 99.999% accurate instead of 100% accurate with our numbers. I think someone needed work and -Whoops!- a hard drive gets erased and the friggin backup and look here! We've got 220 grand of work to do. I've seen it done consistently in construction and other contracting jobs.
Is someone having a case of the Mondays? LOL
"I always mess up some mundane detail..."
Thanks for outlining that VedicHymn; so that proves yet again that Engadget is just tweaking facts as they see fit in order to get some hits.
And of course no one tried to run some data recovery software or just plunk down a few k for a company to do professional data recovery. It was obviously so much easier to just re-enter all 300 boxes of paperwork.
Why does everyone think there was $38B on the disk? There wasn't! The money stays at the bank people...not on the disk. BTW, the bank wouldn't have made this kind of mistake.
The information lost, was the names and accounts of where the dividend money goes when it gets paid out quarterly or yearly. The only money that was lost, was the money to pay the data entry people.
When governmental agencies say erase, they mean it. Erase, over-write and randomize. There was no "going back." Besides they had the paperwork and it's probably cheaper to pay people to re-enter it than go through the futility of trying to recover it.
The cynic in me says someone this admin knows is getting rich after the data is re-entered. "Sis, it looks like you hit it big this year."
Actually, Microsoft as well as a couple other data recovery companies were hired to try to recover the data, but obviously, they were unable to save any of the data. Because they had no other choice, they then reentered all the information. Now they have a real data backup system, with full recovery software if they should play stupid again. And this is very true ,but they did not punish any employees for the incident, in fact, because everyone felt so bad about what happened,they let it go without anyone being fired or suspended or anything. Hehehe.
there's a standing rule here: if you want to keep something, don't leave it on your workstation HDD.
no one ever just erases a RAID5 array, unless it's a decommissioned server, and if they didn't migrate and verify before decommissioning, well, you get what you get.
Why is everybody demanding this poor tech's head, or assuming that he is in some way a criminal? Ye gods, have none of you heartless, cruel people ever lost important data? ;)
And at any rate, firing the tech - or holding him or her responsible in some way - would not solve the management problem that resulted in a crappy backup system. Ever heard of a no-blame corporate culture? It makes a lot of sense. If people aren't constantly in terror of getting fired, they are much more likely to own up to and fix their mistakes swiftly.
Exactly! It's not that some employee was stupid, it's that the agency doesn't manage their information process so that it is impossible for this type of mistake to occur. This is the essence of management. Make sure that what needs to happen, happens every time, and make sure that what shouldn't happen, cannot happen.
"no one ever just erases a RAID5 array"
Uh, are you serious? Even on a SAN it's easy to wipe out the data. I've had to redo cross-site data mirrors and migrated terabytes of data between two SAN fiber attached servers, trust me, it's very easy to wipe out the data (alto this situation was preventative maintenance not an "accident"). The moral of this story is not to let idiots administer critical data.
I don't think they only had it on one HD and one tape. The way I read it, they had the info copied to two HDs (which both got wiped for some reason) and the tape backup was corrupt (this, unfortunately, is not at all uncommon, tape backup sucks).
I am surprised that they were not able to recover the data. The only time I haven't been able to recover data off of a reformatted drive with a recovery program is when I do DOD wipe (I was testing the DOD wipe, it works). Very fishy this story, I wouldn't have been surprised if the hard documents had also spontaneously combusted.
Hahah.
# mkfs -F ufs /dev/rdsk/c0d0s3
Crap!!! I meant to type c1!!!
I'm a recipient of the yearly dividends - the Alaska Permanent Fund dividend. Although in this case 'permanent' didn't stick :) - I'm also a state employee. Yes, it is money from oil. Our 'savings' account hold over $38 Billion dollars. This account generates monstrous amounts of interest. It is this interest that, every year, is portioned up and a big slice of that interest gets divided between the 600,000 people that live here. The dividend is generally over a grand but has been as low as $800 and as high as around $2000. Take a household with both parents and two kids - you could be getting an $8000 bonus in October just for being Alaskan. Cheers! I look at it as state sponsored skiing. Get yourself a season pass, a new board/boot package and have money left over to party.
This is Alaska we are talking about here folks. The land of fake-it-til-you-make-it. I've experienced way too many I.T. and Data Processing idiots in charge. Then it gets worse, idiots tend to hire idiots so that they don't look bad. The rest of the folks blindly trust the decisions made by people who I wouldn't trust choosing my lunchtime sandwich. There are so many one-off solutions, proprietary backends, abandoned but still functioning services, lightning strikes that take out entire State of Alaska communications, etc. that it would behoove a major company who knows how to handle tens of thousands of employees, security, computers, phones, and so on to come up here and take the reigns. The poor horses are getting whipped from too many directions for this coach to ride smooth.
Can you say, "Fired" with a capital "F"?
If this dummy doesn't lose his job due to some miracle, then the ADoR aren't doing their job.
Try this sometime...take a basic RAID5 array. Don't have a hot spare - anywhere. Have one of the drives fail. Answer whatever question pops up during restart from WindowsNT about "Fixing" the problem in the affirmative. Watch as the operating system eats your critical data for lunch.
OnTrack couldn't touch it - they tried.
The business would have been more fortunate had they had a fire. Really.
The budget for computer equipment was increased shortly afterward...