According to reports, a
single computer crash on Saturday in the Customs office of LAX caused hours of delays for more than 17,000 airline passengers. US Customs officials say that a malfunctioning network card on a single desktop created a "domino" effect with its other computers, leading to a total system failure that caused massive wait times. According to a Customs spokesman, "We lost access to our national systems, as well as our local area network." He went on the claim that it took over ten hours to diagnose the problem, halting screening operations and leaving passengers stranded on planes or in the airport -- unable to enter or leave the US. From the sounds of it, Customs need to hire a handful of Engadget readers, who we're
pretty sure could have located the source of the problem in considerably less time.
[Thanks, Darleene]
posted Aug 16th 2007 10:29AM
damn they should have waited for Vista SP1
thats what you get when you mess with SkyNet
LAX got pwned by some 1337 hax0rs
I just saw a commercial on SciFi this morning in which the dude who played the guy who created SkyNet issued a PSA advising to always read the manual and follow safety precautions when building a robot.
Maybe it's been on for a bit...I don't watch much T.V.
Which moron daisy chained them in the first place... Freakin' ghetto airport.
10 Hours!!! Hey, whats that I smell....hmmm, yep its BS.
someone send them a link to newegg.com
I'm glad I wasn't one of the THOUSANDS of people who were stuck there. Ten hours to diagnose? You gotta be kidding me. They need to be on top of their game.
The sad part is, they ARE on top of their game. What we really need is new players.
They should of called in Geek Squad.
"One customers officer said, 'Yea, we actually had to just reboot the system. However, we needed to make a call to the manufacturer of the system, as well as the government and department of homeland security to ensure we had the proper location of the switch. In addition, we needed to pay a $20,000 consultation fee to consider how exactly to press the 'reset' button. After much deliberation using our new, more efficient procedures, we finally succeeded. In addition, this has helped prevent the spread of terror."
"Customs need to hire a handful of Engadget readers, who we're pretty sure could have located the source of the problem in considerably less time."
Is that an admission that the majority of Engadget readers are probably as dumb as those at LAX?
Couldn't say I am surprised.
Huh? You have it the other way 'round.
See 'handful'
They mean to take a few of any Engadget readers, not that a handful know when a network card bursts into flames :)
Ironically, you're doing a good job of proving your own point by example.
ummm ... not to say that there aren't frequently mitigating circumstances, but if that's really what happened then 10 hours is borderline criminally negligent.
"A domino effect"??? LAX really needs to upgrade that Token Ring!
LAX, skip the cat5/6/7 etc... just go stright for the Fiber!
Domino effect huh... something sounds sorta fishy.
Agreed, most of us probably could have solved that issue within an hour. Glad I was nowhere near that mess yesterday.
As someone who used to work in the IT part of Customs, I can't say I am surprised by this. This is what happens when politics become more important than accountability and working collaboratively towards success criteria.
They should use an IBM AS400 instead of a piece of sh1t windows box
We don't know what OS the system was running, and it hardly matters when hardware fails.
Actually I think that they still use the Wang VS750 running COBOL 85.
We use AS400 and its NEVER failed. *touch wood* We've been using it since 1990... new cheap mass produced hardware from hp fails more often than our AS400.
Word up Yoshi!
is that a windows 98 BSOD :O
Yes, it is a pre NT OS kernel error.
No redundancy? 10 hours to identify the problem? Apparently no one scratched down a disaster recovery plan.
Too busy scratching their head with one hand, and thumb securely placed in the bum with the other hand. Their hands were tied man, they probably called GeekSquad which promptly installed a USBkey and started ripping pr0n.
What about just plugging a $500 fluke network tester and finding the problem in a matter of minutes? doesn't even require much brain power, so I'm sure they could have handled it.
Well now I don't feel quite so bad that my company's Disaster Recovery plan isn't quite where it should be. ONE computer taking down the entire system? And a desktop at that! I weep for my country...
hardware failure a disaster?!?
They ought to have routine plans for that sort of thing.
The problem is as follows: underpaid government IT guy + 1 cup really bad government coffee sitting on top of a US customs PC tower + newtons law = 1 damaged nework card and a system crash. That would never make it to the media because according to King Bush and his Department of Homeland Security monopoly, they don't make mistakes.
This is definitely Bush's fault.
The transmission on my truck ate it last night. I think Bush put sand in my transmission fluid while I was at work yesterday.
Isn't this fad over yet?
My guess is that they are using old and outdated networking hardware that does not give them the kind of diagnostic abilities available from most modern switches. They may actually be using unmanaged 10 base-T hubs (or heaven forbid 10 base-2 coax) rather than switches. With most managed switches a packet storm (which I am assuming is what caused this) can quickly be identified, isolated and stopped at the port in question.
Not that it is excusable but it really isn't inconceivable, depending on the hardware configurations involved.
I once had an extremely cheap network hub that suffered a port failure. This port failure essentially created a packet storm which took down our entire building for a brief period of time. It certainly didn't take 10 hours to track down, however.
The campus uses very nice managed Cisco switches. They quickly shut off the port my hub was connected to as soon as they realized what was happening. The cheap hub has since been replaced with a managed switch.
Apparently it was a comedy of errors. When they first noticed the problem they waited for an hour assuming it would come back on its own. Apparently they were used to momentary "glitches" that cleared themselves.
Once they realized things were not coming back up they decided that the issue was with their WAN link to the US Government. Sprint is the provider of the WAN link.
They called Sprint. Sprint tested routers and lines remotely and determined that the problem was not with the WAN link. Customs called Sprint again 3 hours later and they dispatched a tech. The tech took another hour and a half to arrive and then determined two hours later that the problem was not with the Sprint lines or equipment but with the LAN.
Their "backup" system was a few laptops presumably connected via dialup. They did not start setting that up until another hour or so after Sprint had told them the problem was not with the WAN link.
Nothing like the left hand not knowning what the right hand is doing...
My guess is that they are using old and outdated networking hardware that does not give them the kind of diagnostic abilities available from most modern switches. They may actually be using unmanaged 10 base-T hubs (or heaven forbid 10 base-2 coax) rather than switches. With most managed switches a packet storm (which I am assuming is what caused this) can quickly be identified, isolated and stopped at the port in question.
Not that it is excusable but it really isn't inconceivable, depending on the hardware configurations involved.
I once had an extremely cheap network hub that suffered a port failure. This port failure essentially created a packet storm which took down our entire building for a brief period of time. It certainly didn't take 10 hours to track down, however.
The campus uses very nice managed Cisco switches. They quickly shut off the port my hub was connected to as soon as they realized what was happening. The cheap hub has since been replaced with a managed switch.
hah, I was at LAX when this happened, but I wasn't flying international so it didn't affect me.
hard to believe it ended up taking them 10 hours to fix though.
I've had this problem at work...with a few IBM 4000R servers...their network card would get "stupid", and kill everything on that network. Took months to figure it out...packet captures wouldn't show anything, switches thought everything was OK...but rebooting every switch solved the problem. Finally isolated it to one server...a few months later, it happened to another one. We then took all 20-some of those servers out!
Figures US Customs would still be using Token Ring.
For God sakes!! Where are the Terminators People!!
I don't buy this story. Yes, it took them 10 hours, and yes, a lot of people were stranded. I don't buy that one network card on a desktop caused a "domino effect". I think that they had to put out a plausible story that the "everyday joes" with a computer could scratch their heads over and say "huh - 17,000 people stranded!" - and not think about the tech side. The real problem of course is the lack of a BACKUP SYSTEM!
LOL - they must have a crappy IT consultant. I live near there and I could have solved that in no time. While it could happen on an ethernet network, it is far far more likely token ring.
From what I understand about Token Ring it would be far more likely on unswitched Ethernet rather than Token Ring...I suppose it's POSSIBLE they were using Token Ring but not likely...I haven't seen a Token Ring network in operation in YEARS!
While most companies have long moved on from TR- there are still some government and legal clients who use it under the misconception from their "IT" departments that it is more secure. A single beaconing network card or even a video card can bring down a TR network with ease. Only a layer managed switch will assist in preventing a firestorm on a ethernet network however that will occur far far less often than a beaconing TR problem - bad wiring or a bad card will cause this and it does happen often as TR networks are often far more aged than ethernet.
I'm guessing it took 10 hours just to find someone that actually still knew something about Token Ring.
So let's see... primary network - Token Ring.
Backup network - dial up modem.
Well, HELLO 1990!
Makes me think of the scene in Terminator 3 where they are in a bunker outfitted with state of the art equipment from 1965...
Hmmm, 10 hours to diagnose a problem? Sounds like that question: How many monkeys does it take to screw in a lightbulb?
Hour 1 = System crashed.
Hour 2 = Another system crashed
Hour 3 = More systems succumbed to the first *Houston! We have a problem!
Hour 4 = Talk to superiors
Hour 5 = Superiors talked to their superiors
Hour 6 = Lunch break
Hour 7 = Bathroom break
Hour 8 = Superiors talked to their subordinates
Hour 9 = ??? *everyone scratches their heads and make monkey sounds*
Hour 10 = Ah! we found the problem! *Profit!*
Total amount of taxpayer dollars wasted = Maybe a tens of thousands if not hundreds of thousands of dollars.
Total amount of hours wasted by those stuck at LAX = 10 hours
Knowing that one person with half a brain more than George Bush could have found the problem sooner = Priceless
wow! it's Bush's fault that the system they purchased over a decade ago is inadequte! people never cease to amaze me. how about it's customs fault, or even better LAX....oh wait Commiefornia nknows what's best for all of us, I forgot. In fact they are probably monitering this transm
Commiefornia?
Is that our new name across the country now? Oh well I won't fret. As much as you hate California, the U.S. would be worthless without it so pipe down. California single handedly (as a state) makes up the fifth largest economy in the world, just ahead of France.
actually you're 4th, ahead of Germany get your facts.
But revenue doesn't always equal value, Saudi is one of the richest countries in the world. Thier people are almost all poor, and they would shove our bill of rights right out the door and put you in a burka. (yes I know you think you're a male) Wealth does not make you right, and without the midwest how long do you think Cali would last on oranges?
By the way listen to Arnold and fix those levees!! when you all get washed out to see like new orleans I'm going to sue you all for polluting my ocean!! >:^ )~
It's Bush's fault the systems haven't been replaced and integrated properly. I'm guessing there's been numerous memo's asking for better systems to handle the kind of "increase" in security measures that have supposedly been made. This is just another weak link that shows how much MORE vulnerable the homeland really is. I can tell you with factual knowledge that there are out-of-date and failing networks that some of the most TOP secret of secret projects our government is carrying out have a tendency to fail constantly on some level or another. Troubleshooting is done on an almost daily basis. Sometimes a new piece of equipment does make it into the mix. At that point all hell breaks loose because the old equipment can't figure out how to talk to the new equipment. If you're lucky, the old hardware will try to chug along. If you're unlucky, it burns itself out because it's OLD!!
"U.S. would be worthless without [California]"
Wake up. Exports from CA to the rest of the country are not vital as is the case with other states that supply items such as the bulk of the countrys food. Just because revenues are high doesnt mean what is produced is needed for the country to run. We'll soon find that out when CA drops off into the ocean. I have a feeling the country will be able to go on without all the BS that CA produces.
If you are wondering how the bulk of the country views people in CA. Just watch those CA based "reality" shows. Most of the country views CA as a bunch of self involved, spoiled people offering nothing to better society except what benefits them. The further you get from CA the stronger that sentiment gets. I’m not trying to be rude, just cluing you in on reality of life outside CA.
Considering the billions Bush has spent creating the American KGB, err, Stasi, oops, I mean DHS, yes, it is his fault.
token ring strikes again!
Ummm...Last I checked Token Ring didn't use terminators...It's a RING...
Each workstation connects to a MAU (Multiple Access Unit). These MAUs are then connected to form the ring. More useless knowledge that I can't seem to get out of my brain...
Ummm...Last I checked Token Ring didn't use terminators...It's a RING...
Each workstation connects to a MAU (Multiple Access Unit). These MAUs are then connected to form the ring. More useless knowledge that I can't seem to get out of my brain...
We have a bunch of dumbasses running our national infrastructure!!!
And you thought terrorists were scary!!!
Inow have the overwhelming urge to go home and watch The Terminal.
FOR KRAKOZHIA!
This kind of stuff happens all the time because the end users at their terminals are instructed not to try and fix it if they encounter a problem. They are to wait for IT to handle it. A good example of this kind of stupidity is when I was trying to check into hotel (small days inn or motel 6 type). Well they said I couldn't check in because their computer was down. I was looking right at it. Sure enough it was 98 or 95 sitting at the blue screen. but instead of just restarting the thing. they called an outside source to come diagnose the problem. I watched as thy guy simply reset the thing. apparently this "outage" last 3 hours. Luckily I was there when the calvary arrived (some level 1-2 support guy) and didn't have to wait more than 20 minutes.
You'd be surprised.
I have experienced this exact phenomena myself. We have had a single malfunctioning NIC bring down and entire network in a cascading effect.
The problem was not the NIC producing a packet storm or anything like that. It would occasionally send out a malformed frame intermitently (like every 5 minutes or so) that would just crash everything. You know the frame is something special when it causes your network analysis tool to bluescreen (lol).
Took us ages to find it. We literally had to unplug everything and plug each one of the systems back into the network one at a time and wait inbetween systems to see if the network would stay up.
Took us hours to bring the network back up. When we found it we removed the network card and even the Cisco guys were amazed, still coulden't figure out why it was crashing everything even after we subjected it to tests. This was a network of over 400 nodes. We keep that card framed today in the office as the "Network Card o Doom".
So i would not discount the scenario or the fact that it took them that long to get it back up.
MAn i used to work for that place as a Network operator and it was a darn mess i swear! no wonder they had this BS happan to them
I was actually in the line, coming in from Mexico.
Five hours. I got there at 2.
All the natives kept complaining, so they handed out free BK burgers and fries. I think they felt bad that they didn't have an answer to "Well isn't there a backup system? Or manual processing?"
"uhhh..."
People.. listen to reason. how can a END USER computer crash the whole network? Unless that desktop was infected with some virus that was doing an attack.
It was not an end user computer that caused the network failure.. it was indeed a NETWORK SWITCH, which makes more sense. However, a 10 hr downtime just means there are too many chiefs and not enough indians.
http://cbs2.com/local/local_story_225075202.html
I can't believe one NIC took out the entire Network. Have they never heard of a switch or are they still using hubs or worse token ring. It would be trivial to identify and isolate the faulty Network card by checking the port status. Hmm, the more I think about it you guys are right something is fishy with this story. I think they were hacked or someone is covering their behind.
A friend of mine was caught in this mess. He was fortunate because there was one computer working for the 17,000 passengers. The system crashed as he was a few people behind the customs officer whose computer was still working. He made it out in 10 minutes while the other passengers all queued behind the one line. Talk about luck.
In a properly structured and segmented network this problem would have been way easier to isolate/control. Too bad the LAX guys are clueless...
Um, well that is why I have always been told to check for hardware problems before anything else.
They probably outsourced their network support to a third world country.
CALL THE IT! durrrrrrr dnt call the police for help call the IT or email bill gates for help