Network card crash leaves 17,000 stranded at LAX

According to reports, a single computer crash on Saturday in the Customs office of LAX caused hours of delays for more than 17,000 airline passengers. US Customs officials say that a malfunctioning network card on a single desktop created a "domino" effect with its other computers, leading to a total system failure that caused massive wait times. According to a Customs spokesman, "We lost access to our national systems, as well as our local area network." He went on the claim that it took over ten hours to diagnose the problem, halting screening operations and leaving passengers stranded on planes or in the airport -- unable to enter or leave the US. From the sounds of it, Customs need to hire a handful of Engadget readers, who we're pretty sure could have located the source of the problem in considerably less time.

[Thanks, Darleene]

[F3d3r3r @ Aug 16th 2007 10:36AM]

damn they should have waited for Vista SP1

[Manuel @ Aug 16th 2007 11:17AM]

thats what you get when you mess with SkyNet

[Tim Goldsmith @ Aug 16th 2007 11:39AM]

LAX got pwned by some 1337 hax0rs

[John @ Aug 16th 2007 12:39PM]

I just saw a commercial on SciFi this morning in which the dude who played the guy who created SkyNet issued a PSA advising to always read the manual and follow safety precautions when building a robot.

Maybe it's been on for a bit...I don't watch much T.V.

[E71 @ Aug 16th 2007 3:40PM]

Which moron daisy chained them in the first place... Freakin' ghetto airport.

[jimmywash @ Aug 16th 2007 10:37AM]

10 Hours!!! Hey, whats that I smell....hmmm, yep its BS.

[Froggy @ Aug 16th 2007 2:47PM]

someone send them a link to newegg.com

[Rob @ Aug 16th 2007 10:37AM]

I'm glad I wasn't one of the THOUSANDS of people who were stuck there. Ten hours to diagnose? You gotta be kidding me. They need to be on top of their game.

[yoshi @ Aug 16th 2007 10:41AM]

The sad part is, they ARE on top of their game. What we really need is new players.

[oshean @ Aug 16th 2007 11:39AM]

They should of called in Geek Squad.

[paul34 @ Aug 16th 2007 3:02PM]

"One customers officer said, 'Yea, we actually had to just reboot the system. However, we needed to make a call to the manufacturer of the system, as well as the government and department of homeland security to ensure we had the proper location of the switch. In addition, we needed to pay a $20,000 consultation fee to consider how exactly to press the 'reset' button. After much deliberation using our new, more efficient procedures, we finally succeeded. In addition, this has helped prevent the spread of terror."

[Jon @ Aug 16th 2007 10:42AM]

"Customs need to hire a handful of Engadget readers, who we're pretty sure could have located the source of the problem in considerably less time."

Is that an admission that the majority of Engadget readers are probably as dumb as those at LAX?

Couldn't say I am surprised.

[anthony @ Aug 16th 2007 10:45AM]

Huh? You have it the other way 'round.

[Jon @ Aug 16th 2007 11:17AM]

See 'handful'

[Robotochan @ Aug 16th 2007 11:21AM]

They mean to take a few of any Engadget readers, not that a handful know when a network card bursts into flames :)

[Joel @ Aug 16th 2007 12:02PM]

Ironically, you're doing a good job of proving your own point by example.

[mike @ Aug 16th 2007 10:46AM]

ummm ... not to say that there aren't frequently mitigating circumstances, but if that's really what happened then 10 hours is borderline criminally negligent.

[Xzavier @ Aug 16th 2007 10:47AM]

"A domino effect"??? LAX really needs to upgrade that Token Ring!

LAX, skip the cat5/6/7 etc... just go stright for the Fiber!

Domino effect huh... something sounds sorta fishy.

[tweak @ Aug 16th 2007 10:48AM]

Agreed, most of us probably could have solved that issue within an hour. Glad I was nowhere near that mess yesterday.

[Chicksta @ Aug 16th 2007 10:48AM]

As someone who used to work in the IT part of Customs, I can't say I am surprised by this. This is what happens when politics become more important than accountability and working collaboratively towards success criteria.

[Chris @ Aug 16th 2007 10:51AM]

They should use an IBM AS400 instead of a piece of sh1t windows box

[Eric @ Aug 16th 2007 10:54AM]

We don't know what OS the system was running, and it hardly matters when hardware fails.

[AbacusUser @ Aug 16th 2007 11:02AM]

Actually I think that they still use the Wang VS750 running COBOL 85.

[chris @ Aug 16th 2007 2:45PM]

We use AS400 and its NEVER failed. *touch wood* We've been using it since 1990... new cheap mass produced hardware from hp fails more often than our AS400.

[jimmywash @ Aug 16th 2007 10:52AM]

Word up Yoshi!

[Kurian @ Aug 16th 2007 10:54AM]

is that a windows 98 BSOD :O

[Mike @ Aug 16th 2007 11:01AM]

Yes, it is a pre NT OS kernel error.

[Henry @ Aug 16th 2007 10:59AM]

No redundancy? 10 hours to identify the problem? Apparently no one scratched down a disaster recovery plan.

[Major Malfunction @ Aug 16th 2007 2:07PM]

Too busy scratching their head with one hand, and thumb securely placed in the bum with the other hand. Their hands were tied man, they probably called GeekSquad which promptly installed a USBkey and started ripping pr0n.

[gigo08 @ Aug 16th 2007 10:59AM]

What about just plugging a $500 fluke network tester and finding the problem in a matter of minutes? doesn't even require much brain power, so I'm sure they could have handled it.

[JBo @ Aug 16th 2007 11:00AM]

Well now I don't feel quite so bad that my company's Disaster Recovery plan isn't quite where it should be. ONE computer taking down the entire system? And a desktop at that! I weep for my country...

[Viramor @ Aug 16th 2007 11:14AM]

hardware failure a disaster?!?

They ought to have routine plans for that sort of thing.

[Justin @ Aug 16th 2007 11:47AM]

The problem is as follows: underpaid government IT guy + 1 cup really bad government coffee sitting on top of a US customs PC tower + newtons law = 1 damaged nework card and a system crash. That would never make it to the media because according to King Bush and his Department of Homeland Security monopoly, they don't make mistakes.

[andy @ Aug 16th 2007 12:02PM]

This is definitely Bush's fault.


The transmission on my truck ate it last night. I think Bush put sand in my transmission fluid while I was at work yesterday.


Isn't this fad over yet?

[Shane @ Aug 16th 2007 11:29AM]

My guess is that they are using old and outdated networking hardware that does not give them the kind of diagnostic abilities available from most modern switches. They may actually be using unmanaged 10 base-T hubs (or heaven forbid 10 base-2 coax) rather than switches. With most managed switches a packet storm (which I am assuming is what caused this) can quickly be identified, isolated and stopped at the port in question.

Not that it is excusable but it really isn't inconceivable, depending on the hardware configurations involved.

I once had an extremely cheap network hub that suffered a port failure. This port failure essentially created a packet storm which took down our entire building for a brief period of time. It certainly didn't take 10 hours to track down, however.

The campus uses very nice managed Cisco switches. They quickly shut off the port my hub was connected to as soon as they realized what was happening. The cheap hub has since been replaced with a managed switch.

[Shane @ Aug 16th 2007 1:53PM]

Apparently it was a comedy of errors. When they first noticed the problem they waited for an hour assuming it would come back on its own. Apparently they were used to momentary "glitches" that cleared themselves.

Once they realized things were not coming back up they decided that the issue was with their WAN link to the US Government. Sprint is the provider of the WAN link.

They called Sprint. Sprint tested routers and lines remotely and determined that the problem was not with the WAN link. Customs called Sprint again 3 hours later and they dispatched a tech. The tech took another hour and a half to arrive and then determined two hours later that the problem was not with the Sprint lines or equipment but with the LAN.

Their "backup" system was a few laptops presumably connected via dialup. They did not start setting that up until another hour or so after Sprint had told them the problem was not with the WAN link.

Nothing like the left hand not knowning what the right hand is doing...

[Shane @ Aug 16th 2007 11:31AM]

My guess is that they are using old and outdated networking hardware that does not give them the kind of diagnostic abilities available from most modern switches. They may actually be using unmanaged 10 base-T hubs (or heaven forbid 10 base-2 coax) rather than switches. With most managed switches a packet storm (which I am assuming is what caused this) can quickly be identified, isolated and stopped at the port in question.

Not that it is excusable but it really isn't inconceivable, depending on the hardware configurations involved.

I once had an extremely cheap network hub that suffered a port failure. This port failure essentially created a packet storm which took down our entire building for a brief period of time. It certainly didn't take 10 hours to track down, however.

The campus uses very nice managed Cisco switches. They quickly shut off the port my hub was connected to as soon as they realized what was happening. The cheap hub has since been replaced with a managed switch.

[azn.jalapeno @ Aug 16th 2007 11:39AM]

hah, I was at LAX when this happened, but I wasn't flying international so it didn't affect me.

hard to believe it ended up taking them 10 hours to fix though.

[Brian @ Aug 16th 2007 11:40AM]

I've had this problem at work...with a few IBM 4000R servers...their network card would get "stupid", and kill everything on that network. Took months to figure it out...packet captures wouldn't show anything, switches thought everything was OK...but rebooting every switch solved the problem. Finally isolated it to one server...a few months later, it happened to another one. We then took all 20-some of those servers out!

[Boyo @ Aug 16th 2007 11:59AM]

Figures US Customs would still be using Token Ring.

For God sakes!! Where are the Terminators People!!

[Chasqui @ Aug 16th 2007 12:15PM]

I don't buy this story. Yes, it took them 10 hours, and yes, a lot of people were stranded. I don't buy that one network card on a desktop caused a "domino effect". I think that they had to put out a plausible story that the "everyday joes" with a computer could scratch their heads over and say "huh - 17,000 people stranded!" - and not think about the tech side. The real problem of course is the lack of a BACKUP SYSTEM!

[boe @ Aug 16th 2007 12:45PM]

LOL - they must have a crappy IT consultant. I live near there and I could have solved that in no time. While it could happen on an ethernet network, it is far far more likely token ring.

[Shane @ Aug 16th 2007 1:26PM]

From what I understand about Token Ring it would be far more likely on unswitched Ethernet rather than Token Ring...I suppose it's POSSIBLE they were using Token Ring but not likely...I haven't seen a Token Ring network in operation in YEARS!

[boe @ Aug 16th 2007 1:49PM]

While most companies have long moved on from TR- there are still some government and legal clients who use it under the misconception from their "IT" departments that it is more secure. A single beaconing network card or even a video card can bring down a TR network with ease. Only a layer managed switch will assist in preventing a firestorm on a ethernet network however that will occur far far less often than a beaconing TR problem - bad wiring or a bad card will cause this and it does happen often as TR networks are often far more aged than ethernet.

[rip @ Aug 16th 2007 7:51PM]

I'm guessing it took 10 hours just to find someone that actually still knew something about Token Ring.

So let's see... primary network - Token Ring.

Backup network - dial up modem.

Well, HELLO 1990!

Makes me think of the scene in Terminator 3 where they are in a bunker outfitted with state of the art equipment from 1965...

[JJ @ Aug 16th 2007 12:29PM]

Hmmm, 10 hours to diagnose a problem? Sounds like that question: How many monkeys does it take to screw in a lightbulb?

Hour 1 = System crashed.

Hour 2 = Another system crashed

Hour 3 = More systems succumbed to the first *Houston! We have a problem!

Hour 4 = Talk to superiors

Hour 5 = Superiors talked to their superiors

Hour 6 = Lunch break

Hour 7 = Bathroom break

Hour 8 = Superiors talked to their subordinates

Hour 9 = ??? *everyone scratches their heads and make monkey sounds*

Hour 10 = Ah! we found the problem! *Profit!*

Total amount of taxpayer dollars wasted = Maybe a tens of thousands if not hundreds of thousands of dollars.

Total amount of hours wasted by those stuck at LAX = 10 hours

Knowing that one person with half a brain more than George Bush could have found the problem sooner = Priceless

[mastershake3 @ Aug 16th 2007 12:54PM]

wow! it's Bush's fault that the system they purchased over a decade ago is inadequte! people never cease to amaze me. how about it's customs fault, or even better LAX....oh wait Commiefornia nknows what's best for all of us, I forgot. In fact they are probably monitering this transm

[Geoff Gibson @ Aug 16th 2007 1:01PM]

Commiefornia?

Is that our new name across the country now? Oh well I won't fret. As much as you hate California, the U.S. would be worthless without it so pipe down. California single handedly (as a state) makes up the fifth largest economy in the world, just ahead of France.

[mastershake3 @ Aug 16th 2007 1:21PM]

actually you're 4th, ahead of Germany get your facts.
But revenue doesn't always equal value, Saudi is one of the richest countries in the world. Thier people are almost all poor, and they would shove our bill of rights right out the door and put you in a burka. (yes I know you think you're a male) Wealth does not make you right, and without the midwest how long do you think Cali would last on oranges?
By the way listen to Arnold and fix those levees!! when you all get washed out to see like new orleans I'm going to sue you all for polluting my ocean!! >:^ )~

[BigD145 @ Aug 16th 2007 1:52PM]

It's Bush's fault the systems haven't been replaced and integrated properly. I'm guessing there's been numerous memo's asking for better systems to handle the kind of "increase" in security measures that have supposedly been made. This is just another weak link that shows how much MORE vulnerable the homeland really is. I can tell you with factual knowledge that there are out-of-date and failing networks that some of the most TOP secret of secret projects our government is carrying out have a tendency to fail constantly on some level or another. Troubleshooting is done on an almost daily basis. Sometimes a new piece of equipment does make it into the mix. At that point all hell breaks loose because the old equipment can't figure out how to talk to the new equipment. If you're lucky, the old hardware will try to chug along. If you're unlucky, it burns itself out because it's OLD!!