RIM chalks up blackout to "insufficient" testing

It's probably of little consolation to the addicts among us who spent several terrifying hours in connectivity withdrawal -- and even less consolation, still, to the newly liberated -- but RIM's apparently figured out how its notoriously reliable back end came to a crashing halt this week. The company is now pointing its finger at "the introduction of a new, non-critical system routine" to its caching mechanism as the culprit. "Non-critical," indeed. Anyway, it seems said system routine was not put through enough testing ahead of its deployment to RIM's production systems -- and to make matters worse, its failover setup (hamsters on wheels, perhaps?) didn't pull through, significantly delaying the amount of time to get everything back online. We suspect most users are still a little too shocked at the chain of events to be steamed (yet), so consider this your strike one and two, RIM; just make sure it never happens again.

[Louv @ Apr 20th 2007 2:03PM]

Bugs are never *caused* by testing (or a lack of it).

[oGMo @ Apr 20th 2007 2:22PM]

No, but an outage *is* caused by lack of testing.

That said, I've got a BB, use it pretty heavily, and I didn't even notice there was an outage. :)

[The Big Fudge @ Apr 20th 2007 2:32PM]

I would think the Test First crowd is cackling with glee right now.

[techguy @ Apr 20th 2007 3:32PM]

not sure about the "test first crowd" but I know us smartphone people were pretty happy seeing all the crackberries not working, while our smartphones continued to fulfill our addiction.

[jodybrown @ Apr 20th 2007 3:38PM]

uh, rim didnt put in the new system yet...

[Tom @ Apr 20th 2007 5:31PM]

i think we should all cut them some slack: how many emails go through their system, and this is the first big issue in as far as i can remember?

[Tagbert @ Apr 20th 2007 9:34PM]

note that many companies that use blackberries have their own email servers for blackberry traffic and so were not affected by RIM's outage. It probably only affected those use use RIM's own email systems.

[techguy @ Apr 20th 2007 10:19PM]

Actually your wrong about the companies having their own servers not being affected. Yes many, probably most companies do have their own BES servers, but those servers depend on a connection to RIMs servers. Having your own BES does not eliminate RIM from the picture, and thus is an added point of failure.

Smartphones on the other hand depend on the cellular provider connection to phone and internet, and the businesses internet connection and exchange servers.

Blackberries depend on the cellular provider connection to phone and internet and RIM as well as businesses internet connection, BES Server and email server.

Much more can go wrong, but on the flip side blackberry did beat MS to the punch, many companies have already heavily invested into the devices and such and for what it does it works great no doubt and offered many features not available until only recently. The best feature in my opinion that blackberries offer over exchange and smartphones is the built in VPN tunnel to the corporate side of things if you have your own BES server. Closest thing for smartphones is third party client like Netmotion XE.

[Flagheimer @ Apr 23rd 2007 6:55PM]

LOL, I was asleep throughout this whole ordeal... Didn't even know there was a problem until I read this site.