Cellphones,
%Match-TagSlug-@hd-dvrs%
Sidekick failure rumors point fingers at outsourcing, lack of backups
By Chris Ziegler 
posted Oct 11th 2009 7:44PM
posted Oct 11th 2009 7:44PM
Backing up your personal PC to external media might still be a novel concept for some, but any IT manager fresh out of school can tell you that regularly backing up mission-critical servers -- and storing those backups in multiple physical locations -- isn't merely important, it's practically non-negotiable, and it only becomes that much more critical before undertaking hardware maintenance. Alleged details on the events leading up to Danger's doomsday scenario are starting to come out of the woodwork, and it all paints a truly embarrassing picture: Microsoft, possibly trying to compensate for lost and / or laid-off Danger employees, outsources an upgrade of its Sidekick SAN to Hitachi, which -- for reasons unknown -- fails to make a backup before starting. Long story short, the upgrade runs into complications, data is lost, and without a backup to revert to, untold thousands of Sidekick users get shafted in an epic way rarely seen in an age of well-defined, well-understood IT strategies.The coming weeks are going to be trying times for both Microsoft and T-Mobile, a sideline player in this carnage that ultimately still shoulders responsibility for taking users' cash month after month and keeping tabs on the robustness of its partners' workflows. We're betting that heads are going to roll at both of these companies, formal investigations are going to be waged, users are going to be compensated in big ways, lawsuits are going to be filed, and textbooks could very well be modified to make sure that lessons are learned for the next generation of college grads tasked with keeping clouds running. Why there weren't any backups -- even older ones -- that could've been used as a restore point is totally unclear, so we're hoping Microsoft has the stones to come clean for the benefit of an entire industry that wants to understand how to make sure this never happens again.
WHOA! I guess it is time to say BYEtachi....
I am sorry for all of your Sidekick users. I hope T-Mobile does compensated you dearly.
I am sorry for all of your English teachers. I hope you does apologized to them dearly.
I am sorry : compensate, compensate, compensate, compensate - I am sorry, English is not my first language, I speak poorly. Je suis espagnol. :(
"I hope you does apologized to them dearly"
Zorn, If you are playing the "Grammar Nazi" card, at least do it right.
@Aguiluz
I think he did it intentionally... Surprised you missed that...
He made a mistake to correct another mistake?
Wrong + Wrong = Right?
Thanks guys. I will be more careful and proofread my posts. Sometimes I just get so excited, my nerdrenaline rushes and I just type away.
Lol don't apologised. We alls make mistaks sumtimes.
MS: "I thought you made a backup?"
Hitachi: "No, I thought you made a backup?"
MS: "Oh cr@p, we're screwed."
People with a brain had figured this out already. LOL at MS fanboys trying to pin the blame on T-Mobile and Danger on the other post. It was obviously MS's fault all along. They are a mediocre company, all of their products and services are third rate and half assed.
I hope no one gets compensated. They deserve nothing for the loss of their contact list for pete sakes. They too should have had a backup
nerdrenaline is my new favorite word - thanks!
Yeah, Extinction, every should have their own backups of their contacts... and pictures... and text messages.
Wouldn't you think the company you're throwing over $100 a month at could possibly care for your data or at least not have it do such a stupid thing like overwrite your local data without your consent. If anything, it should have merely contacted your phone regularly to update the cloud and not interact with your phone unless you specifically request the information from the cloud.
whats a computer
@BratPAQ
Microsoft and Hitachi simultaneously: T-mobile did it!
@Ignatius:
That's not how the Sidekick works. The device only stores the user information for as long as it is powered. Once reset, the information is gone and it requests the data from the servers again. Except - oops - the data's not there.
People should keep backups. That means big companies AND individual customers. Protect your data.
Fortunately, this massive loss of data is rather unimportant in the greater scheme of things. Just take it in stride. Sidekick users should just chill out, go for a walk, take a deep breath and look at the blue sky. Besides, I don't own a Sidekick, so what's all the fuss about.
As a former T-Mobile employee who was with the company leading up to, during and after the Sidekick's initial launch back in 2003, I have to say, on many occasions I said this day was coming, and that the device needed some form of local data storage built into it. Even if it were nothing more than a non-consumer accessible NAND capable of making full back-ups of user memory on the fly.
The problem for this lies squarely on the feet of Danger (and I guess now Microsoft, now that Microsoft owns Danger).
At the time, Danger, composed largely of former Apple and former Nintendo employees, wanted to keep everything about Sidekick's back end fucntionings a closed platform and as such, closed off from the consumer - in a very similar way to closed platforms like game consoles and many Apple devices at the time. So no matter how often T-Mo employees cried about the danger (no pun intended) of a lack of local data backup (mostly passing concern directly from the consumer, up the line), our voices and warnings went unheard against the brick wall that was Danger's corporate policy about how their devices and services were managed.
And even later Sidekick models that were updated to include SD memory slots (not long after I left the company), as I have been led to understand, the use of the SD memory was locked almost purely to media usage (mp3s, photos, etc.) and could not be used for backing up critical user memory, locally.
Of course no one (at least no one I had contact with at T-Mo), had any clue that Danger had established a cloud network with no IT industry standard redundancies built into their user data storage backend.
Shame of it all, is that the Sidekick was and still is an awesome device; I personally owned the original three Sidekick models . . . of course back then I owned as many as 70 different GSM phones (import and domestic). One of the perks of being an employee of a mobile phone company, is being able to own half of the phones they carry at any one time, and a kick ass employee rate plan to go along with. But I've always had a soft spot in my heart for the Sidekicks. As many (bad) policies as Danger had in place to keep the device a toy, they had just as much going for it to make Sidekicks easily one of the best devices on the planet - not least of which, is one of the best formfactors, and one of the best portable keyboards ever invented. And so far as being in the cloud, even the original Sidekick was 5 to 10 years ahead of the rest of the computing and CE industries. Of course none of that really matters if Danger firstly provides no way for consumers to safely back up their own memory, and worse still, manages to lose the same user memory they would not allow users to back up locally. It's really sad (for Sidekick users), and shameful (of Danger), that it had to come to this.
That's epic. So so bad. Makes me not want to use any Hitachi drives just to not be party in even an indirect way to such epic fail.
I have never found Hitachi drives reliable after mine decided to unformat itself.
Microsoft is the party to be blamed for cost cutting on their product support.
If I were T-Mobile, I would remove all WinMo phones from the shelf!
Why should a carrier buy a phone from the company that provides
mediocre support.
Hitachi Deathstar drives will never fail!
It's not uncommon for these SAN guys to come in...
...not want to wait on a backup
...reassure you that everything will be ok
...(and it is ok 99.999 percent of the time)
...start an upgrade without a specific backup in place
...blame you because they "thought you would have nightly backups, site redundancy, [fill in blank for (relatively) easy/cheap to implement]" etc
Heh, I find it funny that I got a call from Hitachi at work last week asking if we're interested in buying Hitachi SAN storage and servers...
Danger, Steve Ballmer, Danger! Poor IT management processes up ahead!
One high-velocity chair coming your way. Grrrrr.
i perplexed as to how they don't have any backups of any prior time.
So much information they probably only have the space for one backup per customer.
Seriously. I have a home server, and everything is backed up, and it's just for me and my family. How the hell does a corporation not do this?
Agreed. I'm an accountant and one of the CPA exams has a portion of IT on it... even there, you are expected to know about the "grandfather father son" backup methodology; meaning if you back up daily, and today is Friday, you have Thursday, Wednesday, and Tuesday's backups available.
This really is unacceptable. Too big to have more than one backup is garbage, storage is cheap.
And as someone below also noted, it's just poorly designed to rely solely on the cloud.
Why doesn't the Sidekick store information too?
It does, but if the phone reboots and it syncs with nothing there, I guess its screwed. The best thing someone can do is sync to outlook, I guess.
As an iPhone user, I'm still unclear after that explanation. If a device reboots and finds the cloud has no data why would it erase its local copy? Hitachi screw-up aside, that's some pretty dumb device design right there. How hard would it have been for them to program it: IF NO CLOUD DATA THEN BYPASS LOCAL ERASE ?
I'm guessing the sidekick uses Volatile memory to store its data, which is why it can't store data after a reboot... so yeah this is piss poor design considering how cheap flash memory is these days, and no there is no code in the phone commanding it to wipe out all data inside the phone if the server is empty, it simply is just a case of the memory not being able to remember anything at all after a power cut or reboot
They really fucked the pooch on this one.
I think at this point we are well past pooch and into fucked the T-rex territory.
AHAHAHHAH!!!!!!!!!!!!!!!!!!!!!!!!
What happened to on-board phone memory? And how is that different than Google phones?
from my understanding sidekicks store data and syncs its data with the cloud but what happened was that the servers went screwy so when the side kick sync with the cloud it thought that the user deleted all there contacts because they was nothing to sync. android phones are different because google stores information multiple times in different locations so it would be extremely difficult for this to happen with android.
Android phones are different because they aren't thin clients like sidekicks. Nothing to do with Google's backup policies.
No the blame lies at poorly designed software. Why would you have a machine that doesn't keep any information on the actual device.
Aside from not understand how Hitachi could forgo doing a complete backup before upgrading the hardware, I'm at a complete loss as to how there are NO backups from the past available. I just don't understand... I'm still in high school (I intern with my school's IT department) and we ALWAYS make sure there are good backups available before doing anything to our servers. Our most critical one is always backed up before it's even rebooted. ALWAYS. There's just no question about it. Come on MS/Danger/Hitachi.
@MRCUR, you're right, there is absolutely no excuse for this happening. As you state, there should have been backups made in case of a failure such as this. Moreover, this is why you have a contingency plan (which is inclusive of mission critical backup policies) in place before you make a system change (hardware or software.)
@MRCUR
You're absolutely right. I can see how this can happen though. For us, the SAN IS our primary backup. Every 5 min or so, the 50+ servers back themselves up to the 5 EB SAN. That said, there is NO EXCUSE for not creating ANY backup of any kind before even thinking about touching a SAN. Its just a big expensive RAID array (Gross simplification), but when We upgraded our SAN, we spent 3 Days and 100+ Tapes backing it up because the risk of data loss is so high. Surprised Hitachi or MS or whoever didn't do this.
@PBB - I completely understand that. But applying your backup solution to MS in this situation they should still have all the data on the actual cloud servers. It seems like the SAN was the ONLY place customer's data was stored, and thus should've been backed up continuously - especially before being touched. I still just can't wrap my head around what a huge mistake Hitachi made here.
Agreed.
I'm an engineering coop / I.T. wiz kid for my company, and along with the server running raid 3 (its either 3 or 4, i forget off the top of my head - 3 hot swappable drives anyway), we have a dedicated external hard drive for every night of the week for backups. I can pull a file from nearly 6 months ago at the click of a mouse.
I could understand to a degree if your new hardware didn't play nice with your old data, and thus corrupted it, but to move all the data to new equipment, regardless of the number of tests that were preformed, without any form of backup is completely unacceptable.
My lesson learned. Never buy Microsoft stuff and leave my data with them ever again.
Thanks to them my Sidekick LX is as good as a dead paperweight. All of my contacts and messages and other data gone.
Sprint Hero - here I come. Hello Google. Hello Android.
Sigh.
Just got my Hero today, and boy oh boy, it's bitchin'!!!!
On August 19th, you wrote using the name rocket (http://www.engadget.com/2009/08/19/blackberry-browser-to-get-full-flash-and-silverlight-support/)
"The loading craps out on my BlackBerry Storm and Bold though."
And August 23rd, you wrote using the name rocket (http://www.engadget.com/2009/08/23/video-storm-2s-new-touchscreen-tech-explained-with-mindblowing/):
"This is coming from a BlackBerry Bold user."
So you wrote a number of times that you have a Blackberry, but suddenly now you also have a Sidekick? It is possible, but unlikely - and just trying to get a jab in at Microsoft.
Look, here is the thing. Microsoft needs to take responsibility because they now own Danger. However, it is not exactly their fault. They bought Danger which apparently had no data backup mechanism. It seems as though they were trying to setup their network to do proper backups, but the outsourcing of that failed in the process.
I would also like to point out that when Mobile Me was destroying user's data, where was Engadget calling for investigations, firings, legal action, etc? They called it "FAIL" and that was all.
Do you not allow me to have more than 1 phone? Tons have folks have more than 1 phone. My main phone's a BlackBerry but I used to have a Sidekick LX. Now my Sidekick LX is forced to retire, so I'll go get the Hero.
If you don't know FACTs about other people, don't pretend you know.
Well, the fact is that you talk about the iPhone, Palm Pre, and Android browsers ("Best Buy USA and Canada full site and mobile site both works fine on the iPhone 3GS and Palm Pre", "Please update your OS, especially if you want to keep up with Android, WebOS and iPhone.", "its own in-house browser is still far slower than the WebOS browser, let alone being compared to the iPhone browser,"). But you never made any mention of the Sidekick browser, either positive or negative.
You have experience using all those phones, know them in-depth enough to talk about image compression on the different browsers available for the various phones, never mentioned the sidekick, and now you have that one also?
As I wrote, it is possible you own them, but I have my doubts.
Yeah, it's pretty obvious you're full of shit. It's okay. Just do what everyone else who's full of shit does and make your profile private and stop replying to this article. I'm sure you can Google a tutorial on it or something.
Agreed. My life has been so much better after 5 months of being Microsoft-free. I just wish I had done it sooner.
Danger and MS are inseparable, because MS owns Danger. How can you say this isn't MS' responsibility? MS contracted a company to upgrade the SAN and the data was lost. How is that not MS' fault?
I fail to understand why you're apologizing for MS here. They screwed up. First by cutting Danger's staff to the bone and then for not having backups of critical data when the stuff hit the fan.
LS: "How can you say this isn't MS' responsibility?"
Me: "Microsoft needs to take responsibility because they now own Danger"
Somebody is not reading before they criticize.
It is obvious that Danger did not have an incremental backup mechanism in place, otherwise they (MS) would be able to restore that data easily with minimal loss. It took MS a while to get around to changing the infrastructure (too long IMO), and as you said a few days ago, they were doing a backup but it was not completed before Hitachi started upgrading the SAN. The question is, did Hitachi do the work before they were supposed to start, was Hitachi in charge of doing the backup and screwed it up, or did MS tell them to start before they should have? If it is the latter, then it is purely MS' fault. If it is option 1 or 2, then you pay a company to do the job properly. And while MS is ultimately responsible, it is not them that caused the problem.
Looking up Danger on Wikipedia gives this interesting bit of data: "The company was originally started by Apple Inc. and Philips executives." - interesting for many reasons.
They not only raped the customers, they also bought them puppies, raped the puppies, and then gave the puppies to them and forced them to keep the puppies.
Epic cloud computing FAIL. Also this has all the classic ingredients for a big time class action suit.
I agree.. class action. The question is, who really is the defendant here? Tmo, MS, or Danger? MS does own Danger, but they are certainly separate division. Though it all comes from the top, I am guessing.
@TZK, I assume that the lawyers will find a way to name all parties involved (T-Mobile, Microsoft, Danger) as defendants.
The suit is with T-Mobile, of course. The customer has no contract with Microsoft/Danger.
I think T-Mobile could demand MSFT and Danger cover damages though, since a lot of this was out of their hands. They will have to pony up to their customers, but they need to make the other 2 pay those damages. They themselves may need to file a lawsuit if MSFT and Danger tries to leave them holding the bag.
FUD
Backups? We don't need no stinking backups!
You forgot "Backups? We don't need no stinking backups! THIS IS SPARTA"
I had a Sidekick for years -- starting with the b/w model -- before I moved to the iPhone. Poor Danger... they were doing it right for so long, and now because of Microsoft, their brand is completely screwed.
Good job Microsoft. YOU GUYS SUCK!
yeah i have owned a sidekick since the sidekick color, they were finally getting everything i needed and wanted in a phone, in a size and screen and full qwerty i could easily use. then microsoft came and shit all over then set it on fire and now will probably use danger as a scape goat to deal the finishing blow.
oh well, maybe now they will sell off hinge rights and we can have some more innovative swivel phones from more exciting hardware manufacturers.
Let's keep one thing in mind. This was not Microsoft (as in the company) fault. Yes from the broad sense it was because a handful of idiots at MS didn't implement backups. However do you really think the Windows team was responsible? The Office team? The Xbox team? The Xbox Live team? Etc.
A manager and a number of other employees were probably directly responsible for this. And also this: I know of no company the size of MS who doesn't have a change management system in place where you don't sneeze around major hardware without having a change request in place. I can NOT believe that something of this magnitude didn't trigger questions about backups. So there definitely are a ton of unanswered questions on what went down right now.
On the other hand, if this didn't happen, nobody would have appretiated actual working cloud systems with proper backups! :) Admit it, have you ever thought of those before?
proper backups, nuff said.
One good thing to come out of this, Robin is no longer the most useless sidekick ever
Lol.
Although I heard he had a Sidekick and now he can't call Batman for help
Ooh I have another one!
Those Hitachi engineers must have had their heads in the clouds when they did that update
How about this one:
All their data is really... gone...
is this a way for microsoft to say to google that cloud data is inherently undependable. sounds perverse but think about it for a second.
Microsoft is working on their own cloud service called Azure. I doubt they would want to discredit the entire concept to screw Google when they are trying to publish the same thing - otherwise people may not buy into their service.
Uh, Why doesn't the phone hold its own information. This isn't a MS/Hitachi fail, its a designer fail. None of my info needs to be stored on "the cloud" My phones have been storing my contacts and messages for years now without any problems...
WOW.. and THIS on the Eave of T-Mobile launching some crazy new service update, phones and coverage for their 3G. Bad timing for sure, man.
I have a LX 2007... or had, rather. I bought it since unlimited data/text on pre-paid was dirt cheap (think $40/month), but this was enough to push me away. I just bought a Palm Pre; it should be here later this week.
Not surprised. Hitachi Data Systems reps in the Seattle area (if the Danger/Tmobile/Microsoft joint DC is in the Seattle area) aren't very competent. I used to work for a company that had two HDS SANs and supervised an update to one (HDS updates are only allowed to be performed by Hitachi, doesn't matter if you work for the company or not). We had "problems" half way. The SAN now was reporting failed hard drives all over the place because of this new firmware that we were required to be at in order to get support for next year. I thankfully had a live backup on standby in our backup DC. But I could have been in a very similar boat.
Microsoft let this happen for two reasons:
1. They don't give a shit about Sidekick. Never did.
2. It's a great opportunity to discredit cloud computing and Google services for the enterprise. Which is a threat to Microsoft office products.
I bet there are some IT managers who planned to move to the cloud and are freaking out after this incident.
????? perverse thought isin't it
Pfft, you call that a conspiracy theory? How about this one
This massive crash was engineered to spark up conversation about cloud computing and clouds in general, which would lead people on to talking about Pixars newly released movie, Up. And who is it who used to own Pixar and now owns a large stake in Disney? Steve Jobs of course.
It was all a massive plot to tarnish Microsofts reputation and at the same time boost movie sales. Genius I say
Nah, no big conspiracy here, Microsoft products/services have always been unreliable and crappy like this.
"Never attribute to malice that which can be adequately explained by stupidity, but don't rule out malice" Heinlein's Razor
HAHA; THanks.. this is why the cloud is my pc; and the device is my iphone. Not even gonna touch mobile me now :D!
"We're betting that heads are going to roll at both of these companies, formal investigations are going to be waged, users are going to be compensated in big ways, lawsuits are going to be filed, and textbooks could very well be modified to make sure that lessons are learned for the next generation of college grads tasked with keeping clouds running."
Hahah thanks for the laugh! In the eyes of MS and T-Mo, it's just a few teenyboppers who lost some of their contacts lists. No big deal. I'd be surprised to see anything big come out of this - and no, a token gift certificate only redeemable at danger.com doesn't qualify.
You can't trust nobody in this world, except yourself. This is why cloud computing poses risks!
This is why I would never trust "the cloud."
the only people I have ever seen using a sidekick are screaming teenagers on the subway in nyc. I'm not sure what kind of mission-critical data they have, but it will be interesting to see what kind of compensation is offered after the dust settles.
The future of the cloud is kinda cloudy.
Ok, so it was a cock-up and a pretty bog one at that.
But only in the US would this spark comments about class-action lawsuits, compensation and people being fired..
It's just some contacts and messages - no worse than losing or breaking a phone that isn't connected to a cloud.
Plenty of people have ended up in similar situations before and survived, so what's the big deal?
umm i'm pretty sure losing all of the information that thousands of users have been paying to keep for years sort of warrants serious compensation
Let's see, Sidekick relies on the cloud to sync data and there's no back-up to PC from what I've read. Not to mention people are paying on a contract or plan and expect their phones to work. ALL DATA IS LOST through stupid laziness on the companies' part and there's NO WAY to recover it, screwing over thousands of customers since back-ups to PCs aren't how this phone works.. Yet you say no one should get fired for just willy nilly screwing over paying customers????? And no customer should be compensated for lost data that was placed in these companies' trust???? I don't know about a class action YET because the companies could step forward and do the right thing, but I'd for sure be demanding the other 2 right now!!!!
I am sure there will be a big class action lawsuit.
Every sidekick owner will be well compensated for his loss (I am guessing about $15.87), the lawyer will be well compensated for his efforts ($500 million), and the general public will be well compensated (by the cost of future sidekicks rising by $50).
Class action lawsuits generally only work out well for the lawyers. God Bless America.
This was just a big bag o fail on the server master's and dangers behalf. There is nothing to excuse the lack of backing up on a corporate level. I hope heads will roll and tht the customers will be fairly compensated for this inexusable data loss.
My miserable life is made a little bit happier thinking about the fucks at MS losing sleep over this tonight.
And one wonders why your life is miserable when you spend your time hoping for pain in others.
i can just imagine a lot of execs including steve ballmers face turning red because of too much facepalm. a lot of head will roll here, but that would not make it up for the damage those few people caused to the entire company.
phones hold their own data. the issue arises if the synch server has zero data, then your device reads it as the user deleted that data so it synchs to the server and likewise deletes the phone data. unless that device has settings you can set to always synch from the phone rather than synch from server. although what baffles me would be how the aoftware is written to even delete phone data. when i synch my nokia with ovi cloud, if there are no data on the cloud, my device remains intact. what a strange situation for dangerf and ms to have this happen in this day and age.