In the early years of WoW, whenever Blizzard upgraded their hardware, rumors swirled about what they were buying from whom. People posted messages on forums swearing that their brother-in-law runs a warehouse and they just sent a big shipment of Itanium Blades to Blizzard's Boston data center. No one believed them.
Even journalists who have been invited to tour Blizzard's Irvine facilities could only briefly glimpse (through a window) the "Global Operations" room that monitors the realms. No photographs of this room are permitted, but Ars Technica gave a written description in 2010:
What we saw were weather maps making sure things were stable at the data centers around the world. Data and charts flashing on the large screens hanging on the wall. Rows of computers in the middle of the room being tended to by very serious looking people. Millions of players, around the world, all watched and tended to from that room. It was quite the sight.
That is the monitoring room of today. Only Blizzard's staff knows what it looked like in the winter of 2004, and they're not telling. If Blizzard had such equipment then, one imagines it would have been on red alert around the clock, rows of dials buried at maximum, sirens blaring, perhaps some steam jetting out of pipe joints.
Because there were not enough realms. Not even close.
As word of mouth spread and reviewers gushed, WoW's population climbed to terrifying totals faster than Blizzard ever dared to hope. The sales may been an amazing boon to Blizzard's bottom line, but for the people trying to keep the game running smoothly, they were an absolute nightmare. No one had expected such an unrelenting population surge, and the realms were paying the price.
Technical problems began from the very first day of operation: frequent disconnects, rampant lag, and crashing realms. After a crash, when the realm came back online, everyone tried to log in at once. Overloaded log-in servers couldn't handle the sudden flood. Thousands of angry new subscribers could only stare at their screens, unable to play.
Blogging about his experience with the technical issues, Gamespy's Tom Chick described a particularly irritating play session the day after launch:
2:15 P.M. So the main screen has an announcement that some of the servers are coming down "for emergency maintenance." Sounds dire. Frostwolf isn't listed among the afflicted, so I figure maybe it'll be time for [a] relaxing session of fishing. Instead, I'm treated to a session of the "Retrieving character list" screen.
2:52 P.M. I'm finally in the server after a long session with the loading screen. The game seems to be running smoothly, with the exception that I can't interact with anything. In the process of being unable to interact with anything, I get killed. Now I'm waiting to turn into a ghost so I can go back and get my body.
3:02 P.M. Okay, I'm a ghost and I'm heading back to my body.
3:03 P.M. I've found my body, but I can't get back into it. I'm just hovering here staring at it. Locked out of my own body. How Manichean.
3:13 P.M. The "Resurrect Now?" box has popped up. I press "Accept" and nothing happens.
3:19 P.M. I am finally restored into my body. I hearthstone back to the inn and log out.
3:25 P.M. The server has finally logged me out and now I'm staring at the same "Retrieving character list" screen that started what was supposed to be a relaxing session of fishing a half hour ago. I think I'm done with World of Warcraft for the day.
Note the time stamps. It took Chick 37 minutes to log in. It took 16 minutes for him to resurrect himself once his character had returned to his body (after extreme server lag got him killed). Even after he'd had enough, it took him about 5 minutes just to log out. All in all he spent more than an hour fruitlessly attempting to play the game. Players on many realms had similar experiences.
There was, of course, an outcry on the official forums, and pretty much everywhere else.
One of the most pervasive issues was the strain on the items database. Thousands of players collecting items from millions of slain enemies created a load of information that a realm's database simply could not respond fast enough to manage. As a result, your character would remain stuck in the looting animation: hunched over and reaching toward the ground.
When players gave up on collecting their loot, they'd move away from the enemy corpse, but the running animation wouldn't kick in. Entire zones were full of players zooming around but otherwise motionless and crouching, as if propelled by magic and wishes alone. A .gif of a man experiencing the bug in the real world parodied the eerie crouching glide.
The problem became known as "loot lag." Even if you engaged in combat, your character would return to the looting position afterward. Compounding the situation, gathering nodes sometimes bugged out and stuck you with loot lag even when a realm had minimal lag. Players called these nodes "cursed herbs" or "cursed ore."
Loot lag had two cures. You could either wait for the database to catch up (a process that could take ten minutes or longer) or you could log out and back in (which in those days was a proposition fraught with risk).
By far the most agonizing problem, however, was realm crashes, because they were often accompanied by a "rollback." A rollback occurred when a realm was reset. After a rollback, everything you did for the last few minutes prior to the crash was erased from your spacetime continuum as if it never happened. A player who made a long journey back to town could find herself back in the wilderness again. A player who had just carefully redistributed all of her talent points could log in to see all those points back the way they used to be. Dungeon parties who had finally beaten a tough boss and gotten some coveted items from the kill could find themselves staring at that same boss, alive and well.
There's something psychologically devastating about seeing one's hard work and accomplishments undone. Whenever something exceptionally awesome happened, players suffered a fierce dread that a crash would expunge that event from the server's data banks. Only once enough time had passed did players feel secure that something had actually, permanently happened.
As a result of these difficulties, Gamespy's Mark Asher warned people away from the game.
Let me say that the difference between playing World of Warcraft during the day and evening is, well, the difference between night and day. World of Warcraft is just getting crushed in the evenings by the player load and it becomes unplayable for many players. I'm not saying it's laggy -- I'm saying it's literally unplayable at times . . . It's really bad right now. In fact, if you're on the fence about buying World of Warcraft, don't until Blizzard fixes these problems. I think they will, but save yourself the frustration until they do.
Running damage control over the server issues, Blizzard president Mike Morhaime made an announcement:
We were all extremely pleased with the success of World of Warcraft on its first day of launch. Once we saw the numbers for the first day, we knew that we had to immediately increase [server] capacity to accommodate the huge numbers of players joining our game. We're glad so many people are enjoying World of Warcraft, and we are dedicated to supporting a fun and smooth game experience for everyone.
Blizzard indeed had a plan. There was only one problem: the runaway freight train of game sales outpaced all contingencies.
Blizzard launched the game with 41 realms for North America, Australia, and New Zealand. Before the game went on sale, the company had a plan to roll out more realms gradually over time, hoping that the population of the game would slowly grow and require additional realms.
After the first day of sales, the entire backup plan -- what Blizzard had estimated to be about a year's worth of new hardware -- went into immediate effect. Within just a few weeks, Blizzard had doubled its North American server capacity to 75.
But there were not enough realms. Not even close.
Keepers of downtime
Blizzard's server team continued adding hardware and upgrading existing realms to better support massive populations. Of course, whenever realms were upgraded or undergoing maintenance, those realms were also down. Meanwhile, the population kept growing. No amount of realms, no matter how quickly they were added, seemed sufficient.
In order to save the servers from certain doom, Blizzard took the only step they could: they implemented server queues. Players on popular realms experienced long waits, sometimes up to four or five hours.
Desperate for their WoW fix, players soon learned how to game the queue. Some made a habit of logging in and saving their spot in line long before they actually had time to play. Some would ask their spouses or siblings at home to log them in before they even left work or school, hoping that the wait would be reasonable by the time they got home. Some players who were lucky enough to get into the server would do everything in their power never to log out, whether they were actively playing or not. Since the game automatically disconnects an inactive player after 15 minutes, they had to be vigilant in order to stay online. As one can imagine, such methods caused longer queues for everyone else.
In the era of queues, a disconnect was devastating. You had to start over at the end of the queue. Given that disconnects were a frequent problem on many servers, this vicious cycle of queuing, waiting, loading into the game world, disconnecting, and re-queuing could repeat itself endlessly.
Status of the status page
By December 7, 2004, servers were down so often for maintenance (emergency and otherwise) that Blizzard added a realm status page to their website. Here, customers could see exactly which realms were currently on- or offline for their region. Blizzard hoped that the site would provide up-to-the-minute information and cut down on the amount of forum threads asking about specific situations.
Ironically, the status page to inform players when realms weren't working did not, itself, work. The page often displayed incorrect statuses, leading to more frustration and complaints.
Blogger Chu Yeow vented about the server status page in early January 2005:
When you have a server status page that doesn't work, and which tauntingly indicates that all servers are up even when they clearly aren't, you have to wonder whether it's a marketing tool for Blizzard, or just plain laziness in fixing a clearly broken script. Check out the World of Warcraft Realm Status page -- I've never seen a realm (i.e. a server) indicated as being down when it actually is down. Right now, all servers are down because of weekly maintenance that goes on for 4 hours. And guess what? The server status pages indicates that ALL servers are up, alive and bursting with players killing goretusk boars and pygmy venom web spiders.
To quell the irate subscriber base, Blizzard began to award game-time extensions whenever the majority of players lost a day to realm issues. They weren't exactly refunds, but an extension would allow you to keep playing for extra days beyond the end of your paid subscription (or, as was the case for most players in December 2004, the month-long, free trial period for new subscribers).
One such unplayable day happened on December 22, the day after Patch 1.2 went live. Players freaked. Blizzard announced an extension the day after, but players wanted functional realms to check out the new content much more than they wanted a free day.
What happened next? Stay tuned for part two!
After months of surveying, WoW Archivist has been dug back up! Discover lore and artifacts of
WoW's past, including the Corrupted Blood plague, the Scepter of the Shifting Sands, and the mysterious Emerald Dream.