Why testing matters

People often look at things like the current Friends and Family Alpha test and wonder why they exist. Why not go straight to beta, or even just do everything via internal testing? Well, there's lots of reasons. One of them is one expressed very well in this tweet by Russ Petersen.

What we've seen happen through several betas is that content has been changed, reworked and redesigned based on Alpha or Beta feedback. I'm reminded of the Cataclysm expansion and Paragon's effort to be World First on Heroic Sinestra, a fight that simply hadn't been tested very much externally, in order to keep it relatively unknown. This meant that it was a heroic-only boss, being attempted for world-first status, while the fight was being hotfixed. It was necessary to preserve some of the mystery of the fight, because let's face it, if it had been open to testing someone would have written up the strategy and Paragon would have been going in knowing what the fight was and wasn't. But it also meant that Paragon had to learn what the fight was and wasn't while it was being changed, during the attempts.

Similarly, the drastic revamp of the Jade Forest during Mists beta can't be undersold here. This happened right in the middle of the beta -- players who were testing out content had to be moved to the next zone and leveled up while the Forest was completely redesigned. In both cases, we see how the presence or absence of testing can have wide consequences for everyone.

The reason for expanding tests. Tests that start small and then bring in more and more people. There are many reasons to do this. The initial small group brings focus and is controllable. As they find bugs and glitches and flaws in the content, changes are made, but as they see it over and over again they too start to have difficulty differentiating the newer version form any number of previous versions. Having newer pairs of eyes and fresher minds look at the next iteration means that they don't remember how it was before, they don't make that mistake. The next reason is the shakedown phase -- the more people you have in a test, the more likely they're providing stress -- it's valuable to see if a texture bug happens to a wide variety of players, or if the server can handle hundreds, thousands of people playing X or Y.

Furthermore, there are so many ways a game can break. The Warlords of Draenor alpha is a huge collection of virtual moving parts, with new character models, new itemization, the new garrison system, new kinds of encounters and experiences to be had when exploring the zones. Flooding the alpha with testers now would mean they'd be testing stuff that's already known to be buggy, but having no alpha test would mean throwing people into a beta made up of untested bits. And it's true what they say: a beta you can't even run isn't testable.

Testing doesn't always catch everything. I ended up on a boat wedged into a mountain when the Ahn'Qiraj patch went live, despite months of testing. Similarly, a great many bugs persisted during the original run of World of Warcraft despite a fairly long alpha and beta process. Talents didn't work right, combat logs were imperfect at best, there were desynchronization bugs and looting bugs. I once got stuck trying to mine a copper node and had to slide like a broken-backed Igor all the way to Stormwind before it unglitched. The idea isn't that testing catches everything -- but without it, we all end up catching everything during actual play.