Ask Massively: The Alpha and the...um... Beta? of MMORPGs

Here at Ask Massively, I tend to lean towards old school ways of thinking. I remember the days when hype for an upcoming release consisted of a snazzy video trailer and ads on just about every gaming site known to mankind. Now that I think about it, I remember the days before massive advertising for video games; the days when you had to actually walk into a computer store to see what the newest titles on the shelves were. Apparently, I am not alone.

If you have a question for us, here at Ask Massively, feel free to stop by our tipline, or send us an email to ask AT massively DOT com. I would post yet another snarky line about spammers, but one of our staffers is currently working on a plan to ship all of the known spammers in the world to a very uncomfortable place. (editor: What, you mean like the back of a Volkswagen?)

Dear Massively,

What's the story with the MMO beta releases these days? It seems like game companies are just using the "beta test" phase to generate hype for an upcoming release. What's the point of giving away a crippled version of a game that you're going to be charging 15 bucks/month to play in the near future? One particular beta that I managed to play recently actually required me to subscribe to File Planet just to get a copy of the "free" beta client. Now I'm paying to play a beta release? What's up with that?

Cheers,
Confuzzled

Well, Fuzz, it's not as simple as it seems. I assume that you have had your recent experience with the Age of Conan open beta. Some of us have quite a few problems with how this "beta test" went down, but they aren't what you might think they are. In fact, I don't think that Age of Conan is particularly guilty of anything unusual in the MMO industry with respect to beta tests, the problem is that the MMO industry (with one very notable exception) is using the beta test phase for something it wasn't intended to be used for.
The problem I have with MMO beta testing, as it is currently practiced by many software companies, is that there is very little actual testing done. Many, so called, beta testers are just looking to score a free copy of a game before it hits release in order to decide whether or not they will buy the game. The "beta test" has become part of the marketing cycle. That's not why good developers beta test their code. This isn't the fault of the game developer as much as it is an indictment of the avid MMORPG fan who can't wait to play an upcoming release, especially if the "beta test" is free to play. It is important to distinguish between a "free trial" and a "beta test".

To be completely fair, the recent Age of Conan beta was a rather successful test of Funcom's infrastructure and I am reasonably sure that part of the "bloat" of the client involved snippets of code that provided feedback to Funcom in order to help them improve their product. That said, the bloat of the client, as well as rumors of "old client" usage floating around the gaming community might give potential players a bad first impression of an otherwise wonderful game. If I had been representing Funcom, I would have emphasized that this is a test phase and not a free preview. I'd mention the debugging code and add feedback mechanisms to the beta client in order to make it clear to "testers" that this isn't a free ride. Being upfront and honest about the fact that the beta client contains debugging code and other feedback elements would help soothe the expectation of "testers" who might get upset that their free preview is a bit laggy on their system.

As an example of how other companies have managed their beta test without having to worry about the high expectations of "free riders", Sony, with their beta release of Pirates of the Burning Sea , added feedback mechanisms to their beta client that asked the player for criticism during game play, even interrupting the game in order to do so. Players could complete a mission, but then would immediately have a full screen popup asking for feedback and criticism of their gaming experience. That could be frustrating to someone who is looking to play Pirates for free, but was invaluable to someone who was serious about testing Sony's product.

Another example of how to test an upcoming release, although better suited for games that have already been released and are in the process of adding new content, is the public test server. EverQuest used this model for a number of years, and World of Warcraft has used it as well. Sure, players get to play "for free", but the pool of testers is drawn from paid subscribers, thus eliminating the "free rider" that presents such a problem in an open beta. On test servers, players are well aware that content may be buggy, may change without warning, and that their characters may be reset or deleted with no warning. There is no expectation of a "release candidate" product, and players are encouraged to stress the game in whatever fashion they are capable of.

In the end, the problem isn't the company that stages an open beta test. The problem is usually the pool of testers. The average MMO player doesn't know the first thing about testing code. Sure, they can generate bandwidth and mash buttons with the best of them, but systematically testing and verifying functionality is a hard job, and one that the average "beta tester" has no interest in performing. Lately, it seems that MMO companies are just throwing open the doors to their game and hoping that the sample size of testers is large enough to produce some valid or relevant results. It would be more appropriate to call a public "beta" a "stress test" and set expectations with the testers accordingly.

Once again, if you have any questions that you would like to see answered in Ask Massively, feel free to drop us a line at ask AT massively DOT com or stop by our tipline. If you'll excuse me, I'm going to go play the Age of Conan beta.