Developer round table: MMO betas
Is beta testing something of a mystery to you? Do you wonder about the how and why of a developer's beta process?
Well, you're in for a treat! We recently got in touch with several MMO developers to ask them each a series of questions about MMO beta testing. In this feature you'll find opinions from Cryptic Studios, Fallen Earth LLC, Mythic Entertainment, Turbine, Runic Games and Hi-Rez Studios. It's a four-page long read, but we think you'll find lots to enjoy within those pages. For commenting, hit up page four -- we can't wait to read your thoughts!
Lee Hammock, Fallen Earth Lead Game Designer
What defines a successful MMO beta?
Unfortunately these days betas are about marketing as much if not more so than they are about testing your game, so a successful beta has become one where there are minimal problems as opposed to one where you fixed a lot of problems. Many players use open betas as a free trial of the game, so if it's not working as well as a published game they pass on it even though the purpose of a beta is to fix problems. Also, the vast majority of beta players do not submit bugs or really any feedback at all, making them only useful in that they add stress to the servers.
In an ideal world, a successful MMO beta would be one where lots of bugs were found and fixed, systems tuned, stability increased, etc. Unfortunately, a company that goes in with those expectations (i.e. fixing things in beta) is going to end up with a lot of testers who get turned off from the game. Alpha testing has effectively become what beta testing used to be, and I don't think there will be a shift backwards on that since so many companies now use their beta test as a major segment of their marketing efforts.
So while the ideal successful MMO beta would be one where the game gets better, a more realistic goal would be to have a beta where the game does improve (but is already in good shape before getting to beta), the game serves as an effective marketing tool, and enough of the beta testers become attached to the game that you already have a nascent fan community once the game goes live.
Conversely, what are some hazards to avoid when testing?
Being too stingy with your beta keys. If you don't get a certain critical mass of players you won't get the stress numbers you need to make sure everything is working correctly.
Not providing enough direction in your testing. Your players will not hit everything you need to test in the numbers you need on their own. While allowing the testing to occur organically (i.e. letting them level up instead of giving them high level characters to test, letting them play through the game as they want instead of directing them to specific areas) is better so players have a better understanding of the game systems as a whole, it isn't feasible for testing everything.
Being afraid to make changes. As mentioned above betas are held to a higher standard than they should by many players, but at the same time they're not paying for it so if there are large scale changes to be made do them before anyone pays for the game.
Listening to the loudest criticisms instead of the best reasoned criticisms. Granted this is always a problem with game development, but since betas exist to get feedback this is especially important.
Not communicating with your players. Beta is a great time for telling players how things are supposed to work, giving them formulas, etc, so they can make better informed bugs. The more informed your testers the better info they will give you.
Take testers (and their bugs) for granted. As mentioned earlier most beta testers don't actually submit bugs. You tend to get a minority that submit the majority of bugs, and usually work hard at doing so for one reason or another. If you can motivate the non-bug submitted testers to become bug submitters you can greatly increase the amount of feedback you get. This can be through things like contests, public recognition of testers, personal responses to bugs, etc. Making the testers feel like valued members of the team helps their productivity immensely.
Waiting too long to get outsiders into your game. This may be more of an alpha test issue, but the sooner you can have people outside the company playing the game and giving feedback the better. The dev team cannot really evaluate their own creation effectively and the sooner they get fair feedback the sooner they can really start improving the game. Just make sure those early testers know what they are getting into and understand their NDA.
How do you think the beta process has evolved in the last year or two?
It's become more about marketing and less about improving the game. Now improving the game is more the focus of alpha testing, but that is problematic since alpha testing usually involved far fewer testers than beta testing (thus certain aspects of the game do not get tested). That said, I think the plans for targeted testing have improved and developers are better able to look at specific problem areas. On the other hand it does seem like many developers are less interested in making large changes in beta even if a problem is present, probably because at that point they have so much money invested on the game running along a set schedule they can't make those changes. This to me seems a symptom of not getting their game in front of outsiders as soon as possible.
How do you think the beta process could be improved?
As much as I would love to tone down the marketing element of beta testing I don't think that is likely.
The main improvement I can think of is sharing more information with the players; not just numbers and formulae for them to check, but also overall design goals and themes. If a mission or zone is supposed to evoke certain feelings in a player but doesn't, that's not exactly a bug but is feedback that is useful to the developers. Knowing the goals of the developers is extremely useful for players so they can note when those goals are achieved.
Also avoiding the hazards mentioned previously.
