Developer round table: MMO betas, page 2
Jack Emmert, Cryptic Studios COO
What defines a successful MMO beta?
First, the question needs to be defined – open beta or closed beta?
In a closed beta, the goal is to get everything fixed. Broken missions. Malfunctioning servers. Bandwidth issues. Anything that prevents a player from, well, playing. That's the first step.
Next, developers need to analyze what players enjoy and what they don't. In a successful closed beta, the developer responds to community concerns to everyone's satisfaction. By the time open beta rolls around, everyone is loving the game. By the way, the surest way of noting this is the CCU: if the CCU is stable, then you know people are enjoying themselves (they keep coming back). But if the CCU falls by 10%+ each session – there's a big problem.
Open beta is for shaking out bugs, to a very small degree, but this is really when the game is public for the first time. And a developer needs the game essentially to be finished. Most likely, the game has already gone gold. A successful open beta means that the internet is buzzing and pre-orders are jumping significantly (in the tens of thousands, at least).
Conversely, what are some hazards to avoid when testing?
It's easy to overreact to a community. A group of people might be shrieking about a feature that isn't that big of a deal in the grand scheme. It's easy to pour valuable development time into things which won't really sell any more copies. That's the key during beta: what changes will net more sales, more subscriptions.
On the other hand, it's easy to ignore the community altogether. Developers tend to get into their "ivy towers" and pontificate on how things should be, but if a game isn't fun, all the theories in the world don't help.
How do you think the beta process has evolved in the last year or two?
Bugs in open beta are simply not acceptable. It used to be that games could launch in a somewhat rough state; now, customers expect the same level of execution of a console game (or near about).
I don't know if this has really changed over the years, but customers also don't wait to see where the game grows – what gets added or not (I should say reviewers fall into this trap also). Essentially, a player evaluates the game immediately and chooses whether or not he likes it.
How do you think the beta process could be improved?
From Cryptic's standpoint, I think we should be internally putting milestones in where we play the game and make a judgment on how well development is progressing. We did this last September for Champions and changed a lot of development priorities and principles. I think this was a good idea, but that Cryptic should be taking our bearings and correcting our course more frequently.
Josh Drescher, Mythic Entertainment Producer
What defines a successful MMO beta?
That differs based on all sorts of variables. Time, budget, product, studio, target market and more all factor in heavily. Obviously, in a perfect world, we'd always be able to do relentless testing with enormous user-bases that isn't limited by schedules, finances, player interest and so forth. But no product exists in that kind of "ideal" environment. As a result, you set pragmatic goals based upon the time and resources you have available.
With that in mind, I would say a Successful MMO Beta:
Knows what it wants to accomplish and limits itself to that.
Communicates the purposes, structure and goals of each phase to its testers.
Provides coherent, comprehensive explanations of what should be tested and how it should be tested.
Gives testers a convenient, coherent way to provide both simple and detailed feedback.
Conversely, what are some hazards to avoid when testing?
I could list a lot of obvious stuff about scheduling, resources, duration, etc. Instead I'll call out one, less obvious hazard:
Listening too much to what hardcore players say and – conversely – not listening closely enough to that same group.
While beta tests are a chance for developers to get large-scale bug-hunting done, their most valuable function is providing feedback on gameplay, features, content, etc. MMOs are fairly unique creatures in that our players are significantly more invested in the development of our games than the fans of nearly any other genre. As a result, the fans are very aware of the "nuts and bolts" of design, core mechanics, project goals, etc.
The challenge for developers is to be able to recognize the often very fine line between "informed player" and "armchair designer". The former has an awareness of MMOs in general and your game in particular that can be very helpful to developers, as it gives potential beta testers a very strong, innate grasp of how they should interact with your game. The latter makes inappropriate leaps of judgment based upon their own incomplete understanding of what development is actually like and combines that with their awareness of MMOs to make claims that seem reasonable but which are actually preposterous.
It's like proving the Earth isn't flat. "Common sense" (with which the "armchair designer" is well-armed) tells you that the Earth is flat. It sure looks flat. It sure seems and feels flat. Therefore, it must be flat! Proving that it's not flat takes some fairly complex and challenging evidence. Even with that at your disposal there will still be plenty of people who just reject your evidence and continue to claim that the Earth really is flat.
As a developer, you're never going to be able to explain things well enough to satisfy the armchair designers, so you need to develop the discipline necessary to recognize and filter them accordingly.
How do you think the beta process has evolved in the last year or two?
"Evolve" implies improvement and I'm not sure that's appropriate. If anything, the role and definition of the "beta test" has degraded in recent years.
Beta tests are becoming more and more visible as the genre has grown. As more players start tracking games early on in their development, beta tests loom larger and larger as a significant hurdle rather than simply acting as a development tool. You really can't use it exclusively for its intended purpose (testing an incomplete game) anymore because the beta test is now viewed as an "event" unto itself. It's nearly as significant in the minds of the public as your actual launch, so developers are basically forced to have a complete game ready prior to going into Beta.
This has, in turn, applied pressure to have shorter testing periods in many cases (because so much extra development time was used up prior to the beta). I'm guessing we'll eventually get to a situation similar to what you find in Asia where "beta" tests are actually soft launches of the final product.
How do you think the beta process could be improved?
I'd jettison the idea that it's necessary to have a "public beta" where anyone and everyone is given access to the game. Beta tests are not free trials of your game. If you feel that a trial is appropriate, that's fine – just make sure you offer it as something that is totally separate from your beta test.
