Developer round table: MMO betas, page 3
Peter Hu, Runic Games founder & CTO
What defines a successful MMO beta?
I think a successful beta is one where, at the end of it, the game is fun, polished, and stable. However, it's notable that it's still mostly MMOs (in games anyway) that have large betas. The two main reasons being that final scalability testing requires lots of people and MMOs require better balance than single-player games. In addition, any time you have thousands of people interacting, the complexity of your systems increases dramatically, and there the significance of bugs becomes magnified.
Some of the keys to holding a successful MMO beta include:
Have it early enough that you can have multiple (as in lots of, not just several) rounds of feedback from the audience.
The team needs to have enough bandwidth to actually implement changes based on feedback (they can't be so completely swamped trying to put in the rest of the scheduled features that they don't have time to even read, much less implement, player feedback).
Have it polished enough that people actually enjoy playing the game (despite the inevitable problems) enough to continue playing and giving feedback.
Have enough to do that players don't get bored during the beta. Don't have so many player wipes or changes that players become frustrated.
Have a good method for collecting and prioritizing feedback.
Keep the players informed as to what is going on, what to test, and what kind of feedback is helpful. Make them feel involved.
Invite enough players to actually do scalability testing.
Conversely, what are some hazards to avoid when testing?
The opposite of above, of course, but also:
Treating the loudest players on your forums as if they were the majority.
Ignoring complainers as "because they're not your audience" - if they're playing in your beta, chances are they are your audience.
Relying on the fans to do all of your testing for you.
If an application doesn't work for a player at first, most of them won't tell you, won't stick around to help you fix it, and you'll never know.
Just because an issue doesn't get reported, doesn't mean it doesn't exist.
Neglecting to build in methods for figuring out what players are doing other than reading the forums.
Rationalize the problems reported by players. Even if what they're saying specifically doesn't make sense, they're still voicing a problem with the game.
Being defensive about the game. The purpose of the beta is to make things better. It isn't about coddling the developer's ego.
How do you think the beta process has evolved in the last year or two?
I haven't beta tested anything in the last two years, so I couldn't really answer this.
How do you think the beta process could be improved?
Most game studios are so strapped for cash and manpower that they simply don't have the time or resources to hold an effective beta. Game development is always a balance between what we'd like to put in and what we have time for.
Starting a small beta and involving the community early can be a great way for developers to integrate the beta process into their development cycle while also gaining practice in providing service to their player base. The downside to starting a beta early is that there often isn't enough content to keep players involved in your beta, so it can be two-edged sword. But generally, I think, having a community while you're developing the game is a beneficial process and helps the team realize what features are really important to their players
Todd Harris, Hi-Rez Studios Executive Producer
What defines a successful MMO beta?
I think MMO developers often have multiple, and sometimes competing goals for Beta. Based on where the game is at in the development process the Beta goals include things like: 1. functionally testing code and content, 2. soliciting feedback to refine gameplay, 3. testing out support tools and infrastructure like verifying server scalability, 4. fostering a tight relationship between the dev team and the community, and 5. exposing the game to a larger audience - basically a marketing driven beta.
All that said, for us success comes down to exceeding the expectations of our Beta testers and measuring that via surveys after each test. A successful Beta is defined by a high percentage of our testers saying they would recommend Global Agenda to a friend.
Conversely, what are some hazards to avoid when testing?
You have to be careful to balance player attachment to character with your beta testing goals. It is great to see our beta testers becoming attached to their characters in terms of XP, level, gear, suits&helmets, achievements and other Global Agenda rewards. At the same time testing goals often require a wipe of those attributes. We try to set expectations clearly that while we are in Closed Beta players should expect these frequent wipes but that is something that should be communicated loudly and frequently.
How do you think the beta process has evolved in the last year or two?
This process has evolved such that more people now perceive Beta as a trial for users rather than an actual test for the developer. And the reality is that it is always a mix of both. With Global Agenda we try to communicate the state of the game in as much detail as possible to our Beta community; i.e. which features are very representative of the finished game and which others are still early in development.
How do you think the beta process could be improved?
Specific to Global Agenda we try to learn from each beta and make the next one better. Our most frequent complaint, if you could call it a complaint, is that players want the servers up longer and more often so they can keep playing Global Agenda. That is the sort of "complaint" we don't mind hearing!
