GDC Online 2011: SWTOR's Georg Zoeller on analyzing in-game feedback
Star Wars the Old Republic is rapidly approaching its launch date, and questions constantly pop up about what the game will be like. Some longtime MMO veterans are even a bit skeptical about whether BioWare will be able to release a game of this scope in a finished state. At this year's GDC Online Principal Lead Combat Designer Georg Zoeller demonstrated some of the tools that the team uses to analyze player feedback and data, and then he explained how, through sophisticated technology, the team can make appropriate adjustments to the game as it wraps up beta testing and prepares for launch.
Zoeller began by looking first at Star Wars: Knights of the Old Republic, the single-player game. It was one of the first fully voiced games at the time, but when comparing the content to SWTOR, he said that the MMO had nine times the amount of content. That means a lot more time is needed to polish and get it ready.
He added that just one section of Tatooine has more creatures than all of Dragon Age: Origins because each class has its own story and more than 150 hours in a personal story experience that differs for each class played. It involves different content and even different planets.
Next, he looked at the seeming insurmountable task of creating and polishing content of that scope. He said the team needed lots of resources and smarter workflows, not only to create content but to keep people from butting heads. Lastly, he said, the team needed smarter tools.
He detailed what content creation involved, starting with assets, including creatures (which can take up to many years of work), static geometry, dynamic objects, special effects, visual effects, and animations. Testing has gone on for more than a year, and BioWare has gone through several iteration cycles. Each time the devs test, they gain feedback and make necessary improvements.
To generate feedback, the team uses a variety of methods, such as testing forums, screenshots, in-game bugs reports, CS tickets, and active feedback. But he cautioned that some of these are more valuable than others and must be weighed appropriately. After all the data are collected, he then looks at all of the feedback through a metrics lens. Everyone that tests helps with feedback, though, because every action in-game tells BioWare about the behavior of players and shows potential quit points, AFK points, and crash issues.
All of those data need to be analyzed, and Zoeller showed some sophisticated tools that take all the numbers and put them into perspective. One of those tools is a spatial visualization toolkit that can visualize any data point as long as it has xyz fields from any source.
He offered up an example of how the data can help: In SWTOR, death isn't as painful as in certain other games, and in fact, if you die, a medical robot can be summoned to rez you. The idea is that players want to have a second chance at a fight without a long wait.
Zoeller's example map of Tython showed player deaths, with yellow depicting solo and orange showing group deaths. Player feedback tended to be broad and not helpful, but the map showed specific areas, such as caves, where fleshraiders were killing not only soloers but also groups, so the team realized that it needed adjusting.
The data can even show how players die, illustrating how certain pull spots can generate adds and potential deaths. Seeing spots where patrolling mobs can overwhelm players when pulled from certain spots helps adjust mob spawns to make gameplay smoother.
Another way to analyze data is to build word clouds and look at chat logs in game. The phrases "how" or "how do I" come up a lot, and by filtering them out, you can get a rough idea of where those questions come up and what the most common questions are. For example, players might get confused about their first companion, and without a set tutorial, questions about them might pop up in the chat log.
Chat also shows quick validation. Bugs come up that might not be reproducible, but if the phrase bug or bugged comes up repeatedly in the same area, it's a quick confirmation that there's something wrong that needs to be fixed.
The tools the devs use also help with performance optimization. Numbers and color codes show mobile object density on the map. The same is true with terrain optimization. Areas with a lot of materials are harder to load, so the team can color-code the amount of materials, dial it down if needed, and balance it with the need to maintain the aesthetic quality of the game.
The tool BioWare uses to analyze data is in 3-D instead of 2-D because it shows lot more and can zoom in much better, making it easier to read. On the planet Ord Mantell, for example, the terrain showed death locations on a 3-D map, easily displaying exactly where players are dying.
He also showed a kill map of a warzone, and we could see clumps of deaths in the middle and on equal sides, showing that the map was well-balanced. Even small behaviors are isolated by these tools, like seeing where someone died falling off a cliff or seeing that 68% of players head to the right when moving through a symmetric room with two paths.
In the end, though, he said all the tools in the world can't help if you don't play your own game. And in some cases, just directly asking testers is easier than digging into metrics -- and might yield just as valuable information.
