Gears dev: Hard data, human wisdom both vital to fine-tuning games
Data analysis is growing into a huge, more readily-available tool for game developers, but Epic Games Senior Designer Jim Brown delved into his studio's past struggles with data to warn GDC Europe attendees about misreading the swaths of information appearing on their screens.
"You need to provide that context," Brown said, "which means that you need to have design skills, and in order to get the right answers, you need to ask the right questions."
One of Brown's examples was the Gears of War 3 map Trenches. When testing it pre-beta, Epic's heat maps showed players were sticking mostly to one side of the map. Brown and his team put that down to simple player preference, rather than questioning design they themselves were happy with. It was only until the beta the truth emerged. Despite there only being three maps in that testing phase players avoided Trenches like the plague, and that was because poor design was trapping players on one side of it. Once Epic solved the issue, Brown said, Trenches went on to become one of the game's most popular maps.
"Sometimes you fall in love with a design because it looks cool, maybe you like Trenches because it's your own - pointing to myself here," Brown admitted, "when in reality it might not be the greatest design. When that happens you run the risk of only seeing that data that supports your argument."
When a similar problem came up in the DLC map Academy, Epic approached it differently. This time, heat maps showed deaths bunched up in one spot, where the uber-powerful Boomshot gun was located. Epic determined teams were charging to the spot to grab the weapon, and since the area was elevated, protected, and had tight entrances, it made for an obvious camping spot. So Epic moved the Boomshot to the opposite end of the map, and sure enough the heat maps evened themselves out. Until further into testing, when the same problem rose up; deaths bunching up where the Boomshot was.
"All the data we had was telling us that anywhere we put the Boomshot, people would die," Brown explained. "And this is where our previous lesson came up: Question everything and ask why."
As it turned out, it was a completely different cause this time around. The new spot offered an advantageous line of sight that Epic didn't see in pre-decorative testing. In a basic state the area's walls were higher, but with the more detailed art and decorations the walls were lower. When that was fixed the heat map distribution evened out, even though the solution wasn't where the data seemed to lead.
Chasing data too far, Brown added, can also be an issue. As he noted, the Gears franchise has always struggled with shotgun balance, with the Gnasher shotgun sometimes responsible for as much as 90 percent of all multiplayer kills per month. Every time Epic tried to fixed it, Brown said, it just made things worse, and the player base grew frustrated. Epic continued to watch the Gnasher data over and over, but in the end it became clear the Gnasher wasn't the thing that was broken.
"There was this huge passion for how it felt to move through the maps with the Gnasher in your arsenal," Brown said, "Firing the weapon was fun, the effects and sounds were satisfying, connecting with it was empowering. Using the Gnasher was a successful and rewarding experience. From a design perspective, it was achieving every goal that we could have hoped for. It was meaningful. But it was all the other weapons that were deficient in similar areas."
Ultimately, Brown said, it's a human touch or wisdom that companies need most to really use data effectively. If Netflix followed explicitly where its data leads, Brown noted, it'd be making more Perry Mason. Instead it made House of Cards, and it's up to game designers to use data correctly to make the right decisions.
