They looked first at commercial solutions for game networking, and it turns out that the Quake 3 engine that they eventually used fit their ideas well. The toughest obstacle was of course the lag -- in a fast-paced game like Eliminate, even a delay of 200 milliseconds is too much. But it turns out that the way the Quake 3 engine handled dialup Internet back in the day is very similar to the way many developers are handling the slower speeds of mobile 3G. As the devs said, "a dialup connection from 1999 looks a lot like a 3G connection today."
In addition to the networking code, the Quake engine also gave them lots of other benefits during development, including graphics and lighting engines, a map editor, and an easy way to model animations in the game. The engineers said that using a commercial engine like Quake 3 allowed them to spend much more time on the out-of-game experience (the lobbies, the in-app purchases, and so on), and they really appreciated that.
The next big hurdles were player management and matchmaking. After considering a few different options, they went with an open-source messaging server called ejabberd
-- while it's written in Erlang
, a language that they said had some "crazy syntax" (they showed an inexplicable piece of code on the projection screen to make their point), it scaled very well and clustered the way they wanted it to.
Matchmaking was a little tougher -- they used console games by companies like Microsoft, Infinity Ward, and Blizzard as a model, and decided that they wanted to have players wait a max of about 10% of the time they spent playing. For console games, that turns out to be about a minute of waiting to make a game versus ten minutes of actual gameplay, but for Eliminate's shorter three minute games, Ngmoco decided they only had about fifteen seconds to make a match. Still, they were able to put a pretty robust system in the game even in that short time -- they assign players a number of various qualities (character skill, level, ping time, and so on), and then the matchmaking system searches for other game players, slowly expanding the limits on the search criteria.
In other words, when players first start searching, they'll be matched up with players of approximately the same skill, but as time goes on, that skill window grows. Fifteen seconds in, the skill differential could be up to ten times what it was when the search first started. Not all qualities "degrade" the same -- party size, for example, degrades much slower, so someone looking for four players won't get hooked up with just two or three for a while. And while the devs originally didn't include character level in matchmaking at all (they figured skill was a better match for players than actual level), a "HUGE outcry" by players made them include level in the process. Players really didn't like being connected with opponents who were at a much higher character level, even if the skill level was the same.
Ngmoco runs 16 different servers for each implementation of Eliminate: four for messaging with the clients, two for matchmaking, eight for what they call "game managers" (which are servers that run multiple game instances), and two management consoles that oversee the actual Ubuntu-based servers they're running, and update the 24 apt-get packages that make up the actual game software. Messaging servers are based in San Francisco with the company, but game servers are co-located around the world, in Chicago, Virginia, Amsterdam and Tokyo. Unfortunately, they didn't mention how many people are actually playing, but the servers were tested for up to 30,000 users just for messaging and 50,000 for matchmaking -- Ngmoco actually made a headless version of the game for OS X, installed it on "all of the hardware" in their offices, and ran it like crazy to load-test their software.
It was a pretty interesting talk -- very much on the technical side, but Ngmoco set out to create a competitive online FPS on the iPhone and that's what they did. It was cool to hear some behind-the-scenes details on how a very complicated iPhone gaming network is designed and run.
We'll have more from GDC 2010 all this week, including hands-on of the latest games from Ngmoco and lots of other game developers. Stay tuned!