Ask Massively: The moving goalposts of MMO reviews

Last week, Massively streamteamer Richie Procopio sat down to play Elder Scrolls Online at a press demo. Like most such "all-day" media events, the press demo was approximately four hours long -- four hours for Richie to see as much as he could and take notes to transform into articles worth reading.

"It must be so hard to write an MMO article based on four hours of gameplay," remarked a reader named Dularr.

It is and it isn't. Four hours is more than enough time to grok the basics. Of course, if you're expecting a detailed endgame critique after four hours, you're in for disappointment.

Consider your stock single-player RPG. It probably has classes, most of which amount to "some guy who kills things fast." It usually has one clear path to the end and a main storyline that is 20-40 hours long. It has an obvious ending and no perpetual endgame. A reviewer will probably play it to the finish line and start writing, satisfied that he's seen the vast majority of the game.

Now consider your stock themepark MMO. You've got a plethora of classes and roles, dozens of zones, wide-spanning storylines, factions, achievements, crafting, trading, guilds, raiding, PvP, and on and on. I won't even start in on sandboxes, which must be lived to be understood. The point is, there's more than any one person can do in a lifetime, so expecting a journalist whose salary you do not pay to do all that and then write about it in minute detail for every single game that comes out? This is the very definition of unrealistic expectations. No one in the world is doing that, not even our dedicated one-game experts who have every statistic and piece of lore trivia memorized. It's just unpossible.

This is a large part of the reason Massively doesn't do formal MMO reviews, though I admit the debate flares up at least once a week, if only inside my head because I know that the tl;dr crowd puts a lot of faith in golden stars and letter grades and scores made up of numbers and wouldn't it be great to cash in on that? Instead, we do first impressions pieces and previews and hands-on posts and second looks in columns like Second Wind, Choose My Adventure, The Game Archaeologist, and Rise and Shiny. We focus on what we do know and have seen because that's what seems honest.

But that won't stop people from calling our more focused articles "reviews" and holding them to an impossible review standard, even when they're not portrayed as comprehensive to begin with.

So just how long should a writer be expected to play a game before he's allowed to comment on it? Back in January, a Massively reader suggested that "basing a review on playing for only 20 hours and getting to level 15" was unacceptable. "No MMO should ever be reviewed by someone who hasn't hit level cap and done some of the endgame activities," he wrote. Tuesday, another reader commented that he "forced" himself to play around 20 hours of a game he disliked specifically to acquire "an unbiased view of what the game is" and to "have more experience talking about it" -- a noble goal, but one that counters the mythical 20-hour test. Still other readers in the Vanguard edition of our Second Wind column argued that 20 hours plus a month was insufficient for a Vanguard launch veteran to give the game a brief re-look.

My suspicion is that trying to nail down a magic number would be met with moving goalposts -- that is, anyone who disliked what we have to say about a game would simply wave away our impressions by claiming we hadn't spent enough time. What's enough time? A little bit more than whatever we spent playing, obviously, because if we'd played it enough, then we'd know how awesome/terrible it really is!

I've done the four-hour demos like Richie's, and the weird thing is that they're usually too long for what's available. For example -- and don't you dare take this out of context -- I actually got a bit bored during my WildStar hands-on a few months ago. The game itself wasn't boring (and I'm excited to play it for reals at launch), but once I'd prowled around the newbie zone, figured out my class, killed some mobs, done some quests, paged through the UI, clicked on vendors, and drooled over the map, I understood how the basic game played. I didn't want to play more of that right then because there wasn't much more to write about; I had milked it for all the journalist juice I could. That's just how writers' brains work at these events. What I really wanted to see was housing, crafting, PvP, dungeoning, exploring, and the other elements that round out a game and make it special, and consequently I got a better feel for the game beyond "newbie starting zone" fare during the mid-game systems demo videos and chats with the developers than I did actually playing the game myself.

By contrast, I know my preview with Star Wars: The Old Republic a year before launch gave me a skewed impression of the game because the parts prepared for me -- the combat flow, the graphics, the Jedi questing zone, the dialogue and story, the first teases of group roleplay -- were all things I liked then and like now. If back in 2010 BioWare had instead showed me its vision of endgame, my preview would not have been as positive. The stuff I was allowed to see wasn't enough to hold the game together for me once the game got bogged down in padding out the story with gear progression and quest grinds, but that's not something a writer can possibly know without playing a good long time -- a lot longer than four or 20 hours, frankly. Twenty hours won't get you to endgame, show you how the economy works, or give you a deep understanding of multiple classes. And rushing toward a raiding or PvP "endgame" will ensure you're unqualified to discuss crafting and exploration and housing, for that matter, all of which can be done without hitting the level cap. A good MMO "review" should probably include all of those things, and if it can't -- and it probably can't, given the time and budget constraints game journos are under! -- then it should be postponed until it does, published in pieces over a long period of time, or better yet, not be characterized as a complete review in the first place.

And that brings us back to why we don't like to use the word "review."

Be his post a preview or review, quick-skim or deep-dive, I think the only obligation a writer has is to be honest about the scope of what he saw and the context in which he saw it. From specialists, I like to see opinions on game mechanics and how the game meets promises and expectations; from generalists, I like to hear about how the game feels and fits in with the broader MMO meta. But above all else, I want to hear about video games, whether someone played 10 minutes or 10 years. Even "That Guy" who bailed on a game after half an hour offers something of value: a reminder to the industry that you can't backload your game any more than you can frontload it. After all, if a game we're "reviewing" is so boring that we don't even want to play and write about it for money, then yeah... there's a bigger problem with the game.

I'm very curious to hear what the commentariat thinks about this topic. It is relevant to my interests! How long and how deeply should a writer play before offering up his opinions to you?

What should you play? Where is the MMO industry headed? How does Massively operate? Has Lord British lost his marbles? Why is the edit button on a timer? Should "monoclegate" be hyphenated? Editor-in-Chief Bree Royce submits to your interrogations right here in Ask Massively every Thursday. Drop your questions in the comments below or ping us at ask@massively.com. Just ask!