How exactly do you review an MMO?

Reviewing MMOs is a tricky business, as Eurogamer learnt this week when they published a review of Darkfall and gave it 2/10. Developer Tasos Flambouras wasn't happy about this and published a scathing forum post in retaliation at the low score. They accused reviewer Ed Zitron of playing for a mere two hours (he asserts he played the game for over nine) and most of that time was -- according to Tasos -- spent screenshotting or creating characters. This prompted Eurogamer to respond and promise to do a re-review, which in turn prompted Tasos to respond to the response.

This begs the question: how do you review an MMO? It's not like any other kind of game; there's no beginning, middle and end, just a beginning and an endless middle. Added to that, while video games are without a doubt mainstream, MMOs are a lot more niche. But Lesley, I hear you cry, how can games like Guild Wars and World of Warcraft with seventeen million players between them, be called niche?

Well, MMOs take a lot more getting into. There's often a steeper learning curve that means pick up and play is impossible for the first few weeks/months. Even after that, quests and raids take time and the addictive nature means a quick five minutes to do some dailies or some quest hand-ins can still find you playing two hours later. But the thing about this niche is that you're dealing with, first and foremost, a fanbase. MMO gamers are insanely loyal to their particular title, they tell their friends and get them playing, they read the novels, buy merchandise, attend events and some even cosplay as their favourite characters.

I've been a video games journalist for four years, I've written for magazines and websites on both sides of the Atlantic, so I like to think I have an inkling of how the industry works. This process remains mysterious for most people unless they're directly involved, so here's what generally happens when a game is reviewed.

A PR company will usually approach an editor, offering access to a beta or review code for their hotly anticipated title, Game X. Because the majority of MMOs are played on computers, they are much easier to get access to because you don't need specialist equipment like a debug machine. Alternatively, an editor might approach a company and ask for access to review Game X, but either way the results are the same. Sometimes reviews are tied in with cover deals or advertising, which can make things even more complex if the game company is unhappy with a review. But that's for another day.

Now just because an editor has a beta code or an installation disc for Game X, that doesn't mean he will instantly fire it off to a writer -- that only happens when there are two days to deadline or the editor is insanely organised. Editors generally work in cycles against the monthly apocalypse known as the editorial deadline. Websites still have this, but it's much more flexible and theirs normally revolve around embargoes on game releases and making sure a review is ready the second an embargo lifts. Sometimes game reviews will be dished out according to a staffer's or freelancer's preference or knowledge. Other times, to which ever freelancer happens to be available after all of the staff have declined and is willing to accept a paltry rate of pay.

The problem with reviewing MMOs is simple: Time. There's never enough of it. Unlike other games which you can finish (and the world of video games journalism is still split between those who believe you should finish a game before writing a review and those who live in this reality and aren't given the time), MMOs are black holes. They are viewed as not really being worth the time, effort and money to review except by journalists who -- like me -- already enjoy them. Even I, who enjoys playing MMOs and writing about them, approach reviewing MMOs with this strange feeling of dread. It's almost instinctual because you know several days will be swallowed up for four pages of copy.

Most games can be played and reviewed in a day, but not so with MMOs. Sometimes a week isn't long enough and whereas with console games you can get a PDF of the manual, with MMOs you're very much left to your own devices and the in-game tutorial. Past experience and knowledge is very important and the number of MMOs you've played serve as a way of ultimately reviewing a game. Every journalist has their own bar when it comes to good and bad games. It's worth remembering that, even though journalists will sometimes receive 'guidance' from their editor on how the game should score, reviews are ultimately the opinion of one person designed to help readers decide if they should spend their hard-earned on Game X.

So the reviewer has the code, has downloaded the client, patched it, patched it some more and has logged on. Now what? Our fearful reviewer is probably watching the clock, trying to work out a) how long s/he needs to play the game to get a 'feel' for it, b) calculating how many hours until copy has to be filed and c) how much s/he is being paid per hour to do all this.

The typical time frame for an MMO review from initial phone call/email until copy filing is usually one to two weeks, if you're lucky. Add another one to two days to receive Game X in the post or for PRs/editors to email codes. Indeed I've even borrowed other journalists' active accounts in order to review expansions. Regardless, we are seldom given high level characters, meaning most of the higher level content that comes along in expansions cannot be reviewed properly.

So how do you review an MMO? Well, in some respects, in the same way as any game. You look at the gameplay, graphics, learning curve, storyline and all that jazz. However, there will always been comparisons with bigger games. Be they better or worse, each genre has its benchmark. So our reviewer plays, possibly scribbling some notes longhand or taking into a dictaphone as they go. The problem with game reviewing is that you cannot enjoy the game. I know some journalists who advocate playing a game twice, once for fun and a second time with a critic's eye. Yet again we fall back to the time factor and the lack of a conclusion. MMOs are by their very natural continuous and changing with no real ending.

But for how long do you play the game? Until you reach level 10? 30? The level cap? Obviously the latter is seldom possible and because you are one reviewer - often alone on a half populated server - finding groups or a guild is often difficult. Despite the very essence of MMOs, they are often played solo. At the same time, it is common for games to be played in spurts. Freelancers often have several reviews or features on their plates and will jump between games, they may also have lives and need to do those little things like walk the dog or visit Tesco to buy dinner which obviously get in the way of playing Game X.

As part of accepting the review, the PR reps or the company itself will sometimes ask for your email to create an account or, as happened with Darkfall, supply special accounts. Yes, these accounts can be monitored but this is the first time I've ever heard of the logs being used against a reviewer and made partially public. Of course, we don't know who is telling the truth here (and it's not a subject I will go into), but it's doubtful Aventurine will ever release them. Yet, as a journalist, I'm nervous about the idea of being watched by Big Brother and judged, almost held to ransom should the company not agree. I'm positive, had the review been a better one in terms of score, Tasos would not have gone up in arms.

The idea of being monitored Big-Brother-style gives me the creeps because I wonder if it ties back in to how long you should play a game. How long I think this should be and what a PR or developer thinks are most certainly different figures. Depending on whether the PR is internal or external often affects how close they are to a game. For some, Game X is their baby and cherished regardless of how flawed it might be. After all, devs like Tasos have worked on the game for years, and yes it's probably not perfect -- games never are. But how would you feel if a journalist had taken everything you had worked on for the last 6+ years and ripped it apart? Not happy I'll bet.

The thing is the two sides of this industry -- that of journalists and PR (known affectionately as the 'Dark Side' of the industry) -- are often at odds. Unless journalists have been PR, or more often vice versa, neither has much of an idea how the other side works or the pressures put upon them. I think perhaps if editors were more transparent about the reviews process everyone would be a lot better for it.

So our reviewer has played Game X for as long as s/he can, perhaps for a couple of days, on and off. Once the piece is written they will spend a lengthy stretch of time screenshotting or appealing to a PR for images and will then file the review. In most cases, the editor will take a look and send it to the subs. It might do the rounds with other editors for their second opinion on the score. At the end of the day, it remains one person's opinion about a game, just published under the banner of a particular publication or website.

With the chaos surrounding Darkfall, neither side has been particularly smart. Tasos' rant and Eurogamer's decision to re-review the game have opened a can of worms and at the least, damaged Zitron's reputation. The thing is, given the press surrounding their original Darkfall review, how serious will people take the second? If new reviewer Kieron Gillen gives it a similar score, will Tasos wave the logs in his face as he did with Ed Zitron? If the review is better, readers are more than likely going to question it's validity and whether the score was simply inflated a little to soothe hurt feelings. To even agree to a re-review is almost unprecidented and in my opinion was probably not the best way to proceed, yet I'm just as curious as the rest of you to find out what the 'new' score will be, against Tasos' best wishes.