Have you felt this pain? The directive comes from above that your team bonus, maybe even your jobs, are based not on game sales, but on the number that Metacritic assigns to your game. And the directive comes from execs who were total masters at Asteroids and Combat for the Atari 2600, so yeah, they really have street cred.
I saw this years ago, and it’s still happening today. A studio I worked at introduced their “Road to 80” plan to all internal teams. This directive indicated how important it was that every game team achieved an over-80% Metacritic score. If the game scored less than 80, the team was obviously underperforming.
Hmm.
You should have seen the PowerPoint decks. Instructions from Never-Never Land on how to score over 80! You mean, all we need to do is focus on quality?! Work harder, not lazier?! Make a simpler game design to be used on a one-button joystick? Of course, silly us!
When I was briefed on this, I got in trouble. But I didn’t mind. I was taught in army leadership school that it’s a leader’s responsibility to question bad orders, and file an objection up the chain if those orders are nonsensical. Better yet, use your judgment and do the right thing if the orders put your team at unnecessary risk. Asking something along the lines of, “How can you prove a Metacritic rating results in higher unit sales, when there is no proof tying these two concepts together? Game sales are based on marketing spend anyhow.” results in the response, “Mumble-mumble-bad-attitude-meeting-over.”
Metacritic is cool. You get interesting numbers, some of which may be based on reality, that is, the user ratings (but not always, I’ll get to that later). On the professional critic numbers, does anyone with a working calculator actually give the aggregates credence? I may not have a math degree, but Metacritic math (see below) works fine only after the third bottle of vodka is clanking against the other two empties under the blackjack table at the Red Rock casino. Unless they’re using base-6 mathematics instead of base-10. Anything’s possible.
Like every gamer, I have my favourites when it comes to reviewers. I trust two or three, partly because I know a bunch of the fellas and gals that write those reviews, and they seemed pretty honest the last time we partied at DICE. They also took my questions in stride when I asked them if they ever get it wrong, to which they said, “Sure man. We’re only human. We try our best and we love games.” Fair enough.
But even then, and this isn’t knocking them by any means, there can be huge differences between their professional reviews and the user reviews. If it’s a 10-point spread, in my opinion, I dump the pro reviews in favour of the user ones, whether higher or lower. Am I right? Who knows, but it works for me.
As for tying a team’s reward scheme to Metacritic ratings:
A Metacritic number can be a fair benchmark, once you forget that it isn’t based on math, review aggregation or common sense. Even crazier, the hottest sellers over the last few years received 85%-plus ratings from the pros. User ratings were lower, in some cases, a LOT lower. Why? Is the system corr– erm…never mind. My math is definitely off.
The only thing I know for sure is this: If the rating is below 60 when I make my own average between pro and user reviews, I don’t spend the money on a game. If it’s above 90 on both, I *may* spend the money on it, if it’s my kind of game. If it’s in the 70 range, and I like the concept, sure, I’ll buy it and play it.
The reverse side is that any studio can get their community/employees to boost their user review numbers. It’s like having Bernie Madoff on your marketing team, yelling to everyone, “I’m cheating! I’m cheating!” and it’s just as obvious if the numbers don’t make sense.
Like a lot of us, I’ve worked on some great games, and I’ve worked on some dogs. One game, I recall, was the top-selling game of the year on two console platforms, and yet the player had to move around the map, sometimes for up to a half hour, trying to find something to do. Why did this game sell a bajillion copies? Two reasons: First and foremost, MASSIVE marketing dollars for TV and license spending, and second, decent gameplay when you actually found the playable part.
Metacritic for that game was only 77 for XB, and 82 for PS and PC. And yet it sold so many units, it was single-handedly responsible for everyone in the entire studio I worked at to get their bonuses. Heck, I was surprised I even got one, what with my asking uncomfortable questions of my manager. What was important was that the team was compensated financially for the nights and weekends away from their loved ones, finishing an over-scoped project that sold a ton.
In contrast, an FPS produced by the same studio, which cost as much in dev dollars but had a crappy marketing budget, scored 79 to 82 Metacritic across the board, sold almost nothing, lost money, no bonuses were awarded to the team, and most of the team was let go.
The best part of all this? Metacritic uses a secret formula to calculate ratings. Credibility for the, uh…user ratings FTW!
IMG Consulting 