Support NeoGAF

[Ravidrath]

Metacritic is a huge problem, in my opinion.

While I grudgingly admit it has lead to an increased attention to quality in the industry, so many aspects of game production are geared around gaming the Metacritic system.

PR people pressure publications for certain scores, and certain scoring methods in order to make all reviews have less impact on their Metacritic score. Developers have Metacritic clauses in their contracts for bonuses, but often times aren't given the time to actually make a good game.

The focus on Metacritic, I believe, is also a big part of the reason why publishers are focusing on AAA titles, and those AAA titles get samier and samier each year. Metacritic has defined very clearly what reviewers like and don't, and they're making games targeting those things. And because AAA titles are what sell, most game retailers don't really want to even stock mid-range games any more - just AAA and budget titles.


So, yes, while the intention was good and it has had some good effects, the fact is that it has become a mindless, uninformed shortcut to help executives factor game quality into their calculations. Them ignorantly charging forward, armed only with a single number, has ultimately only exposed all the more how little they understand the production of games, and skewed the industry around it.

...And all because people are too lazy to do real research.

 

[godofearthx]

A journalism website is always going to get picked apart for many things by different kinds of people. Giving the consumer a portal that can allow members to find out why it is rated mediocre, low, or high highly improves purchasing decisions. Just go to Amazon.com and you'll pretty much find fake reviews, accurate reviews, and perhaps reviews that are extremely ridiculous. The gaming industry has much more problems than the perception of the media. If you release great games then that is totally fine.

 

[Incendiary]

It's interesting that the games industry puts so much emphasis on metacritic, when it also aggregates TV, movie, book, and music reviews as well. But as far as I can tell, nobody from those industries give a shit about metacritic. When's the last time you went on metacritic to determine if you wanted a book? Or a CD? And I don't think any movie moguls will say "Well, our movie got an 89 on metacritic, so you should go see it!"

Is it because gaming was born at the same time and is heavily connected with the internet? While the other mediums are still catching up in that aspect? It's an interesting question.

 

[Ushojax]

It isn't helping but it's more the fault of publishers who take the word of their own advertising stooges as a good indicator of a games reception. Heavy Rain has 87 on there but the game is absolutely reviled by many who've played it (myself included). It's a very gaming-specific problem, the relationships between gaming-specific media outlets and publishers is far closer than in the rest of entertainment.

 

[SolidSnakex]

I remember Gamespot put a 95, but then quickly changed to 10. Those years were really controversial, also the Kane&Lynch review getting someone fired.

This is the story behind the GTA4 reviews

In the weeks prior to GTA IV’s release, Rockstar made promises that print and online publications would receive early review code so that they might fully ingest and digest Liberty City in order to deliver mature and balanced opinions on its day of launch.

In reality, this was not the case, with precious few publications getting to spend prolonged time with the game ahead of release. The first review of the game came from the UK’s Official Xbox magazine bearing the worrying caveat “based on unfinished code”.

Eurogamer, wise to the fact promises of AAA title retail code ‘a week before release’ are rarely upheld, arranged to play through the game over a period of days in Rockstar’s offices instead (along with a couple of other UK publications). From speaking to other editors (some of high profile titles) this was not an opportunity offered to all and, when review code failed to turn up the week before release, many were left panicking about how they were going to serve their readers in a timely manner with any integrity.

The reason for the withholding of review code was, according to Rockstar, a result to the game’s leaking onto the internet seven days before its release. Speaking to the company at the time it was claimed that this leak came from an unscrupulous journalist.

As a result, there was a lock down on all review code: everybody would get their copy just one day before the game’s release, and, despite the wonky logic (after all the game had already leaked to those with the capability to play it so why punish the many for the indiscretion of the few) there were to be “no exceptions, no arguments”.

At best then, by the time the game had been played, copy written and subbed ready for the Tuesday morning, most journalists (both in the UK and the US) had played for only a few hours, experiencing just a fraction of the game’s content, a situation testified to by various admissions in professional reviews.

Time Magazine dubbed their piece Grand Theft Auto IV: The 6.24% Review while the Associated Press reviewer, Lou Kesten, admitted to having spent only spent eight hours with the game.

Slate Magazine’s excellent Chris Baker admitted he only had chance to ‘scratch the surface of the game’ going on to say in a comment on N'Gai Croal’s Level Up blog: "I couldn't even attempt to be definitive…it was kinda liberating”.

The BBC noted the phenomenon saying: "Most reviewers were not sent advance copies of the game, and instead had to attend Rockstar offices or sit in booked hotel rooms to play the game,” where Rockstar could keep an eye and some pressure on them. While these few admitted the partial and necessarily subjective nature of their reviews, how many passed off their impressions as being definitive of the whole?

http://www.gamasutra.com/php-bin/news_index.php?story=18761

You have to wonder how often this actually goes on.

 

[whalleywhat]

It's necessary because it allows you to scope out different point of views easier.And if i had that mindset i would of never bought Demon's Souls an AAA game in my eyes with no marketing.

I'm actually shocked that Demon's Souls has a high metascore. But are you seriously saying you wouldn't have heard about it without Metacritic? GAF had a huge import thread long before it was released in the West, and it had built some notoriety off its more punishing mechanics.
As for linking a bunch of reviews, I think that's a good thing, it's just the score aggregation which gets on my nerves.

 

[Doctor Decimate]

the reliance on Metacritic is a result of the industry's current path, not the cause of it.

 

[Taruranto]

Maybe if Obsidian didn't make such a buggy game they would have got their bonus?

A performance related bonus, who'd have thought.

It should be surprising they managed to get the game out to beginning with, considering how awful the Gamebryo is. Obsidian also tends to get fucked by publishers and deadlines a lot.

Anyway, when a game like Skyrim gets rated 11 on 10 on every site, it's quite clear that performance problems have little to do with scores.

 

[Sapphire Dreams]

Next article:

"What Metacritic can learn from IGN."

 

[The Faceless Master]

Metacritic isn't the problem, Publishers are the problem.

 

[hyduK]

This is the story behind the GTA4 reviews



http://www.gamasutra.com/php-bin/news_index.php?story=18761

You have to wonder how often this actually goes on.

Eh, at the end of the day I say the problem is more in the < 80 range, not the 80+ range. Sure, games like GTA might deserve to be a few points lower. But most games < 80 deserve to be 20-30 points lower if we want a properly utilized 1-10 scale..

 

[Zeliard]

I've been banging on this drum for a while now, including things like this:

This doesn’t apply to all publishers and all contracts, of course. But it shouldn’t apply to any. It also creates a very weird problem for games critics, who suddenly find themselves potentially responsible for a developer’s livelihood – as commenters who feel you have underscored a game will sometimes point out. This absolutely should not be a consideration in the critical process. Any reviewer who ups a score out of sympathy for the developer is not doing their job properly, and is not doing their readers justice.

Problems which most people don't even seem to consider. I said this a while back:

Frankly there is probably already enough pressure on reviewers when you take into account Metacritic-based bonuses. Only in the gaming industry is a press review aggregate tied directly to people potentially losing their jobs or even a studio being shut down. So when they are giving these games low scores if they feel they deserve them, I'm not sure how that's something to also criticize.

When a reviewer knows their review can have an impact on the lives of others, how can anyone expect them to give out a score that truly conveys how they feel? And that isn't even looking at the impact marketing has on reviews. The score is ultimately still taken into account in the aggregate, and simply worsens the situation. Along with Metacritic's highly dubious averaging out of scores that use entirely different scales, you have reviewers scoring games based on completely external factors, so any attempt at some "accurate" aggregate is already a lost cause.

Yet publishers still use something so flimsy as an all-important metric, when sales should be the most important thing, as it works with every other industry. You figure if they only truly care about Metacritic due to its correlation with sales, then they should cut out the middle man and simply look directly at sales, but I suppose that makes too much sense. The notion that publishers look at Metacritic to gauge a game's objective quality is laughable for a number of reasons.

 

[teruterubozu]

If it's not Metacritic it'll be something else. It's like crying about the weather.

 

[Canis lupus]

Eh, at the end of the day I say the problem is more in the < 80 range, not the 80+ range. Sure, games like GTA might deserve to be a few points lower. But most games < 80 deserve to be 20-30 points lower if we want a properly utilized 1-10 scale..

If we were to do that then GTA4 should be rated an 8.

 

[Loxley]

I feel like this article has been written five or six times over the last few years.

 

[Sky Chief]

Metacritic isn't the problem, Publishers are the problem.

Pretty much

 

[MintLemonade]

I would say IGN's wiki articles are more of an insult to the industry than Metacritic.

 

[Canis lupus]

This is the story behind the GTA4 reviews



http://www.gamasutra.com/php-bin/news_index.php?story=18761

You have to wonder how often this actually goes on.

Horrific. Bashing GTA4 was taboo back then.

 

[HamPster PamPster]

I don't get it. Metacritic just aggregates scores, anyone could do that by themselves if they wanted, Metacritic just makes it easier.

If there is a flaw with the system its the data going into they system (individual scores) not the average being spit out

 

[Thoraxes]

Well there is truth to it, publishers attaching payments to developers based on metacritic scores is complete shit. If a game makes money why the fuck does it matter if it only got an 83 metacritic score?

Some devs are under contract to produce games with a specific metacritic threshold in mind, and if they fail, they lose potential money.

 

[clav]

 

[tkscz]

The slightest five-point drop in that average score as the result of maybe 10 small-time blog reviewers questioning an adjustment or risky new element in a game can have disastrous consequences for the people who actually made it.

The first thing metacritic needs to do is stop taking in opinions of small time bloggers. Or at least have two separate scores for them (one for professionals and one for fans/bloggers).

Next, if other sites have an issue with Metacritic, then those other sites need to tell metacritic not to include their scores.

 

[Tellaerin]

Using Metacritic rankings as a measurement of overall quality may be a flawed metric, but it's better than having no metric at all. Setting some kind of bar for developers to strive for is a good thing, IMO, and sales figures aren't always an indicator of how good the product actually is.

Rather than just throwing out the idea of tying incentives to Metacritic scores or something similar, I'd like to see publishers work on refining the model somehow.

 

[2+2=5]

Too easy.
Who blames metacritic doesn't see(or doesn't want that others see) the real problem, metacritic is only a database, sites fill it with their reviews, games with beautiful graphics and no gameplay have better scores than games with more focus on gameplay? Some scores are crazy? Scores are too high? Don't blame metacritic, blame the reviewers.

 

[Rebel Leader]

I can't remember when I last used metacritic to base my purchase

 

[PatzCU]

Does anyone even use review scores as a basis to purchase a game anymore? I stopped using reviews about 5 years ago and haven't looked back.

 

[sueil]

Really most AAA games I wouldn't even put as high as an 8 on a proper scale. They tend to be the most sterile kind of committee made junk that is hated in every other medium. Just because something has good production values doesn't make it a good game. Space invaders is a 10/10 games. Assassins Creed is average at best. Games that don't try to do anything new and just retread the same formula endlessly are worthy of only contempt. When a game does something new and unique that is when it becomes truly noteworthy.

 

[Daydream Sequence]

An interesting article from a site that cuts deal after deal with publishers to ensure that their review score is posted to Metacritic first. IGN, you're the problem. Part of it, anyway.

 

[Doctor Decimate]

Does anyone even use review scores as a basis to purchase a game anymore? I stopped using reviews about 5 years ago and haven't looked back.

i find it useful to be informed about a possible purchase, but that's just me.

 

[Tagg9]

Ideally, reviewers should have to look back at the game six months on and see if they would have given it the same score. I can guarantee that most would change their score substantially. I'm looking at titles like AssCreed1 and GTA4.

 

[toadster101]

IGN is ruining the games industry. Their lack of journalistic integrity would make Fox News and MSNBC look proud.

 

[apana]

For most of us here Metacritic is useful because we also have other sources of information from which to base our purchasing decisions. If you are just using metacritic and looking purely at the number then it can be deceptive.

 

[CamHostage]

Also, I would love for IGN to wrestle up the balls, take a stand against the site and be delisted if they're going to run pieces like this questioning Metacritic's existence. However, we know that would never happen......so they should just shut the fuck up and continue covering the top ten female _______ in games.

You don't request to be listed by Metacritic or any other review aggregate, the site simply adds content from user requests (although you as a webmaster could be one of those users) and editorial choices. Back when GameRankings used a URL redirect with that header of theirs, lots of sites in fact did send cease-and-desist requests to the service because they were taking the traffic numbers while using somebody else's content and bandwidth. (Oh you, internet...) I don't know what would happen if you asked to be removed from MetaCritic now?

Sites don't seem to depend on it much for traffic anyway. I know smaller sites liked getting listed on RottenTomatoes or GR back in the day because it gave a chance of people discovering their site, but I've not heard in ages any site (major or indie) brag about strong MetaCritic referrals (any fansites here have a differing experience?) Personally, I don't even know too many people who use MetaCritic, I hear it talked about a lot but at most I see a quick pop-over to the % as part of a thread and not a lot of discussion of everything else on the site. (As opposed to say right now RT has one "rotten" review for Dark Knight and everybody is freaking out that its Fresh Meter is 96%, I've never seen that for a MetaReview.) Companies care, but if it could ever be measured somehow, I'd be willing to wager a few bucks that Amazon/GameStop user reviews move or hold back more product than MetaScores. If Google ever gets an aggregate review scorebox going (I know Bing and Yahoo have had boxes on and off, some powered by sites like MetaCritic,) I'm not sure how many would take notice of what would happen to MetaCritic.

 

[Rolf NB]

The game reviews industry needs a good ruining.

 

[ChaosInTheRain]

I don't get all the people who are replying here with "I don't use Metacritic so it doesn't affect me". If you read the article, and I can personally attest to this as one of my best friends is a dev, Metacritic scores are dictating to an unreasonable extent what games are being made, how they're being made and to an extent the success or failure of some smaller dev teams. It's a tool that publishers are using as leverage over developers in terms of what they're allowed develop and whether they get paid bonuses or not.

It's the classic head in the sand response to say Metacritic doesn't influence the games industry cos you don't look at it.

 

[les papillons sexuels]

publishers should create review embargos on games for the first week of the games release.

I've had friends tell me they hate certain games but bought it because reviews said it was good, mass effect 3 and uncharted 3 are the biggest offenders in this category, at least between my friends.

 

[linko9]

I agree that it's disgusting that people's bonuses can depend on a Metacritic rating, but other than that, I think it's a useful site, as long as you know how to use it properly.

 

[hyduK]

publishers should create review embargos on games for the first week of the games release.

I've had friends tell me they hate certain games but bought it because reviews said it was good, mass effect 3 and uncharted 3 are the biggest offenders in this category, at least between my friends.

That seems to have more to do with your friends. I doubt those games would have reviewed much lower if they had waited a week. Sounds like your friends just aren't into those genres/settings/etc...they should probably consider what type of game it is instead of just looking at the number.

 

[SerRodrik]

Ideally, reviewers should have to look back at the game six months on and see if they would have given it the same score. I can guarantee that most would change their score substantially. I'm looking at titles like AssCreed1 and GTA4.

I'd say Dragon Age 2 is a prime offender. Even I got swept up in the craze at first, but the farther out I got from release the less I liked it, and I get the same impression from journalists.

 

[les papillons sexuels]

That seems to have more to do with your friends. I doubt those games would have reviewed much lower if they had waited a week. Sounds like your friends just aren't into those genres/settings/etc...they should probably consider what type of game it is instead of just looking at the number.

no, they love blockbuster films like indiana jones and star wars so I don't think that applies. And they play tf2, cod and halo often enough so I dont really see how that's at fault.

If anything i think they just bought into the hype then got disappointed, much like i did with mario galaxy, a game that definetly does not deserve anything above a 7.

 

[ciridesu]

Metacritic wouldn't be an issue if reviewers did a sincere job, I'm looking at you IGN.

What Metacritic does is promote the real issue in the game industry, review scores. This is not a recent phenomenon and can be tracked far back with uneven scoring depending on different factors. With the arrival of Metacritic, the review scores' importance is stronger as information flow to the consumer has grown, which directly raises their significance. However, it could be truly great for ambitious newcomers to garner attention with high review scores, if only reviewers did not tend to look for flaws rather than plusses in some of these cases.

So yes, currently Metacritic is damaging to the industry in the way of 'safe over different', yet this wouldn't be an issue if the much of the games media didn't suck.

 

[megalowho]

As an educated consumer I rarely lean on Metacritic to tell me what's good, but I won't lie - if I see a random but mildly interesting Steam game with a sub-70 Metacritic score, it gives me pretty big pause. I don't often click through and read individual opinions, I just move on.

I don't see a problem in trusting that site to work like any review aggregator would, telling me at a quick glance the general consensus of a product. Situations where developers must reach a certain Metacritic score in order to be compensated fairly are pretty disheartening though, especially if the publisher isn't willing to go the extra mile for said developer so they can be in the best possible position to achieve that goal. Same goes for marketers - the pressure to raise that overall number by a point or two seems unreasonable and disproportionate from the stories I read.

Also, 20-point and 99-point review scales are pretty much broken, but that's nothing new and not entirely Metacritic's fault. If publications really cared about affecting the way Metacritic operates, they'd implement more reasonable and relatable scoring methods across the board.

 

[hyduK]

no, they love blockbuster films like indiana jones and star wars so I don't think that applies. And they play tf2, cod and halo often enough so I dont really see how that's at fault.

If anything i think they just bought into the hype then got disappointed, much like i did with mario galaxy, a game that definetly does not deserve anything above a 7.

I think you'll find that you're in the vast minority if you think Mario Galaxy, ME3, or UC3 deserve anywhere near a 7.

 

[Rolf NB]

Metacritic is awesome because it takes a lot more money to game the whole lot than it takes to game an individual site, like IGN.

Metacritic is also garbage because they don't quantize incoming scores to the for:against scale before further processing.

 

[ElFly]

I think the most damaging practice is tying developer bonuses to metacritic scores.

Most prominent example is Obsidian being denied a bonus cause New Vegas didn't get a 85 in MC score.

 

[PK Gaming]

Nice article.

Can't say I disagree, and there's definitely something wrong when mediocre games like ME3 sell like crazy because of it's "high metacritic score" and legitimately good to decent games are passed over because "itdidn'tgetan80orabovesoitsucks!"

Annoying phenomenon.

I think the most damaging practice is tying developer bonuses to metacritic scores.

Most prominent example is Obsidian being denied a bonus cause New Vegas didn't get a 85 in MC score.

 

[Brera]

I get all my news and views from gaf plus YouTube vids.

Not been on IGN or anywhere for a review in years and don't intend to.

Didn't even know meta critic was a site, I thought it was just a tool for companies!

Eitherway, I'd rather listen to people who actually pay for their games.

 

[JordanN]

Metacritic is an odd one. It can be used as a force for good or bad.

I believe I've seen employers that look for on your resume if the games you worked on scored higher than 70%.

So it kinda adds pressure to you no matter how good your programming or modeling skills are.

 

[Kai Dracon]

As an educated consumer I rarely lean on Metacritic to tell me what's good, but I won't lie - if I see a random but mildly interesting Steam game with a sub-70 Metacritic score, it gives me pretty big pause. I don't often click through and read individual opinions, I just move on.

I don't see a problem in trusting that site to work like any review aggregator would, telling me at a quick glance the general consensus of a product.

Let me just say one thing here - you're not seeing an 'honest' aggregate. That's one of the biggest problems with Metacritic.

Metacritic wouldn't be an issue if reviewers did a sincere job, I'm looking at you IGN.

What Metacritic does is promote the real issue in the game industry, review scores. This is not a recent phenomenon and can be tracked far back with uneven scoring depending on different factors. With the arrival of Metacritic, the review scores' importance is stronger as information flow to the consumer has grown, which directly raises their significance. However, it could be truly great for ambitious newcomers to garner attention with high review scores, if only reviewers did not tend to look for flaws rather than plusses in some of these cases.

So yes, currently Metacritic is damaging to the industry in the way of 'safe over different', yet this wouldn't be an issue if the much of the games media didn't suck.

I largely agree with this perspective.

The problem with review scores in the first place is attempting to pin a number on an opinion. At a certain level, it's complete absurdity. Entertainment is heavily subjective; one man's joke is another man's bad taste. While stars, scores, numbers are used as a shortcut and personal yardstick by individual critics, when the entire thing is approached thoughtlessly by others - such as the audience of critics - problems arise.

In a perfect world, relying on an aggregate such as Metacritic wouldn't be a problem. We're never going to live in a perfect world. Critics are people, and people are swayed by emotion, mood, trends, and yes, moneyhats. By politics, by pressure. Gaming criticism is in a generally poor place due a multitude of factors.

As a result, you cannot rely on mainstream critics, in general, to paint a good picture of a given game. It's a crapshoot whether you get a good review, or one that is tilted and warped, or suffers from the classic problem of "editor gives JRPG to the jockular sports guy who only plays Madden, what could possibly go wrong?"

There are also differences among kinds of aggregation. Metacritic uses a weighted 'voodoo' system of tabulation where in addition to trying to average mathematical scores, not all critics are weighted equally. Some are more important than others. This makes a Metacritic 'average' even more warped and misleading.

A simpler aggregate of 'thumbs up' and 'thumbs down' reviews, a bit like Rotten Tomatoes, might actually be more useful and reinforce that these are only the aggregate opinions of a random sampling of people. You may get 100 people in a room who don't like a certain thing, but that doesn't mean there aren't 1000 outside the room who disagree.

With that kind of system, at least, people tend to be more savvy. It doesn't matter if a particular movie got a 34% on Rotten Tomatoes due to that film going against the current trends and tastes of a typical mainstream critic. Potential viewers might go into it with a bit of caution, but they can understand that sometimes a good film (or one they might like) gets shafted by mere opinion.

With gaming, the gaming audience itself still seems to look at this stuff with extreme simplicity. "I only buy games that get 9.0 because that means they're good!" is pretty naive, especially considering how much bloated "too big to fail" titles get propped up by the industry these days.

 

[SatelliteOfLove]

Sounds to me like this is also a huge problem with the suits in the gaming industry, but I guess metacritic "ruining" in the industry gets more hits. It's not like metacritic is pointing a gun to their heads. The industry is doing a pretty good job of messing itself up these days.

This is my take. You just cannot trust some people, especially brass in the boardroom, to make helpful decisions with metrics. You have no clue what'll come out of their heads, especially with so much raw data streaming out now.

I don't see why metacritic is to blame, score inflation from critics is.It's what caused the "8 and below is shit" in the video game world and it's here to stay forever.

Both hands have been washing each other for years now.