• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.

Metacritic's weighting system revealed

Toma

Let me show you through these halls, my friend, where treasures of indie gaming await...
If people (and publishers) started using the unweighted Metacritic score as the basis for their purchasing decisions or crazy bonus schemes, then there's nothing to stop them using another site with identical unweighted scores.

You seemed to miss my point. With that, I was not saying they need to outright scratch that system. They should simply offer a SECOND one that is unbiased for their users to choose. What publishers to with that is out of our hands, but it would be infinitely better for the customer.

Although, in a perfect world, this wouldnt need to be "weighed" in the first place.
 

Omikaru

Member
Are people seriously outraged by this? Publications with a higher reputation probably should get a higher weighting.

I disagree with this, but assuming I did agree, do the higher weighted sites listed there necessarily have a higher reputation?

EmCeeGramr pointed out Da GameBoyz -- weighted High (1.25) -- as a prime example of how fucked up this is.
 
One more reason why it should be unweighted in the first place.

Why should it be unweighted if certain review sites generally are better than others and have more of a following with consumers?

Its like the Nate Silver election aggregate. Some polls are better than others so they have more weight.
 

DTKT

Member
Are people seriously outraged by this? Publications with a higher reputation probably should get a higher weighting.

I like the guys at GiantBomb too, but comparatively, ya, they don't have as high of a reputation. The average gamer has never heard of them. Reputation level is vital in every industry, even if it occasionally results in scandel.

This is not a golden defence of sites like IGN and GT, but they built themselves over years and have achieved mass appeal for good reasons. They do carry more weight with or without metacritic's scheme. As the smaller guys get more hits and a big following, they will escalate.

What the hell is that "reputation" you are talking about? Oh wait, you aren't talking about integrity or journalistic talent. That's kind of a skewed metric to rate games on.
 

MechaX

Member
Guys, you really should start visiting some of these highly weighted sites.

Game Boyz is really just the tip of the iceberg.
 

jtb

Banned
Why should it be unweighted if certain review sites generally are better than others?

Its like the Nate Silver election aggregate. Some polls are better than others so they have more weight.

One is a predictor/measure of election turnout (objective), one is opinion (subjective). This is a ridiculous false equivalency.
 
Compared websites like IGN very few care or even know about giant bomb


I myself don't care for their pseudo elitism. The way some people on GAF seem to worship these guys is also shocking
 

Toma

Let me show you through these halls, my friend, where treasures of indie gaming await...
Why should it be unweighted if certain review sites generally are better than others?

Its like the Nate Silver election aggregate. Some polls are better than others so they have more weight.

Because they arent "better". Those websites have more money and are bigger, but for me Giantbomb and Eurogamer are an infinitely better reviewing outlet than IGN, yet IGN counts 50% more? Thats just bullshit. Which websites you consider "better" is purely subjective, and should also be treated as such on a grand scale without weighing, so that everyone can make up his own mind. The website shouldnt make a decision for which website is "better" for me to read.
 

sarcoa

Member
What the fuck?

Firing Squad?

Digital Chumps?

Extreme Gamer.ca?

These are no name sites, and they have the highest ranking. Just look at Firing Squad. The site is a fucking abomination.

Metacritic is officially no. 2 on the worthless data collection list behind Nielsen.
Thanks?

No idea how or why we're weighted that way, if it's any consolation.
 
I say let's kick up a massive fuss and act like it's legit until Metacritic goes public. Transparency is for the best.
Do you have any examples of Giant Bomb's pseudo elitism?
They don't talk about WiiU enough.
I actually think Jeff in particular uses the line "the reality of modern game development" a bit too much.
 

Withnail

Member
The general idea of Metacritic is actually quite good. By averaging lots of subjective ratings you can approach something objective. It's kind of like the wisdom of crowds.

The problem is that Metacritic invalidates their own system by applying this extra layer of (closely guarded) subjectivity. In the end a Metacritic score is hardly any less subjective than any single review. Publishers using Metacritic scores as an absolute indicator of quality is absolute madness.
 

Thraktor

Member
You seemed to miss my point. With that, I was not saying they need to outright scratch that system. They should simply offer a SECOND one that is unbiased for their users to choose. What publishers to with that is out of our hands, but it would be infinitely better for the customer.

Although, in a perfect world, this wouldnt need to be "weighed" in the first place.

I understood your point, although I probably could have been clearer in my reply. My point was that if they give people the option between Metascore and unweighted average, and people start using unweighted average, then those people (and publishers) could drift away to other sites that also offer identical unweighted averages.

Strained analogy time: It'd be like Microsoft releasing a version of Linux. It'd be great, but they aren't going to torpedo their own product like that.
 

Toma

Let me show you through these halls, my friend, where treasures of indie gaming await...
The general idea of Metacritic is actually quite good. By averaging lots of subjective ratings you can approach something objective. It's kind of like the wisdom of crowds.

The problem is that Metacritic invalidates their own system by applying this extra layer of (closely guarded) subjectivity. In the end a Metacritic score is hardly any less subjective than any single review. Publishers using Metacritic scores as an absolute indicator of quality is absolute madness.

Especially since they just said they change the weighing from time to time. That means for the release of game A you weigh Website XYZ more, but if it comes to the release of game B, website XYZ might be rated worse and has less impact (if the score didnt appease to someone)?

Thats just all kinds of fucked up for a gaming score aggregate website.
 

jtb

Banned
The general idea of Metacritic is actually quite good. By averaging lots of subjective ratings you can approach something objective. It's kind of like the wisdom of crowds.

The problem is that Metacritic invalidates their own system by applying this extra layer of (closely guarded) subjectivity. In the end a Metacritic score is hardly any less subjective than any single review. Publishers using Metacritic scores as an absolute indicator of quality is absolute madness.

The general idea of Metacritic is terrible. Something like RT is far far more useful for gauging the quality of something (and even then, just a "good"/"bad" at best)—because everyone reviews on their own terms and at least RT recognizes that. a 5/10 to me is different from a 5/10 from IGN is different from a 5/10 from [insert another publication here]. Trying to aggregate all these different reviews, all on different reviewing scales... it's a pointless endeavor. Though I agree, adding in this layer of subjectivity only shits it up even further.
 

Toma

Let me show you through these halls, my friend, where treasures of indie gaming await...
I understood your point, although I probably could have been clearer in my reply. My point was that if they give people the option between Metascore and unweighted average, and people start using unweighted average, then those people (and publishers) could drift away to other sites that also offer identical unweighted averages.

Strained analogy time: It'd be like Microsoft releasing a version of Linux. It'd be great, but they aren't going to torpedo their own product like that.

Fair enough, though Metacritic are seriously fucking their reputation through this. Ultimately a website without MC's secret sauce is BETTER. If anything, I get the impression that people who get to know about it will use different websites (I am a fan of Gamerankings myself).

I mean, after all the shit the gaming press went through in the last few months, MC is now hoping that people WONT believe that the weighing options are paid by some (not all) websites?
 

Oersted

Member
Especially since they just said they change the weighing from time to time. That means for the release of game A you weigh Website XYZ more, but if it comes to the release of game B, website XYZ might be rating worse and then its also rated worse and has less impact?

Thats just all kinds of fucked up for a gaming score aggregate website.

Polygon reloaded.
 

Thraktor

Member
Fair enough, though Metacritic are seriously fucking their reputation through this. Ultimately a website without MC's secret sauce is BETTER. If anything, I get the impression that people who get to know about it will use different websites (I am a fan of Gamerankings myself).

I pretty much just avoid review aggregation sites completely. They're useful as a collection of links to actual reviews, but even then I simply go with my gut or with GAF's recommendations most of the time (I've actually bought three or four games based on threads you've made, as it happens).

I suppose if they included standard deviation as well as mean they might trigger a bit of statistical curiosity on my part, but that's pretty much it.

The general idea of Metacritic is actually quite good. By averaging lots of subjective ratings you can approach something objective. It's kind of like the wisdom of crowds.

The problem is that Metacritic invalidates their own system by applying this extra layer of (closely guarded) subjectivity. In the end a Metacritic score is hardly any less subjective than any single review. Publishers using Metacritic scores as an absolute indicator of quality is absolute madness.

You're assuming that there is such a thing as an objective numerical rating of how good a game is. I don't see any reason to believe there is.

That's not to say I have any problem with Metacritic, I just don't feel that what they do (even if they dropped weightings) could be described as objective. Objectivity isn't just subjectivity in aggregate.
 

Toma

Let me show you through these halls, my friend, where treasures of indie gaming await...
I pretty much just avoid review aggregation sites completely. They're useful as a collection of links to actual reviews, but even then I simply go with my gut or with GAF's recommendations most of the time (I've actually bought three or four games based on threads you've made, as it happens).

I suppose if they included standard deviation as well as mean they might trigger a bit of statistical curiosity on my part, but that's pretty much it.

I use these sites for different things. I am not basing my buying decisions on Gamerankings, but I use it to get a general overview of the reception of a game. Not sure whether I articulate properly what I mean by that.
 

kswiston

Member
Probably mentioned 10x already by page 4, but based on the Rpgamer vs Rpg fan weightings, I would guess that sites that give consistently high rankings are ranked higher (or perhaps games that gives scores closer to the metacritic average, which tend to be high). Pretty much anything under 80% is a bad score on RPG Fan. Rpgamer uses more of the scale and gives 2/5s pretty often.
 

TheSeks

Blinded by the luminous glory that is David Bowie's physical manifestation.
That entire list ranking is a joke. GiantBomb being low? No Rockpapershotgun? COME ON SON.
 

Acheron

Banned
Should probably just assign an arbitrary rating to IGN (the most popular one) and weigh everything else relative to it via Alexa rating.

That or do as Nate Silver and readjust weights based on how often you deviate from consensus.
 

Thraktor

Member
I use these sites for different things. I am not basing my buying decisions on Gamerankings, but I use it to get a general overview of the reception of a game. Not sure whether I articulate properly what I mean by that.

Yeah, I get what you're saying, and I actually do have a bit more interest in it in that respect, but I suppose that's more motivated on my part by an interest in the gaming press than an interest in the games themselves.
 

Toma

Let me show you through these halls, my friend, where treasures of indie gaming await...
"That list is totally inaccurate, but we won't release actual data, so take us at our word."

"The people who made 6 months of research with clear distinctive patterns for these categories are totally wrong!"
 
"That list is totally inaccurate, but we won't release actual data, so take us at our word."

Translated:

"We don't want people to stop taking all of our data seriously, so we'll just make up excuses and say their mature and thoroughly analysed research is all fake!!!"
 
It must have to do with some statistic, like based on a standard deviation of where an average review lies from that publication based on the overall true aggregate average review. I can't see how any of these sites would benefit or not from their reviews being weighted more heavily or lightly.
 
I wonder how big of a deal this is in practical terms for the people who are ranked low. I mean does a low weighting somehow translate to monetary damages for the reviewer in question?
 

Toma

Let me show you through these halls, my friend, where treasures of indie gaming await...
I wonder how big of a deal this is in practical terms for the people who are ranked low. I mean does a low weighting somehow translate to monetary damages for the reviewer in question?

Probably not, but it destroys the integrity of the website for the user. Why would a website rating a game 9 be weighted more than another one weighting it a 7? What if I agree with the website that rates it 7, but MC still decides that its better for me to trust the other website more?
 

Varth

Member
I'd really like to have an official statement about sites/mags getting in MC panel via publishers suggestion. Does MC ever get "hints" from publishers about including some website?

You know, when people's bonus is determined by the metacritic vote, inflating the score is not really that much of a danger, since in the end, score is not that important to sales. To me, it seems far more dangerous that a publisher in danger of having to pay a huge bonus could "push" for the inclusion of a vote that brings the average low enough to avoid paying bonus to dev teams, so I guess - and that's not the first time I say that, that all the process could use *a little more* transparency than the way it is now.

NOTE: to be completely trasparent on that, some time ago, when we had both the best selling mag around here and a pretty high ranking website running, I applied for inclusion, as I thought it would have been a good showcase for our work, but we were refused for reasons that - if applied to all - would basically mean cropping 9/10 of the list.
 
Probably not, but it destroys the integrity of the website for the user. Why would a website rating a game 9 be weighted more than another one weighting it a 7? What if I agree with the website that rates it 7, but MC still decides that its better for me to trust the other website more?

Integrity of Metacritic or the review site?

If you mean of the review site, then if it hadn't been for this megaton, nobody would have ever known what the weights were, so there would have been no effect on anyone's integrity. Maybe this changes things? Or maybe people only a few in-the-know people, such as GAFfers and people in the industry, will know about the weightings and not care. The average person still won't know
 

Toma

Let me show you through these halls, my friend, where treasures of indie gaming await...
Integrity of Metacritic or the review site?

If you mean of the review site, then if it hadn't been for this megaton, nobody would have ever known what the weights were, so there would have been no effect on anyone's integrity. Maybe this changes things? Or maybe people only a few in-the-know people, such as GAFfers and people in the industry, will know about the weightings and not care. The average person still won't know

Integrity of MC.

Admittedly, WE arent the average person, but just pushing that to the frontpage of GAF changes MC's perception in the eyes of some people. It always starts small, and I actually hope some other gaming websites pick up on it and inform their readers accordingly.
 

DEADEVIL

Member
Wow.

This is literally the Smoking gun in regards to the Metacritic process.

If this doesn't ruffle some feathers I don't know what else will.

Popping the popcorn now.
 
Nobody has any idea how the weightings are given so there's no point raging about it. Could be part reputation, hits, visitor demographic (age, country). Could be pseudo-random.

Who says this list is official anyways? I might have missed it when reading through on my phone. I have a hard time believing they just made it public.
 

hammster

Archbishop of Canterburny
Heh. I emailed Metacritic about this just to say we'd appreciate transparency and whatever (because that's what I do, I email people). They literally just responded with a "thanks for your opinion on the matter" and that was it. Responded pretty quickly though. Suggest more of you do just so I'm not a lonely man shouting at (read: emailing politely at) a cloud (read: a probably disgruntled editorial team).

Contact us page: http://www.metacritic.com/contact-us

Editorial Inquiries: editorial@metacritic.com
User Support: support@metacritic.com
 
I don't know why they would do it this way. Wouldn't it be better to lower the importance of outliers? Sometimes you get a game getting all 7s and 7.5s and suddenly you have a 2 thrown in there or a perfect 10. At least then you don't have people at meta critic deciding which sites deserve more weight.
 
Top Bottom