-
Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.
Metacritic's weighting system revealed
- Thread starter RotBot
- Start date
michaelius
Banned
I'm suspecting that the weight is judged by how close your site is to the average score.
If site is giving scores closest to the end rating game receives it could be valued higher.
Importance based on what websites will play ball confirmed.
The Mana Legend
Banned
$$$$$$$$$$$$$$$$$$$$$$
Most likely Sony/Microsoft pay to have their official magazine reviews matter more. It's bullshit, and Official Playstation/Xbox/Nintendo magazines need to be ignored completely.
I don't know whether these are qualitative rankings or not (it doesn't seem like it), but if you were to try to do it in a statistically unbiased manner, it would basically be a matter of weighting the reviews by how much statistical information they provide. As an example, they could be weighted based on the following three criteria:
- Total number of reviews from publication
- Granularity of scores
- Distribution of scores (ie how large a standard deviation their score distribution has)
Publications with lots of reviews, very granular scores and a wide range of scores (relative to that granularity) will, statistically speaking, give you more precise information to base your metascore on than publications that have reviewed only a handful of games, operate on a 5-point scoring system and have very tightly clustered scores. It would therefore make sense to weight the former more heavily than the latter, which might be what they're doing.
Another possibility is that they model the predictive value each publication's scores have (ie how accurately that publication's scores predict every other publications' scores). Basically, this would mean that the publications with scores closest to the norm would be weighted more highly.
- Total number of reviews from publication
- Granularity of scores
- Distribution of scores (ie how large a standard deviation their score distribution has)
Publications with lots of reviews, very granular scores and a wide range of scores (relative to that granularity) will, statistically speaking, give you more precise information to base your metascore on than publications that have reviewed only a handful of games, operate on a 5-point scoring system and have very tightly clustered scores. It would therefore make sense to weight the former more heavily than the latter, which might be what they're doing.
Another possibility is that they model the predictive value each publication's scores have (ie how accurately that publication's scores predict every other publications' scores). Basically, this would mean that the publications with scores closest to the norm would be weighted more highly.
Yeah, a critic can always affect someone's job, but it shouldn't be so DIRECT. If a game sells lower because of that review and they fail to meet sales based bonuses, fine, but it seems kinda crap to do it based on those scoring averages as that can put pressure on some people to score a bit higher just to protect someone's livelihood.
XANDER CAGE
Member
UberTag
Member
Seeing how this is seemingly an arbitrary weighting decision, I have to wonder if certain sites with low scoring weights could file antitrust suits against Metacritic if their lack of influence cost them lucrative ad revenue opportunities with various publishers.
Of course, that would open up the whole Pandora's box of reviews being bought and paid for.
ItWasMeantToBe19
Banned
It's basically the same. 84 for the 360 and PC versions on Metacritic (with skewing), 83.82 for the 360 and 83.60 for the PC on GameRankings (unskewed)
It hasn't affected mine... On a one-dimensional opinion scale where the minimum is zero. On a scale that allows negative numbers, my opinion of them can now only be written with up-arrow notation.
Stick that in your weighted pipe and smoke it, MetaLaughingtstock.
fluffydelusions
Member
We need a gafcritic.com
Couldn't agree more, those things are ridiculous. Nobody should be taken them seriously.
whatsinaname
Member
A few posts on the first page seem to be along the lines of "Pubs now know who to bribe" or "xyz magazine will now be rolling in money".
br0ken_shad0w
Member
Would adopting RT's system of like/dislike be better as a whole? Review sites can keep their grading scale and it's certainly better than averaging all the scores and in this case aggregating them too.
I never followed metacritic and gamerankings has sucked since being bought out by CNET.
I never followed metacritic and gamerankings has sucked since being bought out by CNET.
Riposte
Member
IDK, I laughed so maybe I would call it a comedy.
XANDER CAGE
Member
That or not weighting things so crazily, yeah.
Messofanego
Banned
This effectively destroys all the stupid but incessant conspiracy theories that Tom Chick purposefully lowers games to below average (i.e. using the whole scale properly) to skew the Metacritic score.
Keep on trucking, Tom Chick.
Let's just get rid of the 1st Amendment altogether while we're at it.
michaelius
Banned
Very nicely put
grimshawish
Banned
Well the format is out of synch with metacritic; they honestly shouldn't even be there. So instead they reduce their influence.
End of the day an 80 from them is 4 stars.
I've said this before; someone with web skills - get to it!
From this angle that would make sense why bonuses corrollate to metacritic scores. Critical reception, blah - MC is a measure of how well a game is going to sell, not how good it is
XANDER CAGE
Member
I doubt there's anyone at any of these sites that is directly responsible for any of this stupidity. This is pretty much solely MetaCritic's problem.
Starwolf_UK
Member
So if I want to be weighted highly on gamerankings I should review games to 3 significant figures, right?
But that said average on its own is not that interesting. It would be interesting to see standard deviation. It might give a glance of view of how divided reviewers are (yeah yeah look at higher orders to figure exact distribution by which point you might as well open a histogram).
I think the future in the context of a personalised Internet is weighting based on an individuals preferences. Then again the merit is kind of limited.
But that said average on its own is not that interesting. It would be interesting to see standard deviation. It might give a glance of view of how divided reviewers are (yeah yeah look at higher orders to figure exact distribution by which point you might as well open a histogram).
There is a Gaming Age forum? From what I understand NeoGAF is NeoGAF because it parted ways with Gaming Age.
This might lead to confusion as scores constantly change.
I think the future in the context of a personalised Internet is weighting based on an individuals preferences. Then again the merit is kind of limited.
Yup, major props to Kotaku for doing that. More power to the review itself.
All those sites, including RottenTomatoes, are pretty terrible. Just link the reviews and stop giving an average rating.
Broder Salsa
Banned
Imagine a world without scores, guys.
br0ken_shad0w
Member
Rotten Tomatoes had a gaming section, it's just that nobody cared for it for some reason. Of course that was back before Metacritic became "a thing".
grimshawish
Banned
Love Kotaku's review scoring; actually pressures you lot to tell us why you say YES or NO
Do you know if metacritic pick the reviews themselves; or if they're 'sent' to them by the sites?
Zen_Arcade
Banned
Seriously. I'd trust their opinions over everyone in the top tiers.
partyphone
Banned
what would fanboys bicker about instead? : (
It doesn't seem like that list is comprehensive. There are two publications with the name *****'d out, one may be Quarter to Three. Also, I doubt metacritic would list a review if it didn't actually factor into the aggregate score.This effectively destroys all the stupid but incessant conspiracy theories that Tom Chick purposefully lowers games to below average (i.e. using the whole scale properly) to skew the Metacritic score.
Keep on trucking, Tom Chick.
firehawk12
Subete no aware
I guess it's good that although Metacritic and Gamespot are CBSi properties, Metacritic doesn't give Gamespot the highest weight?
I wonder if they have to treat Giantbomb better now though. lol
I wonder if they have to treat Giantbomb better now though. lol
XANDER CAGE
Member
Love Kotaku's review scoring; actually pressures you lot to tell us why you say YES or NO
Do you know if metacritic pick the reviews themselves; or if they're 'sent' to them by the sites?
MetaCritic goes and grabs the reviews themselves, as far as I know. In all of my writing experience I never had any interaction with an actual MetaCritic person, but then again I've never written for anybody weighted at like 1.5 like IGN. I'm not sure how it works for big publications.
CosmicQueso
Member
Because publishers did not have a version of this list.
Anyways, if you check the OP, some researchers backwards engineered the rankings to get to these figures. It doesn't appear to be the real list, but a close approximation.
The list is not comprehensive but the *** sites are banned on neogaf. The link has no censored sites.
Riposte
Member
Scores would still exist even if you removed the explicit number at the end. What we would be left with would be people being more vague about whether 1) they hated the game, 2) they didn't like the game much, 3) they had mixed feelings on the game, 4) rather liked the game, or 5) loved the game.
They also fucked it up relative to movie scores: as I recall if a game got a 7 or lower it was marked as a "rotten" score, leaving only 8-10 mattering. As much as people talk about the skewed review scale it wasn't THAT skewed on average from what I saw, and as I recall we had plenty of pretty good games get ravaged there as a result.
Domstercool
Member
Love Kotaku's review scoring; actually pressures you lot to tell us why you say YES or NO
Do you know if metacritic pick the reviews themselves; or if they're 'sent' to them by the sites?
Metacritic does all that. I know on DarkZero when a new review goes up we don't do anything, Metacritic just pull it themselves and puts it on the site.
That is true, but unless there is a dramatic shift in how games are scored (it's been pretty much the same, for like, forever) it won't change noticeably, if at all. In any case, the distribution of past site scoring practices could be fixed for each review, to reflect the time period when it was scored and it might also be pertinent to only use, say, the past 2 years of scores to calculate a site's distribution.