CrunchinJelly
formerly cjelly
I like you.
-
Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.
Metacritic's weighting system revealed
- Thread starter RotBot
- Start date
I like you.
Ivan A Nguyen
Member
Is this list up to date? I thought G4 was gone.
We also live in a world where Metacritic scores are used to judge whether developers are entitled to bonuses or not.
Pennywise83
Banned
Which makes it extra silly when they could simply look at sales directly, instead of score aggregates that feature the wise musings of KillerBob's Gaming Emporium.
You know, like it works in every other industry.
Messofanego
Banned
Someone list all the review sites that don't use scores.
Kotaku
Gameological Society
Rock Paper Shotgun
Kotaku
Gameological Society
Rock Paper Shotgun
baxter2231
Member
i bet most dont though
Mama Robotnik
Member
Regarding the Facebook response, the comment below it with the most likes really cuts through the crap:
We also live in a world where people don't read reviews, they use objective numbers to decide on purchases.
For the new page, I dont get why they dont make it public.
How about making that rating algorithm public, hm? Or offering an unweighted option, HMMM?
How about making that rating algorithm public, hm? Or offering an unweighted option, HMMM?
giant_frying_pan
Member
Cool, a site I used to write for gets a 1.0.
But honestly, this shouldn't come as a surprise to anyone on this forum. Metacritic was trash before this revelation and it's trash afterwards. But sadly, will still be a critical part of the business side of gaming.
But honestly, this shouldn't come as a surprise to anyone on this forum. Metacritic was trash before this revelation and it's trash afterwards. But sadly, will still be a critical part of the business side of gaming.
The tiers Gamasutra published were incorrect, as we now know.
And I don't see the site as a joke. The wisdom of the commons, in principle, is a pretty useful tool. Metacritic's processed may or may not be flawed. But Metacritic isn't just a videogame site. Yet, the videogame portion of the site is the only section that comes under fire (which is especially weird since other industries don't rely on numbers in their reviews as much). Why does Metacritic earn such wraith from the game's industry? Because the videogame publishers are the only ones infantile enough to judge financial compensation off a review aggregator. That's the crazy part. Not that Metacritic exists and not the service they provide.
shinra-bansho
Member
"It's wrong, but we won't tell you specifically what's wrong. Trust us."
It's also critical to the coverage of games that we consume. More valuable outlets get coverage and outlets that don't confirm to Metacritic lose
And that's fucking nuts and yet it still isn't Metacritic's problem. That's not the business model they setup. As my above post explains, Metacritic exists for all media, yet only comes under fire from the videogame portion. The problem isn't with Metacritic. It's with the publishers.
Pachterballs
Banned
lol at the list.
shocking.
How the F is giantbomb a 0.5??
lmao
shocking.
How the F is giantbomb a 0.5??
lmao
chickdigger802
Banned
I remember when 1up scores were decimated by metacritic. Let's convert a letter score system into a 20point scale! in which a B was like 67 or something. It got so dumb that 1up pulled out of metacritic, and now they dead 
Its still a problem with Metacritic that they arent disclosing what kind of ratio they are using and that they arent offering an unweighted option since that makes it obviously biased in one direction or the other.
If that applies to their movie scores as well, then that doesnt make it less of a problem with Metacritic.
giant_frying_pan
Member
I believe that the pool of 'clicks' that Metacritic generates wouldn't just vanish if the site did too. Viewership would be driven through other channels. Games would still be bought and reviewed.
It's their proprietary information and they are under no obligation to reveal it. They are not a public trust. They also don't coordinate the bonuses of videogame publishers. No one has yet provided a compelling reason why Metacritic should bear the brunt of the mistakes made by Videogame publishers.
I posted about this here last time we had a general Metacritic thread. I would dig out my post but I'm currently on the mobile site. It's surprising how many people just assumed that Metacritic is a straight average.
The fact that they apply subjective weightings invalidates the whole concept in my opinion.
The fact that they apply subjective weightings invalidates the whole concept in my opinion.
transience
Member
Whoa.
Forget Giant Bomb, where's Quarter to Three? I want to know how far Tom Chick moves the needle.
Forget Giant Bomb, where's Quarter to Three? I want to know how far Tom Chick moves the needle.
chickdigger802
Banned
This point here is worth considering:
Given sufficient data (and there's plenty of data there), then deducing the weightings Metacritic uses is almost trivial to anyone with a computer a decent understanding of statistics. The reason I say "almost" is that you have to make one particular assumption, which is that the weightings don't change over time*. Without that, though, it becomes a hell of a lot more difficult, and downright impossible if you don't know how often the weightings change.
*Equivalently, you could assume that the weighting changes are retroactive, but this would cause old games' scores to change from time to time, which Metacritic might not find desirable.
Metacritic said:
Given sufficient data (and there's plenty of data there), then deducing the weightings Metacritic uses is almost trivial to anyone with a computer a decent understanding of statistics. The reason I say "almost" is that you have to make one particular assumption, which is that the weightings don't change over time*. Without that, though, it becomes a hell of a lot more difficult, and downright impossible if you don't know how often the weightings change.
*Equivalently, you could assume that the weighting changes are retroactive, but this would cause old games' scores to change from time to time, which Metacritic might not find desirable.
I havent said anything about any videogame publisher. In fact its a problem for their whole site if its in their general concept. I simply said that they are operating on a premise that is falsely communicated to any user of the website. I did NOT know that they are weighing their sources, which is actually a rather big deal for a supposedly unrelated 3rd party website just measuring the general output of reviews.
Correct.
Cosmo Clock 21
Banned
While Metacritic dug their own grave for it, revealing reviewer weights is probably a bad idea. Publishers and maybe developers (who determine bonuses, hiring practices, and greenlighting projects based on past metascores) would start giving preferential treatment to certain outlets. Same reason why Google doesn't publish its exact PageRank algorithm.
They may well be incorrect, they also might not be, Metacritic saying they're wrong means nothing unless they publish data disproving these rankings.
I do agree that publishers are at fault - basing anything off a review aggregation site is ridiculous; but Metacritic's entire process of weighting publications is flawed - we have no idea who decides these ratings, how many people decide them, or what criteria go in to deciding them. As shown in this thread, if these were indeed accurate ratings that have been discovered then Metacritic's opinion of who goes in what "tier" would differ from many other people's opinions. The solution to this would be to do away with weighting systems entirely - a simple average of all review scores can't be disputed (even though the final number might be disagreed with based on opinion) - a weighted number where the weights have been decided based on set criteria by a person or persons, can be.
Either would allow competitors a stronger foothold in Metacritic's market. Their rankings are their secret sauce, for better or worse.
One more reason why it should be unweighted in the first place.
Diablo Rosso
Member
The site I write for is in the 1.25 bracket. Pretty cool but yeah, the whole system is shit.
Why would offering an alternate rating give competitors a stronger foothold in Metacritics market? They'd have a feature more through that. Give it a PR spin ("We are the only website offering an unbiased ranking and a weighted one on several secret sauce aspects to give you the real insight!") and go.
Without knowing how the weighting system actually works, the Metacritic average is meaningless. Even if we did know, it's crazy that peoples' livelihood actually depends in part on that arbitrary number.
Plus, how do they decide which outlets are providing "higher quality" reviews? Is it based on how their scores correlate with other websites? Is it just the personal opinion of someone at Metacritic? Do they manipulate weights to give certain games a higher or lower average? Is it possible that they actually assign different weights to the same outlet for different games? That last one could help explain why the published weights are "wrong"... if they actually are.
I guess this makes us all trust Metacritic a little bit less, which is probably a good thing.
Plus, how do they decide which outlets are providing "higher quality" reviews? Is it based on how their scores correlate with other websites? Is it just the personal opinion of someone at Metacritic? Do they manipulate weights to give certain games a higher or lower average? Is it possible that they actually assign different weights to the same outlet for different games? That last one could help explain why the published weights are "wrong"... if they actually are.
I guess this makes us all trust Metacritic a little bit less, which is probably a good thing.
shinra-bansho
Member
It would probably be better, but it's still easily disputable. There is subjectivity in deciding which review scores warrant inclusion.
Messofanego
Banned
He doesn't, so people can finally take off their tinfoil hats.
If people (and publishers) started using the unweighted Metacritic score as the basis for their purchasing decisions or crazy bonus schemes, then there's nothing to stop them using another site with identical unweighted scores.
vasametropolis
Member
Are people seriously outraged by this? Publications with a higher reputation probably should get a higher weighting.
I like the guys at GiantBomb too, but comparatively, ya, they don't have as high of a reputation. The average gamer has never heard of them. Reputation level is vital in every industry, even if it occasionally results in scandel.
This is not a golden defence of sites like IGN and GT, but they built themselves over years and have achieved mass appeal for good reasons. They do carry more weight with or without metacritic's scheme. As the smaller guys get more hits and a big following, they will escalate.
I like the guys at GiantBomb too, but comparatively, ya, they don't have as high of a reputation. The average gamer has never heard of them. Reputation level is vital in every industry, even if it occasionally results in scandel.
This is not a golden defence of sites like IGN and GT, but they built themselves over years and have achieved mass appeal for good reasons. They do carry more weight with or without metacritic's scheme. As the smaller guys get more hits and a big following, they will escalate.
I can't find 3DS, Vita, or WiiU stuff on Game Boyz, but I sure as hell can find original XBox coverage!
Man, if only this was powered by GeoCities/Angelfire and had Mew and Pikablu codes for Pokemon Red/Blue too.
Also, I like how the Metacritic update on FB does the expected "Those guys are totally false, but no we will not tell you our actual weighting system."
Man, if only this was powered by GeoCities/Angelfire and had Mew and Pikablu codes for Pokemon Red/Blue too.
Also, I like how the Metacritic update on FB does the expected "Those guys are totally false, but no we will not tell you our actual weighting system."
True, I hadn't thought of that. The actual score calculation though is more worthwhile than a system such as Metacritic's.
No, they shouldnt.
Mama Robotnik
Member
Given that this data was established mathematically in a span of work covering six months (if I read it correctly), then its the closest we can get to understanding MetaCritic's actual weightings. MetaCritic can retort that its "wildly inaccurate" but without disclosing their actual exact weightings - informed by whatever faux science they use - I'm disinclined to fully believe their hysteria.
The data tells us that there are measurable weighting differences, vastly so with sites such as say IGN and GiantBomb. Until MetaCritic gives us something concrete - then I'm going to keep this dataset in mind as the best information regarding their weightings.
They won't, of course. The wizard never wants you to look behind the curtain.
The data tells us that there are measurable weighting differences, vastly so with sites such as say IGN and GiantBomb. Until MetaCritic gives us something concrete - then I'm going to keep this dataset in mind as the best information regarding their weightings.
They won't, of course. The wizard never wants you to look behind the curtain.
Has the "average gamer" heard of Digital Chumps, Just Adventure, Gamers Europe and so on?
Given the bolded assumption, couldn't we re-crunch the numbers and try to figure out their weights as of today? Who feels like firing up R?
You should start explaining to the rest of us why Xboxic has a "higher reputation" than Edge and Giant Bomb. If there is a logical explanation, I sure as hell am not seeing it.
Reputation shouldn't be weighted. In fact, I almost am inclined to distrust big name review outlets, because of the incestuous relationship with the rest of the industry. Metacritic should just do what Rottentomatoes does; weigh them all equally, and then have a separate button to see the average of only what "respected" reviewers think.
I don't dislike Metacritic. I dislike how the industry has somehow convinced themselves that they need to rely upon Metacritic for whatever reason as a barometer of quality/sales.
I don't dislike Metacritic. I dislike how the industry has somehow convinced themselves that they need to rely upon Metacritic for whatever reason as a barometer of quality/sales.
chickdigger802
Banned
Probably more due to their rating scale which is just 5 points.
Publishers want 85+, which is pretty much just 5/5 for GB.