• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.

Metacritic uses OpenCritic data

Lumination

'enry 'ollins
Digital plagiarism. OpenCritic has APIs that you can license and use to get the data you want from them, essentially paying them for their work in aggregating this data. Instead, Metacritic is scraping their data for free.
 
giphy.gif

giphy.gif
 

danowat

Banned
Why? The situation seems extremely clear-cut.
I just think is always better to try and sound these things out in private and see if there isn't some kind of solution that can be found, rather than splashing it in public in the first instance.
 

ps3ud0

Member
Digital plagiarism. OpenCritic has APIs that you can license and use to get the data you want from them, essentially paying them for their work in aggregating this data. Instead, Metacritic is scraping their data for free.
Ahh thats the word I was looking for instead of webcrawler - scraping. Lazy as fuck if it is...

Reminds me of that IT bloke that got a number of Chinese IT guys to do his job for him - at least he paid them :p

ps3ud0 8)
 

Lanrutcon

Member
Ahh thats the word I was looking for instead of webcrawler - scraping. Lazy as fuck if it is...

Reminds me of that IT bloke that got a number of Chinese IT guys to do his job for him - at least he paid them :p

ps3ud0 8)

You don't have to sign your posts, you know. Friendly advice ;)
 

Shifty

Member
lol

That more or less sums up my feelings on this. Yeah.

Seriously though, did nobody at Metacritic see the irony of doing this? Sourcing reviews from a site intended to be their less evil non-corporate equivalent?

You don't have to sign your posts, you know. Friendly advice ;)

He won't take it.
 

Mattenth

Member
They were fast over at Metacritic. Those links have been changed.

That was fast.

What's so bizarre about the incident is that we know Metacritic has to have a working system. There are multiple publications that are unique to Metacritic and vice versa. They're decently fast with major titles (though we're still faster).

But we're also pretty sure that this can't be a fluke. We checked a lot of places, including Neogaf and reddit. We feel that the only way they got those URLs is from OpenCritic.

Since multiple people have asked, we also don't think there's any legal case, and there's nothing that would kill our momentum and drain our funds faster than a lawsuit. The exception would be if Metacritic starts deliberately circumventing new protections that we'll be launching shortly. In some sense, this statement wasn't optional: if we ever did want a case, we're required to put out statements scolding this type of behavior.

But we do think it's a scummy thing to do. We're proud of the technology and relationships we've built and really do want to make the industry better for everyone: critics, developers and gamers alike. We enjoy the competition with Metacritic but would much rather that competition result in innovation than stagnation.
 

Jeffrey

Member
Now I'm curious how metacritic and opencritic work. how much is automated how much is manual entry?

Why would metacritic be pulling from opencritic anyway? It's not like they weren't getting PCGamer reviews before opencritic existed.
 

Justinh

Member
Oh. Nevermind then. Just thought it was odd.
If it really bothers you, you could always just put him on your ignore list.
Recently I've been seeing a lot of the review threads linking to them both. I thought it was cool to see the thread creators start including Opencritic as well.
That's cool. I honestly didn't really differentiate one vs the other before, but now if I were to look at a review aggregator, I'd go to OpenCritic.
 

Shifty

Member
Why would metacritic be pulling from opencritic anyway? It's not like they weren't getting PCGamer reviews before opencritic existed.

A. Someone at Metacritic be trollin'

B. A policy of 'aggregate absolutely everything'

Odds are pretty even as to which.
 

Mattenth

Member
Now I'm curious how metacritic and opencritic work. how much is automated how much is manual entry?

For OpenCritic, all of the reviews are gathered automatically by loading/parsing review listing pages of each publication. In most cases, the review data can be automatically extracted from the page.

If (for whatever reason) it can't extract all the data, it flags it for admin review. Each of us have apps on our phones that lets us manage reviews very quickly with round-the-clock coverage. The most common reason for failing to auto-add are reviews-in-progress, as we don't post a review for scoring publications if we can't extract the score. Every review gets posted in a chat river so that we can quickly scan for errors (though our error rate is very low).

We also have a few other small edge cases. Critics have our email and if they email any review URL, we'll automatically load it for admin review in advance of the email.

I have no idea how Metacritic works.
 

Jeffrey

Member
For OpenCritic, all of the reviews are gathered automatically by loading/parsing review listing pages of each publication. In most cases, the review data can be automatically extracted from the page.

If (for whatever reason) it can't extract all the data, it flags it for admin review. Each of us have apps on our phones that lets us manage reviews very quickly with round-the-clock coverage. The most common reason for failing to auto-add are reviews-in-progress, as we don't post a review for scoring publications if we can't extract the score. Every review gets posted in a chat river so that we can quickly scan for errors (though our error rate is very low).

We also have a few other small edge cases. Critics have our email and if they email any review URL, we'll automatically load it for admin review in advance of the email.

I have no idea how Metacritic works.

Impressive, and I don't even see ads on the site. Is the only thing supporting your site atm is referral purchase links?
 

Mattenth

Member
Impressive, and I don't even see ads on the site. Is the only thing supporting your site atm is referral purchase links?

And the Patreon. We're working on licensing our API with a few retailers now.

We really hate ads. And if our early data is correct, gamers use adblock at a much higher rate than the average internet user.
 

Cyrano

Member
For OpenCritic, all of the reviews are gathered automatically by loading/parsing review listing pages of each publication. In most cases, the review data can be automatically extracted from the page.

If (for whatever reason) it can't extract all the data, it flags it for admin review. Each of us have apps on our phones that lets us manage reviews very quickly with round-the-clock coverage. The most common reason for failing to auto-add are reviews-in-progress, as we don't post a review for scoring publications if we can't extract the score. Every review gets posted in a chat river so that we can quickly scan for errors (though our error rate is very low).

We also have a few other small edge cases. Critics have our email and if they email any review URL, we'll automatically load it for admin review in advance of the email.

I have no idea how Metacritic works.
What defines an error? Can you give an exact error rate?
 

TalonJH

Member
For OpenCritic, all of the reviews are gathered automatically by loading/parsing review listing pages of each publication. In most cases, the review data can be automatically extracted from the page.

If (for whatever reason) it can't extract all the data, it flags it for admin review. Each of us have apps on our phones that lets us manage reviews very quickly with round-the-clock coverage. The most common reason for failing to auto-add are reviews-in-progress, as we don't post a review for scoring publications if we can't extract the score. Every review gets posted in a chat river so that we can quickly scan for errors (though our error rate is very low).

We also have a few other small edge cases. Critics have our email and if they email any review URL, we'll automatically load it for admin review in advance of the email.

I have no idea how Metacritic works.

I think I'll start using OpenCritic more because that description of your workflow just made climax a bit. Very nice work. Always wondered how it worked. Very efficient.
Sorry for being so crude.


Edit: Also, your site is so much cleaner.
 

Mattenth

Member
What defines an error? Can you give your exact error rate?

Good question that I unfortunately can't answer. We don't define them as "errors," we just flag for admin review. These range everywhere from "there's a terrible, terrible quote" to "we couldn't figure out what platform they reviewed the game on" to "searching for the game resulted in 2 similar results so we didn't auto-add." Some aren't actually "errors"

As an example, GamesRadar+'s DOOM review was flagged for review because our system couldn't automatically extract what platform the review was conducted on. There's no clear mention of "this review was done on X." So even though it's flagged for review on our end, it's not actually an error.
 

kenta

Has no PEINS
I only heard about OpenCritic a few weeks ago and it's already replaced Metacritic in my mind. Happy to have been validated so quickly!

Keep up the great work, Mattenth and crew

Edit: Decided to go ahead and sign up for an account and I can't just sign up with an email address? Am I missing the option?
 

Mattenth

Member
Edit: Decided to go ahead and sign up for an account and I can't just sign up with an email address? Am I missing the option?

Storing passwords = inviting hackers :-\ We're still just 2 developers so we put our effort where it matters. We don't think the incremental gains of signing up with email are worth it over multiple other options.
 

Cyrano

Member
Good question that I unfortunately can't answer. We don't define them as "errors," we just flag for admin review. These range everywhere from "there's a terrible, terrible quote" to "we couldn't figure out what platform they reviewed the game on" to "searching for the game resulted in 2 similar results so we didn't auto-add." Some aren't actually "errors"

As an example, GamesRadar+'s DOOM review was flagged for review because our system couldn't automatically extract what platform the review was conducted on. There's no clear mention of "this review was done on X." So even though it's flagged for review on our end, it's not actually an error.
So an error (admin review requests) seems to typically be something missed by the algorithm running what appears to be language-learning software to scan for keyphrases.

Fair enough I guess, though am still curious to know how often that happens (trade secret is a fair answer though).

Also curious about what constitutes scoring vs non-scoring, but probably also not an answerable question.
 

kenta

Has no PEINS
Storing passwords = inviting hackers :-\ We're still just 2 developers so we put our effort where it matters. We don't think the incremental gains of signing up with email are worth it over multiple other options.
You are as smart as you look. I like you guys
 

gamerMan

Member
Even if you don't have access to the API, it is pretty to write an HTML parser that can scrape OpenCritic. Since the links have unique identifiers, it appears that the links have been lifted from Open Critic.

The links appear to be switched now on Metacritic. But I wonder how hard would it be to correct all the links at once through a database query looking for these identifiers as opposed to be changing each one.
 

TSM

Member
Couldn't Metacritic just reformat all their links on the fly in the future and thus make it impossible to tell now that it's been pointed out? I wouldn't think it'd be overly difficult to sanitize the links to a point where no one can tell anymore.
 
Couldn't Metacritic just reformat all their links on the fly in the future and thus make it impossible to tell now that it's been pointed out? I wouldn't think it'd be overly difficult to sanitize the links to a point where no one can tell anymore.

Yep, they should be able to
 
Couldn't Metacritic just reformat all their links on the fly in the future and thus make it impossible to tell now that it's been pointed out? I wouldn't think it'd be overly difficult to sanitize the links to a point where no one can tell anymore.

Caps and double slashes are only two of the techniques that OpenCritic uses. They said they use other tricks too but did not describe them to avoid what you're suggesting.
 
Top Bottom