• Hey Guest. Check out your NeoGAF Wrapped 2025 results here!

study confirms the obvious - game scores don't matter

M

monchi-kun

Member
www.joystiq.com

journalists/reviewers am cry?! what's the point? 50 Bulletproof FTW

Game ratings don’t matter
Related entries: Culture, Gamecube, PC, PlayStation 2, Xbox



Research analysts Jason Kraft and Chris Kwak examined a sample of 275 video games released over the last five years and analyzed the ratings data for those games. They determined that the correlation between ratings given to games by reviewers and the subsequent sales of those games is statistically insignificant. The chart above demonstrates this. On the Y-axis are unit sales. On the X-axis is the average rating of various titles.

In their own words: “After going through multiple scenarios, we believe a game rating, in most cases, is not a reliable tool for predicting game sales…. The next time someone suggests that a game will shine or fade based on a rating, we encourage the reader to take it with a grain of salt.”

So what does affect sales? The best indicators of sales strength appear to be the strength of the publisher behind the game, the performance of prior iterations of the game (if it is a sequel) and the performance of the box-office sales of the movie version of the game.

Why? We’re guessing it’s because the hardcore gamers who tend to pay attention to ratings have very little impact overall on the financial performance of a game. Consumers who don’t read gaming publications dominate as a percent of the gaming population.

Send an email to kraftandkwak [at symbol] sig.com for a copy of the report.

[Image credit: Game Ratings correlation chart from Susquehanna Financial Group analysts Jason Kraft and Chris Kwak]
Click to expand...

There's a picture of the chart on Joystiq to make it even more official...
 
so...this doesn't factor in word-of-mouth and buzz that's brought on by game reviews and the hardcore gamers that pay attention to these things?
 
I believe this news.

Selling games to casuals and mass-market ends up being no different than selling other products - so many people don't buy things after careful study and fully comprehending what it is they're buying. Their reasons are much more hit-and-miss. Hey, it's probably one reason why so many crappy products from 3rd rate companies exist... consumers just buy whatever, frequently for unsound reasons, making terrible mistakes over and over again.
 
Doom: The Movie was #1 at the box-office, yet was almost universally panned by critics, just like Stealth and Transporter 2.

I'm sure the Lindsay Lohan album will be #1 on the Billboard charts, even if I don't think Spin or Rolling Stone will agree with that.

It's all the same.
 
i used to think that appealing to the hardcore is essential to building a great videogame franchise when our impact is obviously very small. but i'm not too saddened by this study. casual gamers peobably never think about going into game development which means that the hardcore gamers who DO cross over can hopefully impact game content in the long run.
 
Error2k4 said:
ICO
FE PoR
Psychonauts

all those games got very good scores and they bombed so this is nothing new.
Click to expand...

I wouldn't call Path of Radiance a bomb by any stretch of the imagination. I think it could easily get to 200K sold by next summer, which would put it in line with how the two Fire Emblems on GBA have sold. Personally, I was hoping for first month sales of 2.7 million, but I still don't consider it a dissappointment (50K for October). ;)
 
Parallax Scroll said:
I almost never read reviews. Word of mouth from people whose taste I trust is good enough for me.
Click to expand...
Those are the best kind of "reviews." My friends and I are always saying "hey, have you played this?" and 9 times outtta 10, it will affect whether I get a game or not. While I will admit I check out online reviews on occasion when none of my friends have played a game, I use them more as a guideline than a rule (find out the game's details). Case and point, reviews for Polarium on the DS trashed it, but I thought the premise sounded interesting. It might not have been a AAA game, but I've gotten an incredible amount of play out of it and am glad I got it.
 
I'll go ahead and disagree with this article.

There are way too many variables here. If you were to take away all licensed games and sequels, there study would have drastically different results (duh).

Instead, look at where many of these sequels originated. Most popular non-licensed sequels come from games that generally received critical and consumer acclaim.

If Halo, Mario, Metroid, Zelda, Gran Turismo, Soul Calibur, Sonic, Metal Gear, Katamari, etc. were all "5.0" games, they would not be selling nearly the same numbers they do today. Also, for all those complaining about the ICOs or GoWs not selling blockbuster numbers, I imagine if they were actually shitty games, they'd have sold 1/10th of what they actually did.

I know I'm not saying anything profound here, but to say scores don't matter is bullshit. It's just that the power of the advertising dollar, sequels, and licenses help propel gutter trash to sales they should never receive. Take away the fluff and I guarantee this study would produce different results.
 
The scores matter to 100% of the folks reading them, so they matter 100%.

Nobody is surprised when Matrix sells squillions of copies. Magazine readers only represent the tip of an iceberg.
 
I thought we already knew that the average casual gamer was an uninformed idiot?

"It's got 50 Cent and you can shoot people? Sold!"
 
There are some flaws in their research.

Well, actually I guess not. Its more an example of message board posters making a bigger deal out of something than it really is. I mean no shit there's little/no direct correlation between review scores and sales. The only conclusion they drew was that you couldn't look use a game's review average to predict sales... there's way way too many outliers. This is information we all already had.

What would be much MORE interesting would be to take a look at only original, new IP from this gen. THEN compare the review average to sales. Throws licenses and sequels out of the mix, and compare sales of Psychonauts to Psi Ops to PN03, etc.
 
Good news for PDZ!! :)



I've wanted to do this for years now....


But I never figured to just take a sample, but figured I'd need to get ahold of the GameRankings.com database and run it against the NPD.


What was there sampling method?


-- edit. The 50cents game has been advertised non-fucking-stop lately... so that things like that will more than overcome bad reviews.
 
Not a big deal. Video game publications in general only care to a hardcore gamer market to begin with (This wasn't true during the 8 and 16 bit generation when video games were hip and cool with casual audiences so everyone was into it). So naturally only hardcore gamers care about ratings. As long as the publications' target userbase cares, reviews will still be relevant/necessary.
 
Is there any media that is not this way? Books, music, movies, television etc. It is almost always the hype that wins out over critical praise.
 
Pellham said:
(This wasn't true during the 8 and 16 bit generation when video games were hip and cool with casual audiences so everyone was into it).
Click to expand...
What? I think you have that backwards.
 
Billy Rygar said:
Is there any media that is not this way? Books, music, movies, television etc. It is almost always the hype that wins out over critical praise.
Click to expand...
Games are more objective than those things. And word of mouth on those movies and music always wins out in the end, anyways. Folks just end up liking things that the elites hate.

Games are a little differant in that they have a whole set of objective criteria to go along witht he subjective. Challegne, addiction, glitches, control, online, and extra features. Take, for instance, electronic products... which are mostly valued by objective criteria and whcih I'm confident have a positive correlation between reviews and sales.
 
Honestly Gahiggidy you really think games are more objective? Why is it then I tend to get into the most vitriolic arguments with the people in my office over our opinions on games. I believe the only reason they seem more objective is because they are almost always tied to a numerical score and numbers for many people seem a much more objective system than "impressions" but in reality have no more basis in fact.
 
Really, its the retail chains that are most responsible for informing the consumers. They all know what's crap and what's qulaity... but they make no effort to steer consumers in the right direction.
 
As people have already said, quality is never the only factor in sales, in anything. (BTW are these Euro sales only or something? RE4 is at like 100K here, DMC only at 500K?)

That said, someone want to explain to me how this chart proves their point?

gamesales6sl.gif


There are not only more better selling games (dots) above 80% rating, but -no- game that sold more than 1.5 million falls below 80%. From there many of the best selling games go up on an almost perfect curve with the ratings.

Seems to me if ratings/game quality wasn't significant, the dots would be spread out a lot more evenly across the entire graph.

edit: I mean obviously you can never say 'This game got a great score, it's gonna sell great!' but judging from this graph there does seem to be a correlation.
 
Gahiggidy said:
Really, its the retail chains that are most responsible for informing the consumers. They all know what's crap and what's qulaity... but they make no effort to steer consumers in the right direction.
Click to expand...
I am in total agreement with you on this point, especially around christmas time.
 
Gahiggidy said:
Really, its the retail chains that are most responsible for informing the consumers. They all know what's crap and what's qulaity... but they make no effort to steer consumers in the right direction.
Click to expand...
Disagree. Most sales people are as clueless as the customers. Hang around a Gamestop or EB for a little while and try to hold in the laughter at some of the things the employees tell people.
 
Billy Rygar said:
Honestly Gahiggidy you really think games are more objective? Why is it then I tend to get into the most vitriolic arguments with the people in my office over our opinions on games. I believe the only reason they seem more objective is because they are almost always tied to a numerical score and numbers for many people seem a much more objective system than "impressions" but in reality have no more basis in fact.
Click to expand...
Not more objective than subjective... but more objective than movies, music, etc.

Anyways, the data from reviewers would show a pretty strong consensus on the quality of a game.
 
gamesales6sl.gif


^^^


Ehhh, looking at some of th etitles they picked, thatr seems like a pretty flawed sampling... alot of those games haven't even been out a year!! Or are they only counting the first ffew weeks of sales?? Alot of the original IP games take a good while to build up there sales. SMB is a very good example. I really think you'd have to give at least a two year span of sales data on each game.
 
What's really needed is a regression formula to account for all of the various reasons a game sells. I'm user the review scores have something to d with the sales... albeit small and only with certain types of games. This study looks to be more along the lines of a "sociology" study... which looks for correlation to test a hypothesis; i.e. "Review Scores are positively correlated with the sales of a game". An econ aproach would attempt to find all the variables to come up with some kind of formula such as...

p((l*b) + a/2.5) + r == s

p == Publisher's average sales
r == Review score
l == Existance of a lmovie iscense (1 or 0)
b == box-office take of movie.
a == advertising budget
r == sales of game



Any econ majors want to volunteer for the project. ;)
 
This is kinda good, I guess, since all the reviewers who are biased and paid to give games high scores don't have much of an effect.
 
If you average together all reviews and all game sales, I'm sure this report's conclusion is more or less accurate. However, I think there must be some games for which reviews don't matter one whit and others for which reviews matter a great deal. The extremes tend to disappear in the grand scheme of things.

For the people who do read reviews and care about what reviewers have to say, a well written review can make or break the sale. To those people, however few they may be (now that would be an interesting thing to research), reviews are very important. I wish Joe Schmoe would read my reviews and make better purchasing decisions, but I'm hardly surprised to learn that he doesn't. There are plenty of people who do, though.

EDIT: Looking at the graph Gah posted, this report seems even more dubious. Notice that they only looked at games with metacritic scores over 50%. Not only does that throw out some results that could sway the overall conclusion (for instance, they could show that extremely high or low review scores have a greater impact on sales than "middle of the road" scores do), but focusing on that range also limits the scope of the investigation to the region of review scores that is the least well defined and understood from one reviewer to the next. Most people understand the implications of a 30% review or a 90% review, but what about a 70% review? Is the game still worth buying? Many reviews in that range just aren't clear, which could lead to confused readers buying the games anyway. Furthermore, heavily marketed games that are targeted at mainstream consumers who do not read reviews are most likely to fall into the so-called average score region between 50% and 75%.
 
There have been similar studies that have said the same for film and music reviews. But like someone has already said, those that actually go out and read these reviews are affected by the scores/text in them...having a great deal of sway in their ultimate decision to purchase, rent, or stay home and crochet a sweater.
 
I suspect that lukewarm reviews for Super Mario Sunshine accounted for alot of its disapointing sales.

- edit. Of course, how do you seperate that out from Word of Mouth? Almost an impossible task.
 
olimario said:
They matter to the hardcore gaming minority.
Click to expand...

Uh... no they don't. The hardcore gaming minority recognizes that game review scoring is a farce.

Truly bad games in America still get 7s which is, last time I checked, is a bit better than average... and truly bad games in Japan (like Sakura Taisen 4) get a 35/40 in Famitsu. Their judgment criteria is really skewed toward the mainstream user.
 
Link said:
Disagree. Most sales people are as clueless as the customers. Hang around a Gamestop or EB for a little while and try to hold in the laughter at some of the things the employees tell people.
Click to expand...

Here are some things I've heard while in EB / GS

"Final Fantasy VII is being rereleased by Square for the Xbox 360"

"The Playstation 3 will be $499 and won't include a hard drive"

"Nintendo is going to be bought by Sony, so Mario 128 will be a launch title for the PS3"

Those are just some of the things I've heard at various game stores! Seriously, where do these people get this shit from? EB employees are worse than ***** for god's sake.
 
A more interesting study would be to see what effect of ESRB ratings on a game's sales... and if that effect is trending in the "M" direction since it started being used.
 
You must log in or register to reply here.

Similar threads

Games Where Choices Really Do Matter?
31
574
Polaris replied
Yes, Overwatch Characters Have Been Getting Sexier, But Not For The Obvious Reason
Opinion Game Dev 
48
1K
The Cockatrice replied
Dead by Daylight studio Behaviour Interactive confirms layoffs
5
170
winjer replied
Xbox says just because you don't like a game, doesn't mean it's a bad game
2 3 4 5
208
650
mckmas8808 replied
Finishing a video game can trigger “post-game depression,” study finds, with the effect being particularly strong after completing an RPG
2 3
127
1K
KungFucius replied
Top Bottom