Support NeoGAF

jvm · Jun 23, 2008

Simon Carless, of Gamasutra and GameSetWatch, has posted a thorough critique of V G C h a r t z. (I used tinyurl to create that link to avoid the filter here.) He uses hard figures on Iron Man PS2, Ben 10, and two unnamed titles to draw some conclusions.

We all know Iron Man PS2 was a surprise in the last NPD, but here's an example of what Carless brings to the discussion: Ben 10 sales figures.

Simon Carless said:
It's difficult to find examples, but let's concentrate on some games that have sold in significant numbers, but have never made it into the Top 20 in North America for a significant time. One good example is the Ben 10 series of games from D3 Publisher. ******** has the series listed at 590,000 sold worldwide to date. But when Gamasutra interviewed D3's Yoji Takenaka last week, he specifically said: "Ben 10 is selling well over a million units right now, since last Christmas."

That's a fairly sizable error, even if it is conflating shipped to sold (as Carless notes).

Folks may also be interested in the discussion of how the estimate for MGS4 sales went from 1.5 million down to 1.3 million. Brett Walton explains how they estimated European sales before actual European sales were reported (for real) and were off by a couple hundred thousand units.

I realize this is treading in dangerous waters (reporting figures from the site is a reason to be banned), but I think that a serious article like this at least deserves awareness, if not discussion, here. It is one thing to point to our own discussions and history about how we perceive numbers reported by the site in question, but quite another to have a third party -- the editor of a prominent set of sites -- comb through the data, talk to the site owner, and come to a reasonable conclusion. A conclusion, I might add, that I think most people here could agree with (my emphasis):

Simon Carless said:
But that doesn't really change the main problem with the site. There's a place for a resource like ********, but it'd be a site that clearly labels the source of its estimates (whether it be Chart-Track, NPD, Media Create) and then labels which are its estimates based on its own industry knowledge and whatever channel checks it has.

But if I was a writer or analyst trying to extrapolate significant information from the resource, especially regarding those titles which don't chart regularly, given the major discrepancies with figures shown here, I would not recommend it.

Now, when contacting someone about their use of that site's data, we can point to this. I think that may well have a larger effect than just trying to explain it ourselves.

Disclosure: I write occasionally for Gamasutra. I reviewed a draft of the article before it was posted. I also write a monthly column for Next-Gen.biz which depends almost exclusively on NPD data.

devilhawk · Jun 23, 2008

So when this crap site reports bad figures we make threads. When it reports good ones we can't? That's what you are saying

Lets never post about the site again.

avatar299 · Jun 23, 2008

so who is going to make the the thread about this at their forums?

JoshuaJSlone · Jun 23, 2008

devilhawk said:
So when this crap site reports bad figures we make threads. When it reports good ones we can't?

If we have a way to confirm a figure is good, then there's already a better source to use instead.

Mario · Jun 23, 2008

Based on the titles I have hard numbers for, the site is reporting sales volumes of only ~38% to ~50% of their actual volumes which is quite a large underestimation.

There is one local journalist that keeps quoting the site for numbers when reporting about us too which annoys me.

The Faceless Master · Jun 23, 2008

that's a pretty good article that shows how far off that site's numbers are

noonche · Jun 23, 2008

What's really disturbing is that other places have started reporting their numbers as if they have some kind of meaning. That causes me to lose a lot of respect for any place that bunches them in with NPD, Chart-Track or Media Create. When just a little bit of research would reveal how screwed up their methodology is, I think it shows a lack of integrity for a journalist to not do said research.

SapientWolf · Jun 23, 2008

I'm curious to know how close NPDs numbers are, since they have to estimate the sales of places like Wal Mart. Probably not a big deal for the bigger releases, but Deer Hunter on the PC could be a multi-million seller for all we know.

ToxicAdam · Jun 23, 2008

I like VGC references. It lets me discern a shitty journalist/article from a good one.

leroy hacker · Jun 23, 2008

I'm amazed by the existence of the Chartz fanboys who popped up in the comments to that article.

I hope coverage of Chartz like this piece will lead to fewer reporters using them as a source.

Randy Lanzarote · Jun 23, 2008

I hope that as more lazy journalists rely on using ******** in their articles, the more publishers/developers will get pissed off with all these factual errors being reported and start proceedings against it.

Its bound to happen one day, some bored intern will string together a piece about EA/Ubisoft/Activision, the article gets a little too much attention for their liking and *bam* lawsuits ahoy.

dialmydrive · Jun 23, 2008

We shouldn't give this the time of day but I'm happy to read it and equally proud of Simon for taking the time to write it. I knew this would eventually happen. It makes Robert P., the author of the other article mentioned, look like a fool. This is the biggest problem with Wall Street. Most know little to nothing about this industry, refuse to pay a dime for any information, going instead with what is marketed to them as free and equally reliable, then begin screwing the industry by making poor Street decisions based on poorly compiled and estimated data.

Not only that but that site works very hard to discredit other sources then flips the industry on its back by using the information provided by the very sources that it looks to discredit (and of couse, never crediting those sources). Any limited, halfway decent data it has is stolen. Talk about double plus ungood! Are we sure this Brett Walton character doesn't work for the Bush administration?

Vorador · Jun 23, 2008

They are banned for a reason.

It's a good read, but not something we didn't know. They pull the numbers out of their ass and then correct them on the fly when the real numbers appear. But they can only estimate worldwide sales because there's no global sales tracking, and their estimations are usually far off.

I am still flabbergasted of how much people take their numbers as real and defend the site like it's a reliable source. :lol

apujanata · Jun 23, 2008

Very nice article. thanks for the post, jvm. I don't think I have ever mentioned to you that I like reading your monthly (NPD analysis) article at next-gen biz.

Earthstrike · Jun 23, 2008

I think its a good idea to raise awareness about articles like this since I have friends that actually think ******** is "the best site" for game sales. If I know some you probably do too.

squicken · Jun 23, 2008

Banned Site's Methodology said:
- Sampled direct sell-through data
- Industry knowledge and experience - applying past trends in terms of marketshares, regional breakdowns, casual vs hardcore and so on
- plenty of statistical analysis, regression calculations, market projections
- Contact with industry figures - buy-side analysts (such as Pachter / Divnich), sell-side analysts who work with us on specific products / projects, manufacturers who work with us to project sales of their key titles
- Retail checks - we have a team who talk to stores and estimate shipment figures for low-stock and hard to find items which we struggle to track with our normal data samples.

I don't know guyz, that soundz pretty legit to me. Right? You guyz? Right? Guyz . . .

dialmydrive · Jun 23, 2008

apujanata said:
Very nice article. thanks for the post, jvm. I don't think I have ever mentioned to you that I like reading your monthly (NPD analysis) article at next-gen biz.

apujanata, I noticed your name on http://radar.oreilly.com/archives/2008/06/data-on-the-web-********-vs-np.html

Then I looked at your posts. You seem to respond mostly to sales posts or posts that are relevant to that other site. Are you a (nameless site) mole? It appears that way to me.

Jonnyram · Jun 23, 2008

dialmydrive said:
apujanata, I noticed your name on http://radar.oreilly.com/archives/2008/06/data-on-the-web-********-vs-np.html

Then I looked at your posts. You seem to respond mostly to sales posts or posts that are relevant to that other site. Are you a (nameless site) mole? It appears that way to me.

You must be joking. I've known apujanata for about 5 years, first at GameFAQs and then here. He is just interested in game sales generally and you're jumping to insane conclusions.

dialmydrive · Jun 23, 2008

Jonnyram said:
You must be joking. I've known apujanata for about 5 years, first at GameFAQs and then here. He is just interested in game sales generally and you're jumping to insane conclusions.

It looked weird to me. I just made a valid observation, that's all.

cordonbleu · Jun 23, 2008

******** is definitely not a reliable source, but, their ballpark figures are definitely better then absolutely nothing, which is what we get from "official" methods.

I hope that over time their site becomes more legitimate and has better ballpark figures, because otherwise, we will never know how games sell.

Stumpokapow · Jun 23, 2008

cordonbleu said:
******** is definitely not a reliable source, but, their ballpark figures are definitely better then absolutely nothing, which is what we get from "official" methods.

The whole point of this thread is to demonstrate that this is not the case. Their figures could be exactly right--they could be 50% off--they could be 100% off. The only time you ever know if they are correct or even in the same ballpark is when you have actual data, in which case why would you care because the actual data is right there?

kaching · Jun 23, 2008

Mario said:
Based on the titles I have hard numbers for, the site is reporting sales volumes of only ~38% to ~50% of their actual volumes which is quite a large underestimation.

There is one local journalist that keeps quoting the site for numbers when reporting about us too which annoys me.

You've run over more dogs in this industry than anyone else. How do you do it?

Cosmonaut X · Jun 23, 2008

cordonbleu said:
******** is definitely not a reliable source, but, their ballpark figures are definitely better then absolutely nothing, which is what we get from "official" methods.

No, theyre not, especially when people outside of iois little clique start to rely on their numbers.

cordonbleu · Jun 23, 2008

Stumpokapow said:
The whole point of this thread is to demonstrate that this is not the case. Their figures could be exactly right--they could be 50% off--they could be 100% off. The only time you ever know if they are correct or even in the same ballpark is when you have actual data, in which case why would you care because the actual data is right there?

The fact is that the Actual data is NOT right there. If it was, then this site would not exist. In a world where people are starved for information, this mob have come along and offered their version of that information for free, and until another source comes along that is right and actually shares their info with the public, then the site will not go anywhere.

I dont go there believing they are correct, I go there because real or not, they are the only and closest numbers we get, and its better then nothing at all.

Pureauthor · Jun 23, 2008

cordonbleu said:
I dont go there believing they are correct, I go there because real or not, they are the only and closest numbers we get, and its better then nothing at all.

No it's not. It is literally worth the same as 'nothing at all' since we haven't the slightest idea if their numbers are correct or not.

cordonbleu · Jun 23, 2008

Pureauthor said:
No it's not. It is literally worth the same as 'nothing at all' since we haven't the slightest idea if their numbers are correct or not.

Well, what can I say, i'd prefer to be delusional now and corrected later. I'm impatient that way.

Stumpokapow · Jun 23, 2008

cordonbleu said:
The fact is that the Actual data is NOT right there. If it was, then this site would not exist. In a world where people are starved for information, this mob have come along and offered their version of that information for free, and until another source comes along that is right and actually shares their info with the public, then the site will not go anywhere.

What I'm saying is what they're offering is not "the closest numbers we get", it's that they're honestly random numbers. It's not that they're not accurate, it's that you have no idea if they're accurate or not. There's no fixed margin of error for the site, no PI or CI values for games at all.

Let me put it another way; if I started a separate site called ZGZHARTZ.zorg, would you use that for your data instead? It'd just be random figures which no one could ever verify, but at least it's better than nothing, right?

Saint10118 · Jun 23, 2008

Stumpokapow said:
What I'm saying is what they're offering is not "the closest numbers we get", it's that they're honestly random numbers. It's not that they're not accurate, it's that you have no idea if they're accurate or not. There's no fixed margin of error for the site, no PI or CI values for games at all.

Let me put it another way; if I started a separate site called ZGZHARTZ.zorg, would you use that for your data instead? It'd just be random figures which no one could ever verify, but at least it's better than nothing, right?

the proper comparison would be that when NPD started tracking Amazon vgchartzzzz got snah

Vorador · Jun 23, 2008

cordonbleu said:
The fact is that the Actual data is NOT right there. If it was, then this site would not exist. In a world where people are starved for information, this mob have come along and offered their version of that information for free, and until another source comes along that is right and actually shares their info with the public, then the site will not go anywhere.

I dont go there believing they are correct, I go there because real or not, they are the only and closest numbers we get, and its better then nothing at all.

The problem is that they don't have reliable sources, but present their numbers as truth. They never say and will never acknowledge that in reality is a estimation of the real sales, but not the real, hard-proofed numbers, so people believe them(even journalist) and report their numbers as the real thing.

cordonbleu · Jun 23, 2008

Stumpokapow said:
What I'm saying is what they're offering is not "the closest numbers we get", it's that they're honestly random numbers. It's not that they're not accurate, it's that you have no idea if they're accurate or not. There's no fixed margin of error for the site, no PI or CI values for games at all.

Let me put it another way; if I started a separate site called ZGZHARTZ.zorg, would you use that for your data instead? It'd just be random figures which no one could ever verify, but at least it's better than nothing, right?

Your right about the fact that the numbers are wrong, im not saying they are right, but im also not saying that the site is the most evil thing to hit the internet.

If you started your site, id be happy to use it in addition to every other source that is available on the net, what harm is there in that, im objective enough to take things with a grain of salt.

Like I said, my hope is that VgCrap continues on its merry ways, their data becomes more encompassing and reliable over time and that the respected Number Nazi's are bought to their knees, or required to offer more then a Top Ten to survive.

Saint10118 · Jun 23, 2008

cordonbleu said:
Your right about the fact that the numbers are wrong, im not saying they are right, but im also not saying that the site is the most evil thing to hit the internet.

If you started your site, id be happy to use it in addition to every other source that is available on the net, what harm is there in that, im objective enough to take things with a grain of salt.

Like I said, my hope is that VgCrap continues on its merry ways, their data becomes more encompassing and reliable over time and that the respected Number Nazi's are bought to their knees, or required to offer more then a Top Ten to survive.

where is the sense in taking lies with a grain of salt?

cordonbleu · Jun 23, 2008

Vorador said:
The problem is that they don't have reliable sources, but present their numbers as truth. They never say and will never acknowledge that in reality is a estimation of the real sales, but not the real, hard-proofed numbers, so people believe them(even journalist) and report their numbers as the real thing.

Well, I do think that they indicate on the site that they take a sample and extrapolate, I think all ratings firms do the same thing, and none are 100% accurate. Im unsure what their sample is, though the article in the OP indicates 2-4%.

Stumpokapow · Jun 23, 2008

cordonbleu said:
If you started your site, id be happy to use it in addition to every other source that is available on the net, what harm is there in that, im objective enough to take things with a grain of salt.

I just told you the site would have completely invented numbers with no bearing on reality whatsoever, and you'd still use it?

Like I said, my hope is that VgCrap continues on its merry ways, their data becomes more encompassing and reliable over time and that the respected Number Nazi's are bought to their knees, or required to offer more then a Top Ten to survive.

Given that NPD's target market is not internet amateur mathematicians who have no interest in paying for anything, and given that they're a 120 million dollar a year+ company (counting all divisions, not just TRSTS), something tells me they'll be able to survive.

dammitmattt · Jun 23, 2008

cordonbleu said:
Well, I do think that they indicate on the site that they take a sample and extrapolate, I think all ratings firms do the same thing, and none are 100% accurate. Im unsure what their sample is, though the article in the OP indicates 2-4%.

Do you work for VGC? That's the only thing that would explain it.

Their numbers are no more reliable than any of the 200 or so of us who particpate in the monthly predictions thread. If there was any merit to their extrapolation methods, they wouldn't change their numbers when publicly available data is released.

Vorador · Jun 23, 2008

cordonbleu said:
Well, I do think that they indicate on the site that they take a sample and extrapolate, I think all ratings firms do the same thing, and none are 100% accurate. Im unsure what their sample is, though the article in the OP indicates 2-4%.

Wait, you're now trying to say than Chartz is basically as reliable as NPD since they use the same metod??

WTF.jpg

cordonbleu · Jun 23, 2008

Stumpokapow said:
I just told you the site would have completely invented numbers with no bearing on reality whatsoever, and you'd still use it?

Yeah, but you didn't say you were going to advertise that fact.

Hey, I like to read lots of nonsense on the Internet anyway......Obviously.

Given that NPD's target market is not internet amateur mathematicians who have no interest in paying for anything, and given that they're a 120 million dollar a year+ company (counting all divisions, not just TRSTS), something tells me they'll be able to survive.

Lets hope so, that way we can enjoy our list of Top 5 games soon enough, with no numbers at all.

If I was in the retail game industry, i'd be backing a site like this with full enthusiasm, hoping that such information could be free.

cordonbleu · Jun 23, 2008

Vorador said:
Wait, you're now trying to say than Chartz is basically as reliable as NPD since they use the same metod??

WTF.jpg

I didnt say anything about their reliability at all, I just commented on you stating that they do not indicate where they get their numbers.

And you are all right, they could just as easily come out of their ass for all I know. What i'm saying though is that what their intention is, if it is in fact what is stated on their site, should be something that is embraced by a numbers starved community of gamers, who find such things as game sales interesting.

Like I said, my hope would be that as the site continues to grow, their ability to obtain and forecast and broadcast their numbers would be much more cohesive and trusted.

Do you work for VGC? That's the only thing that would explain it.

Their numbers are no more reliable than any of the 200 or so of us who particpate in the monthly predictions thread. If there was any merit to their extrapolation methods, they wouldn't change their numbers when publicly available data is released.

No, I dont work for anyone my friend. I dont actively post there, all I do is visit and see what they believe has sold on a weekly basis. No harm done.

And I agree, the numbers are not very reliable.

dammitmattt · Jun 23, 2008

cordonbleu said:
If I was in the retail game industry, i'd be backing a site like this with full enthusiasm, hoping that such information could be free.

I agree that the information should be freely available just like music, movie, and book sales, but backing a site that makes up numbers isn't the path to make this happen. I don't understand where you see the correlation.

cordonbleu · Jun 23, 2008

dammitmattt said:
I agree that the information should be freely available just like music, movie, and book sales, but backing a site that makes up numbers isn't the path to make this happen. I don't understand where you see the correlation.

Because regardless of their inadequate at times ability to get the numbers right, I dont know where the 'they make the numbers up' line comes from?

Jonnyboy117 · Jun 23, 2008

Ignorant Truth > Ignorant Bullshit.

"I know nothing." > "I know something." (Lies)

It's really that simple.

If they are using statistics to generate these estimates, I think they need to go back to school. There are surely tons of people here who know more about Statistics than me, but I'm pretty sure you can't extrapolate continuous data based on such limited sampling from a semi-chaotic population (the retail market for games). Regression statistics are not nearly as powerful or as easy to use as many people think. For one thing, they require high levels of correlation that are very unlikely in this kind of problem.

Haunted · Jun 23, 2008

kaching said:
You've run over more dogs in this industry than anyone else. How do you do it?

He has a very big car.

Stumpokapow said:
The whole point of this thread is to demonstrate that this is not the case. Their figures could be exactly right--they could be 50% off--they could be 100% off. The only time you ever know if they are correct or even in the same ballpark is when you have actual data, in which case why would you care because the actual data is right there?

This is the crux of the matter.

cvxfreak · Jun 23, 2008

Someone please forward this article to the major news sources if it hasn't already.

I would love for a major game magazine and news outlet to jump on board this article as well.

This isn't a call to nuke ******** off the internet, but their flawed methodology needs to be exposed ASAP.

dammitmattt · Jun 23, 2008

cordonbleu said:
Because regardless of their inadequate at times ability to get the numbers right, I dont know where the 'they make the numbers up' line comes from?

Because they do?

cordonbleu · Jun 23, 2008

dammitmattt said:
Because they do?

Come on? Whats the difference between believing 'it's just because' and the actual numbers?

Yaweee · Jun 23, 2008

I don't see what the big problem is. Yes, real data should replace the that site whenever possible, and, yes, their data carries a relatively low statical significance that is prone to errors, but why all of the hate towards their unofficial data/estimates?

If companies released sales data for all their products on a frequent basis, it would be awesome. But they don't, so exists.

VanMardigan said:
Actually, I hope for the opposite. If more organizations start to quote THAT SITE, then hopefully NPD speaks out to correct whatever mistake is there and we get actual data.

I agree that the site sucks, etc., but I also believe that NPD and their number hiding sucks. I know they have their reasons (profit), but I don't care. I want the numbers, and I want them for free. How much would it really cost NPD to provide the top 20 instead of the top 10 to us. Or top 30, or take a request for 10 random games and supply those numbers? Nothing. So, if other places follow the NY Time's lead and quote THAT SITE, hopefully NPD will react by releasing some data to invalidate the site.

So it's not like I want THAT SITE to flourish, but if that's what it takes to get more numbers from NPD, I'm all for it.

Yup. NPD has many shortcomings, which creates the perfect environment for a free and open but far less accurate competitor.

They tend to be within factor of 2, which, based on their very very very small sample size, ain't that bad.

VanMardigan · Jun 23, 2008

cvxfreak said:
Someone please forward this article to the major news sources if it hasn't already.

I would love for a major game magazine and news outlet to jump on board this article as well.

This isn't a call to nuke ******** off the internet, but their flawed methodology needs to be exposed ASAP.

Actually, I hope for the opposite. If more organizations start to quote THAT SITE, then hopefully NPD speaks out to correct whatever mistake is there and we get actual data.

I agree that the site sucks, etc., but I also believe that NPD and their number hiding sucks. I know they have their reasons (profit), but I don't care. I want the numbers, and I want them for free. How much would it really cost NPD to provide the top 20 instead of the top 10 to us. Or top 30, or take a request for 10 random games and supply those numbers? Nothing. So, if other places follow the NY Time's lead and quote THAT SITE, hopefully NPD will react by releasing some data to invalidate the site.

So it's not like I want THAT SITE to flourish, but if that's what it takes to get more numbers from NPD, I'm all for it.

charlequin · Jun 23, 2008

Yaweee said:
why all of the hate towards their unofficial data/estimates?

Because they are lies. I really don't get what's so hard to understand about this.

ioi claims to create numbers by statistical extrapolation from hard data, but he actually doesn't: instead, he makes up numbers based on hunches and secondary information (that is -- he does what Sales-Agers do when they're predicting) then adjusts his numbers back to more closely match NPD and Media Create figures when his predictions are wrong.

The amount of retail coverage needed to produce accurate numbers, statistically speaking, is drastically higher than anything ioi could possibly have access to; in addition to that, it takes a significant amount of past data and a lot of careful examination in order to produce a statistical model for transforming that data into a reasonable nationwide estimate, neither of which ioi has access to.

The problem essentially comes in because the site is duplicitous. It creates the impression that numbers are available when they are not, and tricks people into using their numbers for arguments (or journalistic purposes) when they're actually just made up.

I do think NPD is dropping the ball here (the way to combat this is to adopt a policy of releasing old data in a semi-public fashion so that sites like this one are shown up as inaccurate) but that doesn't put ioi in a better light; making up news or information is never an acceptable "solution" to a lack of said information.

botticus · Jun 23, 2008

VanMardigan said:
Actually, I hope for the opposite. If more organizations start to quote THAT SITE, then hopefully NPD speaks out to correct whatever mistake is there and we get actual data.

I agree that the site sucks, etc., but I also believe that NPD and their number hiding sucks. I know they have their reasons (profit), but I don't care. I want the numbers, and I want them for free. How much would it really cost NPD to provide the top 20 instead of the top 10 to us. Or top 30, or take a request for 10 random games and supply those numbers? Nothing. So, if other places follow the NY Time's lead and quote THAT SITE, hopefully NPD will react by releasing some data to invalidate the site.

So it's not like I want THAT SITE to flourish, but if that's what it takes to get more numbers from NPD, I'm all for it.

Odds are actual clients of NPD don't have any interest in what VGC provides, regardless of where they're quoted. If that's the case, then from a business sense, NPD has no need to worry about VGC. It really remains the people who have no interest in giving them any money throwing a hissy fit because we aren't getting free proprietary information. If we aren't happy with a top 10, why would we be happy with a top 20 or a top 30? I'd want the whole software list, personally.

Dirtbag · Jun 23, 2008

dialmydrive said:
It looked weird to me. I just made a valid observation, that's all.

Sorry to nitpick, but since you made a false claim about the guy... that would be an invalid observation AKA wrong assumption.

cordonbleu · Jun 23, 2008

charlequin said:
Because they are lies. I really don't get what's so hard to understand about this.

Because there is no proof that they are or are not lies.

Support NeoGAF

A Reasoned Criticism of Site I Can't Name

Gamasutra.

Member

Banned

Member

Sidhe / PikPok

Member

Member

Trucker Sexologist

Member

Member

Member

Member

Banned

Member

Member

Member

Member

Member

Member

Member

listen to the mad man

"GAF's biggest wanker"

Member

Member

Member

Member

listen to the mad man

Member

Banned

Member

Member

Member

listen to the mad man

Banned

Banned

Member

Member

Banned

Member

Member

Member

Member

Banned

Member

Member

has calmed down a bit.

Banned

Member

Member

Member

Similar threads