• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.

Daily Kos founder: Our polling for the last year and a half is "bunk"

Status
Not open for further replies.

Gaborn

Member
At least, that's the upshot of this:

I have just published a report by three statistics wizards showing, quite convincingly, that the weekly Research 2000 State of the Nation poll we ran the past year and a half was likely bunk.

Since the moment Mark Grebner, Michael Weissman, and Jonathan Weissman approached me, I took their concerns seriously and cooperated fully with their investigation. I also offered to run the results on Daily Kos provided that they 1) fully documented each claim in detail, 2) got that documentation peer reviewed by disinterested third parties, and 3) gave Research 2000 an opportunity to respond. By the end of last week, they had accomplished the first two items on that list. I held publication of the report until today, because I didn't want to partake in a cliche Friday Bad News Dump. This is serious business, and I wasn't going to bury it over a weekend.

We contracted with Research 2000 to conduct polling and to provide us with the results of their surveys. Based on the report of the statisticians, it's clear that we did not get what we paid for. We were defrauded by Research 2000, and while we don't know if some or all of the data was fabricated or manipulated beyond recognition, we know we can't trust it. Meanwhile, Research 2000 has refused to offer any explanation. Early in this process, I asked for and they offered to provide us with their raw data for independent analysis -- which could potentially exculpate them. That was two weeks ago, and despite repeated promises to provide us that data, Research 2000 ultimately refused to do so. At one point, they claimed they couldn't deliver them because their computers were down and they had to work out of a Kinkos office. Research 2000 was delivered a copy of the report early Monday morning, and though they quickly responded and promised a full response, once again the authors of the report heard nothing more.

While the investigation didn't look at all of Research 2000 polling conducted for us, fact is I no longer have any confidence in any of it, and neither should anyone else. I ask that all poll tracking sites remove any Research 2000 polls commissioned by us from their databases. I hereby renounce any post we've written based exclusively on Research 2000 polling.

I want to feel stupid for being defrauded, but fact is Research 2000 had a good reputation in political circles. Among its clients the last two years have been KCCI-TV in Iowa, WCAX-TV in Vermont, WISC-TV in Wisconsin, WKYT-TV in Kentucky, Lee Enterprises, the Concord Monitor, The Florida Times-Union, WSBT-TV/WISH-TV/WANE-TV in Indiana, the St. Louis Post-Dispatch, the Bergen Record, and the Reno Gazette-Journal. In fact, just last week, in an email debate about robo-pollsters, I had a senior editor at a top DC-based political publication tell me that he'd "obviously" trust Research 2000 more than any automated pollsters, such as SurveyUSA. I didn't trust Research 2000 more than I trusted SUSA (given their solid track record), but I did trust them. I got burned, and got burned bad.

I can't express enough my gratitude to Mark, Michael, and Jonathan for helping bring this to light. Sure, our friends on the Right will get to take some cheap shots, and they should take advantage of the opportunity. But ultimately, this episode validates the reason why we released the internal numbers from Research 2000 -- and why every media outlet should do the same from their pollster; without full transparency of results, this fraud would not have been uncovered. As difficult as it has been to learn that we were victims of that fraud, our commitment to accuracy and the truth is far more important than shielding ourselves from cheap shots from the Right.

Soon, we'll have a new pollster (or pollsters) to work with, helping us to fulfill our vision of surveying races and issues that are often overlooked by the traditional media and polling outfits. As for Research 2000, the lawyers will soon take over, as Daily Kos will be filing suit within the next day or two.

Well, I'll give Moulitsas credit for the transparency, though avoiding a friday news dump is one thing, who follows news on TUESDAYS though? Still incredibly embarrassing though.
 

usea

Member
I was going to ask what Daily Kos was, but I just looked it up on wikipedia instead. How big is this polling thing on that site?

Either way, pretty embarrassing. The way he posted about it was good though.
 

JoeBoy101

Member
Don't know what he expects to gain from a lawsuit as the firm doesn't exactly sounds like they are swimming in dough.

I will say, despite any dislike of DailyKOS or Markos, he is dead on about this:

But ultimately, this episode validates the reason why we released the internal numbers from Research 2000 -- and why every media outlet should do the same from their pollster; without full transparency of results, this fraud would not have been uncovered.

I don't give credence to any poll I don't get full unvarnished crosstabs (i.e, raw data) for.
 

ToxicAdam

Member
I didn't really know anyone credible that took their polls at face value. It was always kind of viewed to be the "answer" to what Rassmussen was doing, although you kind of assumed they at least had hard data their were basing their results on and skewing it from there.
 
ToxicAdam said:
I didn't really know anyone credible that took their polls at face value. It was always kind of viewed to be the "answer" to what Rassmussen was doing, although you kind of assumed they at least had hard data their were basing their results on and skewing it from there.


I never took their polls seriously, even when posted in the PoliGaf thread.
 

StoOgE

First tragedy, then farce.
They pulled the polls a few weeks ago and said it was based on fivethirtyeight's rankings of pollsters that had R2K near the bottom.

My guess is that was a cover because they found out about this.

It is a shame, because R2K was a pretty big polling shop. Makes you wonder how many other pollsters out there are fake.

I know that Nate Silver uncovered at least one pollster who was fabricating results earlier this year.
 

Yaweee

Member
Hey, one of the guys that wrote the analysis is one of my old physics professors (Mike Weissman). He's awesome. Taught my Physics/Philosophy/Advanced Composition class ("Space, Time, and Matter").

He also busted a ton of people for plagiarizing their papers.
 

Stinkles

Clothed, sober, cooperative
Polling the general public on political matters is an exercise in futility. Ask them leading questions bout things they don't understand. tabulate data. SUCCESS. And by siccess, I mean failure.

Political Polling methodology is fundamentally flawed, since it includes predispositions and biases at both ends of the grinder and not only doesn't filter them, it encourages them.
 

Gaborn

Member
Jason's Ultimatum said:
I never took their polls seriously, even when posted in the PoliGaf thread.

Amazing that none of the poligafers that posted the polls have chosen to comment on this story.
 

GhaleonEB

Member
Gaborn said:
Amazing that none of the poligafers that posted the polls have chosen to comment on this story.
Ahem?

http://www.neogaf.com/forum/showpost.php?p=22135660&postcount=25313

http://www.neogaf.com/forum/showpost.php?p=22136765&postcount=25323

Though to be fair to others, PoliGAF has been very low traffic lately and this story just broke.

It's embarrassing, but hard to see where Kos went wrong. He selected a reputable pollster, commissioned scores of polls and published the full cross tabs of each, which made this kind of analysis possible. And he's fully disavowed the entire set of polls from R2K, right on up to asking poll aggregation sites like TPM and Pollster to purge all of them from their archives.
 
Gaborn said:
He'd be more honest if he specifically denounced this entire poll result and if his book he was writing at the time is based at all on those poll results he should write a forward disavowing that section of his book as well.

Stated in the comments on DKos that he's pulling all R2K-related stuff out of his book.

Edit: Of course that article and this whole mess in general is deeply embarrassing for Markos, even if it's mostly hard luck on his part. Relevant questions to ask him would be why R2K's significant lean and suspiciously inert data didn't invite a closer inspection for so long. But really, he wouldn't torpedo his own book at the last minute if he already suspected something fishy was going on but let it slide.

Also worth nothing that R2K got busted because they were the most transparent with their internal numbers, which is why DKos picked them in the first place. So who the fuck knows what other curiously weighted polls (Rass) are doing behind the curtain.
 

Gaborn

Member
badcrumble said:
Gaborn you're trying awfully hard to make this R2K fiasco an indictment of Moulitsas specifically.

It's his site, so he has to bear some of the responsibility for vetting of people and organizations he associates with, just like the politicians he covers have to take responsibility for their associations. He should have known better. I give him credit for the transparency and if it's accurate that he's pulling all R2K "data" out of his book then kudos to him. It all goes back to the basic question, who is watching the watchers? I hold critics of... well, almost anything really, to a higher standard because it encourages excellence for them.
 

syllogism

Member
Gaborn said:
It's his site, so he has to bear some of the responsibility for vetting of people and organizations he associates with, just like the politicians he covers have to take responsibility for their associations. He should have known better. I give him credit for the transparency and if it's accurate that he's pulling all R2K "data" out of his book then kudos to him. It all goes back to the basic question, who is watching the watchers? I hold critics of... well, almost anything really, to a higher standard because it encourages excellence for them.
Should have known better? R2K wasn't some no name outfit
 

Gaborn

Member
syllogism said:
Should have known better? R2K wasn't some no name outfit

Did I say it was? He should have had a statistician look at the data long before that if there was any reason to be suspicious, and clearly there was if the data was consistently out of step with other polls according to aggregators like 538.
 
I know it was obvious to us in Kentucky that the Kos numbers on Rand Paul were skewed. Apparently they were the only people who couldn't see the irregularities in the numbers compared to every other reputable polling group. Should be an interesting trial if R2K decides to play ball.
 

syllogism

Member
Gaborn said:
Did I say it was? He should have had a statistician look at the data long before that if there was any reason to be suspicious, and clearly there was if the data was consistently out of step with other polls according to aggregators like 538.
Even Nate didn't get suspicious until February this year and not because they were "out of step" as you claim. There certainly should be more scrutiny for pollsters and that is changing, but that has been a problem with the industry as whole and Kos shouldn't be faulted for trusting a pollster that has been around forever.
 
Gaborn said:
He'd be more honest if he specifically denounced this entire poll result and if his book he was writing at the time is based at all on those poll results he should write a forward disavowing that section of his book as well.
I'd say this does cover it.
While the investigation didn't look at all of Research 2000 polling conducted for us, fact is I no longer have any confidence in any of it, and neither should anyone else. I ask that all poll tracking sites remove any Research 2000 polls commissioned by us from their databases. I hereby renounce any post we've written based exclusively on Research 2000 polling.

I can't say I'm too surprised. Much of that Research 2000 polling seemed to not fully agree with other polling.


But I have to say that he is accepting this criticism, making it public, and changing ways is very admirable. This is truly the difference between Kos and his critics . . . accepting facts for what they say and drawing conclusions from them.
 

Gaborn

Member
syllogism said:
Even Nate didn't get suspicious until February this year and not because they were "out of step" as you claim. There certainly should be more scrutiny for pollsters and that is changing, but that has been a problem with the industry as whole and Kos shouldn't be faulted for trusting a pollster that has been around forever.

I'm not sure I ever said that Kos is the only organization which should ever investigate polling ever. This instance appears to catch a polling firm red handed making up data. Find more and I'll criticize those news organizations that use them for not catching it sooner too.
 
Gaborn said:
It's his site, so he has to bear some of the responsibility for vetting of people and organizations he associates with, just like the politicians he covers have to take responsibility for their associations. He should have known better. I give him credit for the transparency and if it's accurate that he's pulling all R2K "data" out of his book then kudos to him. It all goes back to the basic question, who is watching the watchers? I hold critics of... well, almost anything really, to a higher standard because it encourages excellence for them.

Interesting. Gaborn, you just made a strong argument as to why the government should be much bigger instead of relying upon outside contractors. After all, they bear some of the responsibility for vetting of people and organizations they associate with. Why not do the work in-house? :lol
 

syllogism

Member
Gaborn said:
I'm not sure I ever said that Kos is the only organization which should ever investigate polling ever. This instance appears to catch a polling firm red handed making up data. Find more and I'll criticize those news organizations that use them for not catching it sooner too.
Seems like a fairly pointless exercise as they've had and hundreds of clients and still do. I find it curious they were pretty transparent with their methodology and cross tabs. At least Strategic Vision was smart enough not to make cross tabs available.
 

Gaborn

Member
speculawyer said:
Interesting. Gaborn, you just made a strong argument as to why the government should be much bigger instead of relying upon outside contractors. After all, they bear some of the responsibility for vetting of people and organizations they associate with. Why not do the work in-house? :lol

Wat? The government as it is has rather onerous standards and processes (where they're not waived in the name of favoritism) for the purpose of, for example, awarding contracts for different jobs. If you meet the standards (that is, if you can do the job well) you're hire able. It also doesn't take a massive government to run a criminal background check these days, that's computerized. I really don't understand where you think it'd take a large amount of effort on the part of the government to vett an organization like that. Of course, some people (labor secretaries mostly it seems judging by the number that withdraw their nominations based on hiring illegal immigrants the last couple of presidents) are NOT vetted in the executive branch, or are vetted "too late" in terms of an announcement.

If anything this is more bizarre then the response I was expecting from Fone Bone attacking pollsters like Rasmussen and asking me to attack them (which would be ironic based on his vehement opposition to proving you are who you say you are before voting, his defense being "no voter fraud of that kind has been proven")
 
Gaborn said:
If anything this is more bizarre then the response I was expecting from Fone Bone attacking pollsters like Rasmussen and asking me to attack them (which would be ironic based on his vehement opposition to proving you are who you say you are before voting, his defense being "no voter fraud of that kind has been proven")
I'm glad that the fact that he didn't say anything of the sort didn't stop you from posting your response, at least.
 

GhaleonQ

Member
StoOgE said:
They pulled the polls a few weeks ago and said it was based on fivethirtyeight's rankings of pollsters that had R2K near the bottom.

My guess is that was a cover because they found out about this.

It is a shame, because R2K was a pretty big polling shop. Makes you wonder how many other pollsters out there are fake.

I know that Nate Silver uncovered at least one pollster who was fabricating results earlier this year.

^That.
 

GhaleonEB

Member
Gaborn said:
It's his site, so he has to bear some of the responsibility for vetting of people and organizations he associates with, just like the politicians he covers have to take responsibility for their associations. He should have known better. I give him credit for the transparency and if it's accurate that he's pulling all R2K "data" out of his book then kudos to him. It all goes back to the basic question, who is watching the watchers? I hold critics of... well, almost anything really, to a higher standard because it encourages excellence for them.
I agree with what you said, other than the bold. Should have known better? R2K had a decent rep, and are used by scores of other media outlets. What should he have known? That R2K was going to defraud him?

As to bearing some responsiblity, you are acting as if the post you quoted in the OP wasn't written by Kos..
 

FoneBone

Member
Gaborn said:
If anything this is more bizarre then the response I was expecting from Fone Bone attacking pollsters like Rasmussen and asking me to attack them (which would be ironic based on his vehement opposition to proving you are who you say you are before voting, his defense being "no voter fraud of that kind has been proven")
I didn't even remember that thread until now. You're fucking adorable.
 
Gaborn said:
Wat? The government as it is has rather onerous standards and processes (where they're not waived in the name of favoritism) for the purpose of, for example, awarding contracts for different jobs. If you meet the standards (that is, if you can do the job well) you're hire able. It also doesn't take a massive government to run a criminal background check these days, that's computerized. I really don't understand where you think it'd take a large amount of effort on the part of the government to vett an organization like that.
Ah . . . you're right. They shouldn't take everything in-house . . . just need good regulations and enforcement of those regulations. :lol :lol :lol
 

Gaborn

Member
speculawyer said:
Ah . . . you're right. They shouldn't take everything in-house . . . just need good regulations and enforcement of those regulations. :lol :lol :lol

Show me where I ever said I oppose any and all regulations ever? I think there are aspects of our society that are regulated to our detriment (See: The way the FDA handles terminally ill patients access to experimental drugs as a prime example) but I'm not going to go point by point over the hundreds of thousands of regulations the federal government has and say they're all wrong (some are of course, others are just arbitrary and plain "weird," etc). Tilting at windmills isn't a good look for you.

Ghaleon - I didn't say the post wasn't written by Kos, I acknowledged it was and gave him credit for his transparency there. That doesn't mean it's inappropriate to slam him for this as well. Just like I still think Sanford is a fucking MORON for the whole Argentina tryst, the fact that he apologized to his wife doesn't end it.
 

FoneBone

Member
I can't even find a thread on that subject that I posted in. :lol I'm not saying I didn't, but why is this so memorable to you? Hell, what does polling even have to do with voter fraud?
 

Stumpokapow

listen to the mad man
Gaborn said:
Ghaleon - I didn't say the post wasn't written by Kos, I acknowledged it was and gave him credit for his transparency there. That doesn't mean it's inappropriate to slam him for this as well. Just like I still think Sanford is a fucking MORON for the whole Argentina tryst, the fact that he apologized to his wife doesn't end it.

Wouldn't actually being guilty of something be the major distinction between Sanford and Kos here?
 

Evlar

Banned
Did you just compare Kos being defrauded by his pollster to a politician using public funds to enable a cross-continental tryst?

(directed at Gaborn)
 

Tamanon

Banned
Gaborn said:
Amazing that none of the poligafers that posted the polls have chosen to comment on this story.

The only real PoliGAFfer that posts polls is Diablos, and that's down-day Gallups only.
 

FoneBone

Member
Stumpokapow said:
Wouldn't actually being guilty of something be the major distinction between Sanford and Kos here?
Between this and your bizarrely specific recollection of two-year-old debates, are we witnessing a mental breakdown here? :lol
 

Gaborn

Member
Stumpokapow said:
Wouldn't actually being guilty of something be the major distinction between Sanford and Kos here?

True, not a perfect analogy. Ok, better example. Grant appointed a lot of friends and drinking buddies to his cabinet when he was President. The result? a huge amount of corruption and good ole boy politics. He wasn't (apparently) directly involved in it but he still got the blame. Why? Because he was the President and it's his responsibility.

Fone - Far from it. Incidentally I'm relatively sure we had the same discussion last year as well. What can I say, your position was just so completely insane it was THAT memorable.
 

GhaleonEB

Member
I'm a wee bit of a polling junkie, so I've been reading up on the fallout since Kos posted his article earlier today. It's been quite the bombshell in the polling world.

Nate Silver has three posts up about it: he finds the methodology in the analysis compelling, details his own concerns with R2K polling (which he's written about before) and just posted a cease and desist letter from the R2K lawyer who wants him to STFU. It doesn't look like that's going to happen.

Over at Pollster, Charles Franklin chimes in on the implications for the polling industry, as does Mark Blumenthal. He mentions that Franklin will be posting his thoughts on the analysis provided to Kos shortly.

I'll post other news in here as it unfolds; Kos should be filing suit in the next few days, and R2K has said they'll counter sue. Whee!
 

Stumpokapow

listen to the mad man
Gaborn said:
True, not a perfect analogy. Ok, better example. Grant appointed a lot of friends and drinking buddies to his cabinet when he was President. The result? a huge amount of corruption and good ole boy politics. He wasn't (apparently) directly involved in it but he still got the blame. Why? Because he was the President and it's his responsibility.

Yes, this analogy is better, because at least we're in the ballpark. In both cases we're talking about someone who was an administrator overseeing an organization that engaged in corruption and deception. But it's still not good.

Firstly, a boss is accountable for his employees in a way that is manifestly different than the way that a person is accountable for their contractors.

Secondly, a supervisor that understands and can theoretically spot corruption in their subordinates is accountable in a way that's not true of a supervisor who is supervising very technical employees. In other words, while anyone was smart enough to say "boy, something seems a little off here", it's pretty unlikely that anyone except people well versed in statistics (see Nate Silver's arguments for why R2K's polling was bunk toward written and sent to Mark Blumenthal in February 2010--I did a minor in math and while I understand what he said, I'd have never generated that data myself) would have actually been able to come to the conclusion things have come to. Grant, on the other hand, while not directly involved in the corruption, was clearly able to discover that it was happening.

Thirdly and perhaps most importantly, the president of the United States of America is not comparable to the administrator of a small business. The President should be held to a higher standard and they are.

Fourthly, Grant didn't come out and say "Boy howdy, I fucked up"--his apologies were always hedging, they didn't provide any real rationalization, etc.. In this way, your Sanford analogy was better because his confession was relatively earnest, but it was so poor in every other regard that it still doesn't apply.
 

GhaleonEB

Member
A couple round ups on the news. Pollster has a round up of reaction here, and Mark Blumenthal guages the charges; both he and others he consulted find the evidence pretty damning, though he's cautioning that a more thorough examination of the data is required and will occur.

Nate Silver ran an analysis he believes illustrates more statistical indications of fraud. Someone over at Pollster disagreed with the analysis.

Meanwhile, R2K is refusing to release their data set behind the polls, which is as damning as anything else I've seen so far. The founder has said his business has been dealt a "fatal blow". Though he's had serious financial troubles before.
 

Gaborn

Member
Stumpokapow said:
Yes, this analogy is better, because at least we're in the ballpark. In both cases we're talking about someone who was an administrator overseeing an organization that engaged in corruption and deception. But it's still not good.

Firstly, a boss is accountable for his employees in a way that is manifestly different than the way that a person is accountable for their contractors.

I agree with this up to a point. Ultimately Moulitsas is a few different things in this regard. He is de facto a form of journalist, disseminating information to the public. Journalists have a higher standard of care for the information they release (and when there is a fuck up, or a lack of vetting the journalist takes the hit). He's also an author, relying on a specific polling firm he pays for results (as opposed to finding polling results conducted by a neutral party he's not affiliated with. This latter concern I had incidentally, which I noted when I mentioned he apparently had an extensive poll done and implied that it might be part of the thesis of his book, at least as supporting data, has been mitigated to the point it's not an issue when he totally pulled that book prior to it's actual release.

So he's not just a "boss" some paper pusher who gives generic orders, this is not, say, a construction firm contracting with Kruger Industrial Smoothing (the company George worked for in the later seasons of Seinfeld that was incompetent even by HIS standards), this is a journalist posting what is purported to be polling data. If you use that data as part of your website, and if you planned to use that same data in your book, and you paid for the creation of that data, sorry, that's going to need a little higher standard in my opinion.

Secondly, a supervisor that understands and can theoretically spot corruption in their subordinates is accountable in a way that's not true of a supervisor who is supervising very technical employees. In other words, while anyone was smart enough to say "boy, something seems a little off here", it's pretty unlikely that anyone except people well versed in statistics (see Nate Silver's arguments for why R2K's polling was bunk toward written and sent to Mark Blumenthal in February 2010--I did a minor in math and while I understand what he said, I'd have never generated that data myself) would have actually been able to come to the conclusion things have come to. Grant, on the other hand, while not directly involved in the corruption, was clearly able to discover that it was happening.

First I want to say that no analogy is going to be perfect. The deeper you delve into any analogy and pick it apart you're going to find imperfections because every situation is unique and it's not necessarily simple to develop an adequate analogy that is ideal for a particular situation. With that said, sure, I assume Moulitsas missed it, It happens, it happened to everyone. It happened to the Times with Jayson Blair, it happened to the New Republic and Andrew Sullivan in particular with Ruth Shalit and Stephen Glass. Anyone can make a mistake or even be outright fooled by anyone else. I agree with you even that this should be a warning to news organizations in general that they need to look at their pollsters and ensure the data is being gathered and gathered honestly using standard methods

With that all said, that doesn't excuse Moulitsas entirely. The NYT and Sullivan (who I actually like and find quite readable, though I often disagree with him) both suffered a hit to their respective reputations with their inability to notice the plagiarizers they had hired, it's only natural the same should happen to Moulitsas. I don't think that's picking on him. I think the need to pick apart every statement someone who criticizes someone you like makes is going to blow that up beyond normal criticism but I don't think I've been overly harsh on Moulitsas. He got snookered, it happens. What I think in the future will be interesting is the vetting process he uses for any future pollster he hires. I'd suggest the full release of the data set to Moulitsas as well as of course the poll results they interpret from it. That is what he's paying for after all.

Thirdly and perhaps most importantly, the president of the United States of America is not comparable to the administrator of a small business. The President should be held to a higher standard and they are.

The Daily Kos is a small business? It's a rather substantial source for information with many prominent posters in various fields, I really think you're underselling them. I do agree the President is held to a higher standard though, I suppose I was constrained by my inability to think of another situation quite like this.


Fourthly, Grant didn't come out and say "Boy howdy, I fucked up"--his apologies were always hedging, they didn't provide any real rationalization, etc.. In this way, your Sanford analogy was better because his confession was relatively earnest, but it was so poor in every other regard that it still doesn't apply.

As I said, an analogy is what it is. An imperfect way to relate how you feel about a particular situation to another situation. No two situations are going to be identical, this in particular was a fairly unique situation as media scandals go.
 

Gaborn

Member
OuterWorldVoice said:
Calm down. You're hysterical.

34fmsnl.jpg


I'm picturing this guy telling me that :lol :lol :lol :lol :lol :lol :lol :lol :D
 

FoneBone

Member
Gaborn said:
Fone - Far from it. Incidentally I'm relatively sure we had the same discussion last year as well. What can I say, your position was just so completely insane it was THAT memorable.
You can't even point me to the thread where I said that supposedly memorable statement. Also I recommend doing 15 seconds' worth of fact checking before you go on for paragraphs about Andrew Sullivan's failings.
 
Status
Not open for further replies.
Top Bottom