Bad Polling, Part Two
Gary Langer, director of polling for ABC News, is not at all happy with the breakdown in the New Hampshire polls. He is calling for a serious look into what went wrong by all of the various polling outfits. Because something surely did go wrong.
But we need to know it through careful, empirically based analysis. There will be a lot of claims about what happened - about respondents who reputedly lied, about alleged difficulties polling in biracial contests. That may be so. It also may be a smokescreen - a convenient foil for pollsters who'd rather fault their respondents than own up to other possibilities - such as their own failings in sampling and likely voter modeling.
There have been previous races that misstated support for black candidates in biracial races. But most of those were long ago, and there have been plenty of polls in biracial races that were accurate. (For more on past problems with polls in biracial races, see this blog I wrote for Freakonomics last May.) And there was no overstatement of Obama in Iowa polls.
On the other hand, the pre-election polls in the New Hampshire Republican race were accurate. The problem was isolated to the Democratic side - where, it should be noted, we have not just one groundbreaking candidate in Barack Obama, but also another, in Hillary Clinton.
A starting point for this analysis will be to look at every significant Democratic subgroup in the New Hampshire pre-election polls, and see how those polls did in estimating the size of those groups and their vote choices. The polls' estimates of turnout overall will be relevant as well.
Langer calls the results in New Hampshire a fiasco. The fact that the Republican polls were dead on while the Democratic ones were a disaster does indicate that something went very badly wrong. But, as Langer also points out, we may never know exactly what. Was it a late break for Clinton or an overstatement of intent to vote by Obama supporters? There will be a lot of finger-pointing in the next few days, but we may never really know.
By NortonPete, Wednesday, 9 January , 2008 @ 8:12 am
One suggestion would be to look into the politics of the pollsters themselves. I believe they are young college age individuals who could possibly write down how they feel instead of the views of who they are polling.
By Mwalimu Daudi, Wednesday, 9 January , 2008 @ 8:51 am
Had this been a case of a Republican defeating a Democrat when the polls said otherwise, the Nutroots would have gone nuclear with conspiracy theories about stolen elections. That’s what happened after Bush unexpectedly beat Kerry, and that’s what happens every time Ron Paul gets clobbered when real votes are counted instead of online polls.
But this was a case of Democrat beating Democrat. Thus, the comparative silence of the Nutroots.
I have always wondered if many (perhaps a majority) of MSM journalists secretly hate and detest the Clintons. Because they find that the idea of holding Democrats accountable for their crimes is blasphemy, the MSM was forced into a permanent 24/7 defense of anything and everything William Jefferson Clinton and his wife did. As a result, the MSM got a deserved reputation as Clinton shills, and it damaged their credibility immensely.
Is it possible that some MSM journalists saw Obama as a ticket out of the Clinton sewer and into a liberal paradise, and that they tried to “help” him with friendly polling data designed to create a bandwagon effect?
By James, Wednesday, 9 January , 2008 @ 8:58 am
Oh, come on. Polls will never get it right. Any pol that listens to polling is an idiot (yeah, I know that means they’re all idiots). Every poll I’ve ever taken gives me a narrow selection of answers to choose from, NONE of which accurately portray what I think. Polls are poorly written. offer no good choices and are focused on, “Do you believe this or this.” I’d invariably select D: None of the above.
Most people are irritated by polling, don’t respond or don’t care how accurate their responses are. When I get a call and they’re conducting a poll, I know they are looking for liberal responses and ways to interpret any response to reinforce their own positions.
By Gaius, Wednesday, 9 January , 2008 @ 9:11 am
Polls are fairly useless when taken as a snapshot look - I tend to look at the trends - all of which showed Hillary Clinton fading. That is the surprise here.
By ted goldman, Wednesday, 9 January , 2008 @ 9:18 am
Bias of polsters may cause skewed results.
By bill-tb, Wednesday, 9 January , 2008 @ 9:22 am
Pollsters fool the electorate … again
IMHO — The independents went for McCain, to screw up the Republicans, after assuming the Obama lead was solid. Too cute by half.
NH has become a ‘gimmick’ primary with same day registration and no real party loyalty, it’s open for the IM scamming of pollsters.
Both Iowa and NH should be relegated to meaningless.
By Rich Horton, Wednesday, 9 January , 2008 @ 11:29 am
Pollsters most often do get things right and for a good reason: they have a long track record of how to do things the right way. Things go wrong when the process isn’t followed for whatever reason. For starters, not every poll had Obama out to a huge lead, only a couple did. Many (most?) polls had Obama/Clinton within the margin of error and Clinton won by 3 percentage points. Where is the surprise?
The surprise comes because the media has a pro-Obama script. (Hell, I just saw a MSNBC reporter say “Obama winning is a good story we can all get behind” implying that a Clinton victory isn’t.)
Now, the poll that had Obama ahead by 12 point on Monday is simply indefensible and is suggestive of bad faith. (It is statistically POSSIBLE but not probable…and polls are all about what is probable.)
By Americaneocon, Wednesday, 9 January , 2008 @ 1:39 pm
I wrote about this as well. I think the “empathy vote” for Hill following the “crying game” shifted a lot of undecideds her way. The Times of London mentioned the data:
“A rare moment of public emotion in a New Hampshire coffee shop was credited today with helping to bring Hillary Clinton back from the political dead and handing her victory in yesterday’s crucial presidential primary.
Analysis of exit polls from New Hampshire showed that women voters, traditionally her most loyal supporters, flooded back after deserting her for Barack Obama in last week’s Iowa caucuses. Mr Obama narrowly edged Mrs Clinton for the female vote in Iowa primary last week but yesterday she enjoyed a clear 13-point lead.
The psephologists will be poring over the New Hampshire results for some time to work out how the pollsters got it quite so wrong, but already it appears that a much-broadcast episode in which she welled up while talking to supporters in a Portsmouth restaurant could have earned Mrs Clinton a priceless sympathy vote in the state.
My analysis is here:
http://americanpowerblog.blogspot.com/2008/01/new-hampshire-results-stir-polling.html
By Nick, Wednesday, 9 January , 2008 @ 4:40 pm
Could the democratic pollsters be so wrong because of non resident voting just a thought
The republican pols seem to be pretty good
By Bleepless, Wednesday, 9 January , 2008 @ 8:40 pm
What? There’s a problem? Democrats lie. That’s all.
By William Hallowell, Friday, 11 January , 2008 @ 7:26 am
Interesting post. After Sen. Clinton’s unforeseen in NH victory, many people are questioning the pollsters. There is so much to consider when exploring what goes into political polling. Predicting elections is one of the most difficult challenges in survey work. Generally speaking, election surveys actually do work fairly well (it’s worth remembering that the polling on the Republican side in New Hampshire was pretty accurate). For all their flaws, surveys are still one of the best available tools for figuring out what the public wants. Check out our Public Agenda blog at http://publicagenda.org/headlines/headlines_blog.cfm for more on this!