Debunking Oversampling

Rabid Republican supporters have constantly attacked me for not following their particular party line in predicting a Romney win. I even have a bet with Leighton Smith that Obama will win, not that I want him to win but simply because the math, the polls and the facts do not support any other contention.

In effect it is me with the safe bet. Nonetheless I get constant emails pointing out the error of my ways, constant comments doing the same and people pointing me to discredited loons who have come up with a conspiracy that suggests that all polling companies with the exception of Rasmussen are colluding to keep Obama president in conjunction with the liberal media paymasters.

Me? I prefer facts. Nate Silver looks at the current conspiracy of the day…oversampling:

In 2004, Democratic Web sites were convinced that the polls were biased toward George W. Bush, asserting that they showed an implausible gain in the number of voters identifying as Republicans. But in fact, the polls were very near the actual result. Mr. Bush defeated John Kerry by 2.5 percentage points, close to (in fact just slightly better than) the 1- or 2-point lead that he had on average in the final polls. Exit polls that year found an equal number of voters describing themselves as Democrats and Republicans, also close to what the polls had predicted.

Since President Obama gained ground in the polls after the Democrats’ convention, it has been the Republicans’ turn to make the same accusations. Some have said that the polls are “oversampling” Democrats and producing results that are biased in Mr. Obama’s favor. One Web site,, contends that even Fox News is part of the racket in what it says is a “trend of skewed polls that oversample Democratic voters to produce results favorable for the president.”

People forget that these conspiracies follow the cycles. They also forget that they are wrong.

The criticisms are largely unsound, especially when couched in terms like “oversampling,” which implies that pollsters are deliberately rigging their samples.

But pollsters, at least if they are following the industry’s standard guidelines, do not choose how many Democrats, Republicans or independent voters to put into their samples — any more than they choose the number of voters for Mr. Obama or Mitt Romney. Instead, this is determined by the responses of the voters that they reach after calling random numbers from telephone directories or registered voter lists.

Pollsters will re-weight their numbers if the demographics of their sample diverge from Census Bureau data. For instance, it is typically more challenging to get younger voters on the phone, so most pollsters weight their samples by age to remedy this problem.

So what about the charge of partisan bias:

If the focus on “oversampling” and party identification is misplaced, however, FiveThirtyEight does encourage a healthy skepticism toward polling. Polling is difficult, after all, in an era in which even the best pollsters struggle to get 10 percent of households to return their calls — and then have to hope that the people who do answer the surveys are representative of those who do not.

So perhaps we should ask a more fundamental question: Do the polls have a history of being biased toward one party or the other?

The polls have no such history of partisan bias, at least not on a consistent basis. There have been years, like 1980 and 1994, when the polls did underestimate the standing of Republicans. But there have been others, like 2000 and 2006, when they underestimated the standing of Democrats.

What does Nate Silver do that is different?

We have an extensive database of thousands of polls of presidential and United States Senate elections. For the presidency, I will be using all polls since 1972, which is the point at which state-by-state surveys became more common and our database coverage becomes more comprehensive. For the Senate, I will be using all polls since 1990.

That is a pretty impressive amount of data, and one of the reasons why Nate Silver is the most accurate political statistician and why I follow his predictions. What does the data actually say, rather than the partisan hackery:

In the 10 presidential elections since 1972, there have been five years (1976, 1980, 1992, 1996 and 2004) in which the national presidential polls overestimated the standing of the Democratic candidate. However, there were also four years (1972, 1984, 1988 and 2000) in which they overestimated the standing of the Republican. Finally, there was 2008, when the average of likely voter polls showed Mr. Obama winning by 7.3 percentage points, his exact margin of victory over John McCain, to the decimal place.

In all but three years, the partisan bias in the polls was small, with the polling average coming within 1.5 percentage points of the actual result. (I use the term “bias” in a statistical sense, meaning simply that the results tended to miss toward one direction.)

Nate Silver also looked at state polls which showed similar traits. His conclusion over all is:

On the whole, it is reasonably impressive how unbiased the polls have been. In both presidential and Senate races, the bias has been less than a full percentage point over the long run, and it has run in opposite directions.

That does not mean the pollsters will necessarily get this particular election right. Years like 1980 suggest that there are sometimes errors in the polls that are much larger than can be explained through sampling error alone. The probability estimates you see attached to the FiveThirtyEight forecasts are based on how the polls have performed historically in practice, and not how well they claim to do in theory.

But if there is such an error, the historical evidence suggests that it is about equally likely to run in either direction.

Nor is there any suggestion that polls have become more biased toward Democratic candidates over time. Out of the past seven election cycles, the polls had a very slight Republican bias in 2010, and a more noticeable Republican bias in 1998, 2000 and 2006.

They had a Democratic bias only in 2004, and it was very modest.

Still, 2004 went to show that accusations of skewed polling are often rooted in wishful thinking.

I can almost taste my lunch now.


THANK YOU for being a subscriber. Because of you Whaleoil is going from strength to strength. It is a little known fact that Whaleoil subscribers are better in bed, good looking and highly intelligent. Sometimes all at once! Please Click Here Now to subscribe to an ad-free Whaleoil.

  • Alloytoo

    It’s been pretty obvious since the primaries that the Republicans do not have a Ronald Reagan up their sleeve, which is what’s required to unseat an encumbant President.

  • Hard1

    Well put , Cameron . Better the hard Right than the easy Rabid…

    No Republican presidential candidate has ever captured victory without the help of Ohio. Mitt Romney’s road to continuing that trend was hit with another uphill poll on Sunday.

    A Columbus Dispatch survey shows the former Massachusetts governor trailing President Barack Obama by nine points, 51 percent to 42 percent. The Dispatch notes that this is the fifth recent poll showing Romney trailing by at least five points in the Buckeye State.

  • MrV

    Is accurate sampling even possible, given that it is very difficult to know in advance what the demographic turnout will be? The US has such low voter turnout, it comes down to which groups are more motivated to show up and vote.

    • Alloytoo

      A fair chunk of the 47% might just show up just to piss Romney off.

  • JC

    There’s some points still missing here..

    How many realise that John McCain “won” the vote on the night of the 2008 election but lost massively to Obama on the write in absentee votes collected the month or so before?

    Up to 30-40% of the electorate does these write in votes and Obama has spent massively to create an early impression that Romney has no chance and thus influence these early voters. And when you have polls showing greater swing state support than he enjoyed in 2008 you can see how the pollsters can help O but then swing to more reasonable polling late in the write in process to regain credibility.

    Democrats have been by far the main write in voters so Obama and his tame pollsters have them targetted early.. trouble is, first results of the write in this time show a big downturn in Dems compared to Reps.

    Second point.. its all about turnout, and the historically high turnout of minorities and youth in 2008 is hardly likely to be repeated when these groups have done the worst out of the Obama policies.

    Basically I’m still strongly favouring Romney because:

    *He represents the 53% who pay net tax
    * He represents the 60% which Gallop recently identified as distrusting the media
    * He represents the groups that traditionally break against the incumbent in a bad economy.


    • Did you even read the post?

      • fozzie

        Nah JC only watches Fox News and the Mormon Times.

  • Karl Rove in this video clip explains what the major concern is with the current US election polls and their sampling methods. He concentrates on Ohio and Florida:

    This article in the National Review Online is about the dwindling numbers of Democratic voters in 2012 compared to 2008 in some key states. Note carefully the changes in numbers that are listed:
    Finally going Back to Rove he notes in his latest weekly summary of polls that a key factor may be which party has the better ground game when it comes to getting their voters out on election day:
    My own view is that Obama is leading though probably only by 3-4%. That isn’t a big lead and this is by no means over.

  • Beans

    Alas, I don’t think the polls are skewed or are being deliberately rigged to favour the Sun God. Even Rasmussen agrees that the Dems have a 2 to 4 point advantage. SG genuinely has a 4-7 point lead over Rumney, which has meandered around there despite (or perhaps because of) the many attempts by the MFM to kneecap the Repubs. However, I will say a few things:

    1) The polls don’t make any sense. SG ahead even though Romney ties or leads with independents? What?

    2) The media is running an active psychological warfare operation to turn down the vote. I understand, they need to to preserve whatever remnants of credibility they have left in case the Sun God gets humiliated.

    3) I don’t think the debates will change much. They won’t be the silver bullet Romney needs.

    4) Continuing with the campaign as is will result in Romney losing. There doesn’t appear to be a strategy, unless bombarding swing states with ads at the last minute is what’s being planned.

    On the subject of polls, the one the media won’t talk about is the recent finding that a record number of Americans distrust the media. I can’t imagine why that would be.

  • Krumkavy

    Former Obama campaign consultant Nate Silver, whose black-box statistical model for the New York Times has shown an outsized lead for the president (and whose 2010 model gave Democrats a 20 percent chance of holding the House on Election Day), as well the proliferation of surveys conducted by Public Policy Polling, which does regular polling for the hyper-partisan union, the Service Employees International Union. I think Leighton is really going to enjoy that lunch the more you talk it up..

  • 2ndAmendment

    There is a rather simpler explanation – many more Dems agreeing to talk to pollsters, than the GOP who hate the MSM and pollsters.

    If the turnout is halfway between 2008 and 2010 Romney wins.
    If Romney wins, welfare ends in the US.
    If welfare ends in the US, it ends in NZ too.

    For all these reasons, we must not give “help and comfort to the enemy in time of war”

  • SaggyNaggy

    If Obama is so far ahead, why has he shut his offices in North Carolina, and why is he still campaigning in Wisconsin?

    Obama knows where the polls are at, and they are not where Nate Silver says they are.