Debunking Oversampling

Rabid Republican supporters have constantly attacked me for not following their particular party line in predicting a Romney win. I even have a bet with Leighton Smith that Obama will win, not that I want him to win but simply because the math, the polls and the facts do not support any other contention.

In effect it is me with the safe bet. Nonetheless I get constant emails pointing out the error of my ways, constant comments doing the same and people pointing me to discredited loons who have come up with a conspiracy that suggests that all polling companies with the exception of Rasmussen are colluding to keep Obama president in conjunction with the liberal media paymasters.

Me? I prefer facts. Nate Silver looks at the current conspiracy of the day…oversampling:

In 2004, Democratic Web sites were convinced that the polls were biased toward George W. Bush, asserting that they showed an implausible gain in the number of voters identifying as Republicans. But in fact, the polls were very near the actual result. Mr. Bush defeated John Kerry by 2.5 percentage points, close to (in fact just slightly better than) the 1- or 2-point lead that he had on average in the final polls. Exit polls that year found an equal number of voters describing themselves as Democrats and Republicans, also close to what the polls had predicted.

Since President Obama gained ground in the polls after the Democrats’ convention, it has been the Republicans’ turn to make the same accusations. Some have said that the polls are “oversampling” Democrats and producing results that are biased in Mr. Obama’s favor. One Web site,, contends that even Fox News is part of the racket in what it says is a “trend of skewed polls that oversample Democratic voters to produce results favorable for the president.”

People forget that these conspiracies follow the cycles. They also forget that they are wrong.

The criticisms are largely unsound, especially when couched in terms like “oversampling,” which implies that pollsters are deliberately rigging their samples.

But pollsters, at least if they are following the industry’s standard guidelines, do not choose how many Democrats, Republicans or independent voters to put into their samples — any more than they choose the number of voters for Mr. Obama or Mitt Romney. Instead, this is determined by the responses of the voters that they reach after calling random numbers from telephone directories or registered voter lists.

Pollsters will re-weight their numbers if the demographics of their sample diverge from Census Bureau data. For instance, it is typically more challenging to get younger voters on the phone, so most pollsters weight their samples by age to remedy this problem.

So what about the charge of partisan bias:

If the focus on “oversampling” and party identification is misplaced, however, FiveThirtyEight does encourage a healthy skepticism toward polling. Polling is difficult, after all, in an era in which even the best pollsters struggle to get 10 percent of households to return their calls — and then have to hope that the people who do answer the surveys are representative of those who do not.

So perhaps we should ask a more fundamental question: Do the polls have a history of being biased toward one party or the other?

The polls have no such history of partisan bias, at least not on a consistent basis. There have been years, like 1980 and 1994, when the polls did underestimate the standing of Republicans. But there have been others, like 2000 and 2006, when they underestimated the standing of Democrats.

What does Nate Silver do that is different?

We have an extensive database of thousands of polls of presidential and United States Senate elections. For the presidency, I will be using all polls since 1972, which is the point at which state-by-state surveys became more common and our database coverage becomes more comprehensive. For the Senate, I will be using all polls since 1990.

That is a pretty impressive amount of data, and one of the reasons why Nate Silver is the most accurate political statistician and why I follow his predictions. What does the data actually say, rather than the partisan hackery:

In the 10 presidential elections since 1972, there have been five years (1976, 1980, 1992, 1996 and 2004) in which the national presidential polls overestimated the standing of the Democratic candidate. However, there were also four years (1972, 1984, 1988 and 2000) in which they overestimated the standing of the Republican. Finally, there was 2008, when the average of likely voter polls showed Mr. Obama winning by 7.3 percentage points, his exact margin of victory over John McCain, to the decimal place.

In all but three years, the partisan bias in the polls was small, with the polling average coming within 1.5 percentage points of the actual result. (I use the term “bias” in a statistical sense, meaning simply that the results tended to miss toward one direction.)

Nate Silver also looked at state polls which showed similar traits. His conclusion over all is:

On the whole, it is reasonably impressive how unbiased the polls have been. In both presidential and Senate races, the bias has been less than a full percentage point over the long run, and it has run in opposite directions.

That does not mean the pollsters will necessarily get this particular election right. Years like 1980 suggest that there are sometimes errors in the polls that are much larger than can be explained through sampling error alone. The probability estimates you see attached to the FiveThirtyEight forecasts are based on how the polls have performed historically in practice, and not how well they claim to do in theory.

But if there is such an error, the historical evidence suggests that it is about equally likely to run in either direction.

Nor is there any suggestion that polls have become more biased toward Democratic candidates over time. Out of the past seven election cycles, the polls had a very slight Republican bias in 2010, and a more noticeable Republican bias in 1998, 2000 and 2006.

They had a Democratic bias only in 2004, and it was very modest.

Still, 2004 went to show that accusations of skewed polling are often rooted in wishful thinking.

I can almost taste my lunch now.

Do you want:

  • ad-free access?
  • access to our very popular daily crossword?
  • access to Incite Politics magazine articles?

Silver subscriptions and above go in the draw to win a $500 prize to be drawn at the end of March.

Not yet one of our awesome subscribers? Click Here and join us.

As much at home writing editorials as being the subject of them, Cam has won awards, including the Canon Media Award for his work on the Len Brown/Bevan Chuang story.  And when he’s not creating the news, he tends to be in it, with protagonists using the courts, media and social media to deliver financial as well as death threats.

They say that news is something that someone, somewhere, wants kept quiet.   Cam Slater doesn’t do quiet, and as a result he is a polarising, controversial but highly effective journalist that takes no prisoners.

He is fearless in his pursuit of a story.

Love him or loathe him.  But you can’t ignore him.