With a backdrop of companies struggling to get both consistent and credible polling, it is interesting to analyse how different polling methods can skew the answers.
FiveThirtyEight generally takes an inclusive attitude towards polls. Our forecast models include polls from pollsters who use traditional methods, i.e., live interviewers. And we include surveys conducted with less tested techniques, such as interactive voice response (or “robopolls”) and online panels. We don’t treat all polls equally — our models account for the methodological quality and past accuracy of each pollster — but we’ll take all the data we can get.
This split, however, between live-interview polls and everything else, is something we keep our eye on. When we launched our general election forecasts in late June, there wasn’t a big difference in the results we were getting from polls using traditional methodologies and polls using newer techniques. Now, it’s pretty clear that Hillary Clinton’s lead over Donald Trump is wider in live-telephone surveys than it is in nonlive surveys.
We don’t know exactly why live-interview polls are getting different results than other types of surveys; there are a lot of potential causes and it’s something we’ll be digging into.
Online polling will become preferred due to the relatively low cost. But unless the results are of a reasonable quality, live interview polls will remain the more credible. We saw this recently when some rag commissioned a poll by a never-before-heard-of US company who had never polled on politics New Zealand previously. The results were ridiculous.
Nonlive polls may have presented a false signal of competitiveness in some red states. You probably remember hearing about how Kansas and Utah might be in play? That was mostly based on nonlive telephone interview polls. A forecast of just nonlive polls had Clinton within 5 percentage points of Trump in both states on July 1. The live-interview-only forecast, in contrast, had her down double-digits in both states. Since that time, the nonlive polls have come more in line with the live-interview polls — Trump’s regained a double-digit lead in both states even in the nonlive interview forecast.2
As the cases of Utah and Kansas suggest, I’d put more faith in the live-interview polls than in other types of surveys, all else being equal. Indeed, our forecast models do just that. If the gap between live-interview polls and everything else persists, though, we’ll need to explore what might be causing the split and which is more likely to be right.