Understanding forecasting and polls


People have criticised me in the comments on posts where I post Nate Silver’s predictions and where the numbers call the election for Obama. They also mistake my calling the election for Obama at this stage as support for Obama. They should not, but that is a separate post.

Ther eis a reason I prefer Nate Silver’s predictions…he is usually right…and uncannily so. Sure you can point to individual polls that show Mitt Romney beating Obama, but that is a single poll.

Nate Silver uses far more sophisitcated modelling than simple polling. Conveniently he has explained what he does in a recent post. It is very enlightening. Especially the comparison of his methodology with prediction markets and betting agencies.

Before people diss what Nate Silver has to say, based ont eh numbers and his unique methodologies, they should really learn and understand what makes his models tick. It is way, way more than just running a “poll of polls” which is what some think.

I sometimes get asked whether I bet money on my forecasts — I don’t, since I would consider it a conflict of interest — or failing that, whether I would recommend a bet on them relative to the odds on offer at Intrade or Betfair.

My answer is probably unsatisfying. I think modeling a presidential election is a pretty hard problem. I think futures markets and sports books (like markets of any kind) can certainly go wrong. But I also think that the statistical methods can go wrong: all of them rely on a set of assumptions and choices made by the forecaster.

Some choices, in my view, are clearly better than others. One or two of the statistical methods, for instance, assumes that the outcome in each state is independent of the outcome in the next one. Ohio might move in one direction — and Michigan, just as easily, in the opposite one.

That’s simply not a credible assumption. The failure to appreciate correlations in risk is one of the things that led to the recent financial crisis. A change in economic conditions, or a substantial gaffe or scandal in the campaign, is likely to be reflected to some degree in all states, and move all of their numbers in the same direction. Our model assumes that the uncertainty in different states is largely, but not entirely, correlated. If you believe the contrary, you probably ought not be let anywhere near a job function in which you are asked to manage risk — although the credit-ratings agencies might be happy to hire you.

These pet peeves aside, elections forecasting is a challenging problem. More often, the assumptions in a model are intrinsically going to be educated guesses rather than being demonstrably right or wrong.

So my default is this: Bet on Vegas relative to the FiveThirtyEight model, but bet on the FiveThirtyEight model relative to Vegas. If you take the average between the FiveThirtyEight model and the consensus betting lines, you’d get about a two-in-three chance of Mr. Obama winning another term.