Friday, May 30, 2008

The Problem with Polling

In the past, I've complained that even though Rasmussen and Gallup are doing these snazzy new "daily tracking polls" with all sorts of Super Statistical Powers, they don't account for margin of error in their pretty graphs... leading many reporters to breathlessly talk about how a 2 point difference in a 3 day rolling average of a sample size of 1000 means that "Clinton is surging" or "Obama was undamaged" or whatever. This is nonsense. I'll grant that when someone looks at polls and says something like that they generally are talking about trends and not absolute differences, but let's leave that aside for today.

A 2 point or 4 point or even a 6 point "lead" is not a significant difference in a sample size of 1000. That's all there is to it, though if you want an illustration of the issues of sampling error, then see here(hattip to TNR commenter AlanSP). You can, of course, attach all sorts of caveats to my above statement... the statistics I quoted were for 95% confidence... surely we don't need P=0.05 to hedge our bets? No, we probably don't... but we never get presented with the option of caring about the confidence of the estimate do we?

Anyway, that's my grumble for the day.