Tuesday, February 19, 2008

Would it kill pollsters to use error bars?


Here is the latest Gallup Daily Tracking Poll showing the national numbers for the Democratic primary. If we lay aside the fact that well over half the country has voted for the nominee, so national numbers are somewhat meaningless, it's a pretty cool poll. They interview roughly 1000 each of Democrat or Republican likely voters each and every single night, which obviously gives them a ton of data all sorts of ability to do zany analysis that makes pollsters go weak in the knees. One problem: sampling error. Now, it's not like everybody doesn't know it exists, since you will certainly hear "within the margin of error" or "statistically tied" a fair amount when people report about polls... but why even show a graph like this without making it clear?

A few minutes in SigmaPlot, adding in their plus or minus 3% sampling error, produced:

Now, my graph isn't nearly as pretty as Gallup's, but I think it's fairly straightforward for the layman to interpret. We're at least 95% certain that the true percentages are in between those bars. If the bars overlap then we can't be certain that the values are different, with more overlap meaning more uncertainty.

Keeping that in mind, it seems clear to me that Obama's been edging on getting some separation in (relatively useless) national numbers the last week. but it certainly hasn't been massive.... and getting in a tizzy about today's numbers seems a bit much... but I guess that isn't a very sexy conclusion.