Notes on the Atrocities
Like a 100-watt radio station, broadcasting to the dozens...

Tuesday, April 20, 2004  

The Post has a poll out today, and it's causing some head-scratching. People don't trust that Bush is telling them truth, yet his numbers on dealing with terrorism, the economy are up, his approval's up, and he's polling better against Kerry. All of this despite pretty amazing claims of administration wrongdoing. So what gives?

People have mainly been attacking this from an analysis point of view, offering explanations in interpretation. I think those are part of the picture, but I think there's another factor--polling accuracy. As you all well know, polls have lately been wildly inaccurate. Even when they got the trending right in the past couple elections, the margins were pretty far off (well outside the margin of error). Why?

In research, there are a couple of basic proofs to determine how accurate the findings are: reliability and validity. Reliability describes whether a measurement gives approximately the same result in repeated tests. Validity gauges whether a statistic measures what it is supposed to measure. In polling, we have great reliability--polls consistently measure the same thing. The pollsters will tell you they have great validity, too--the math lines up. But reality sometimes trumps math: polls aren't good predictors anymore. I have a couple theories.

Two months before Iowa, John Kerry was nearly bankrupt and Howard Dean was leading by twenty points (or something) in Iowa polls. But as the election approached, their calculations changed. It's impossible for pollsters to factor in the time, because they can't know what will happen in the next six months. Like weather forecasting, the further out the election is, the less reliable the poll numbers are.

Polling companies still rely on telephones to gather data. There are two problems here. The first is that they don't call cell phones. That's a major sampling error. Imagine if you were only polling people without computers. The second is a self-selection problem; the people who do answer phones and do participate in a survey self-select: you don't find out what Americans think, you find out what Americans who are willing to take a survey think. As we become ever more resistant to commercialization, we are more resistant to the mechanism of polling. What was reliable in an earlier age isn't anymore.

I'll make this point by example: in this WaPo survey, Ralph Nader was polling at 6%. But in 2000, Nader only got 1% of the votes, and he'll surely do worse this year. That shows that something like 90% of the people who said they were Nader supporters weren't telling pollsters what they'll actually do. They were telling thme something else. People, especially this early out, aren't necessarily responding in ways pollsters expect. What do they believe? Who knows?

The most important thing we see from these numbers is the trending. Two trends are worth noting: people don't trust Bush, but their global attitudes toward him have returned to pre-Clarke levels. Both interesting findings. But as to whether it credibly predicts who people will vote for in November, the answer to that is no.

posted by Jeff | 10:36 AM |
Blogroll and Links