The Problems With Polling

The liberal blogoworld has been up in a tizzy since January's release of a Daily Kos poll showing that the Republican base holds some 'scary' views. The lefties are about to riot again with the teaser of a new Harris poll that

purports to show some similar results.

So how about it? Are Republicans just crazy and we shouldn't listen to them?

Well, probably not. The lefties would do well to remember Mark Blumenthal's analysis from February.

Acquiescence bias is the tendency of some respondents to select affirmative answers where the choice is whether to affirm or reject the statement presented (including "agree or disagree," "favor or oppose" and "yes or no" formats). This topic has been the subject of decades of study and debate among social scientists, and even though pollsters continue to rely on agree-disagree questions, academic survey researchers mostly agree that this format tends to produce more apparent agreement than questions offer a choice between two competing statements.

And later, in that same Blumenthal article,

Given that 10-20% of respondents tend agree with any statement (likely due to social norms), I went through the survey mentally subtracting 15 percentage points from every "yes" answer. That does leave some shocking numbers -- particularly as acquiescence tended to indicate support for gay rights, sex education, etc. -- but suggests that Birthers, for instance, may be outnumbered in the party (a slight consolation at best). I'm not saying this to suggest that the opinions being expressed even with a correction are reasonable, but I worry that not addressing this kind of issue is the reason so many people out there are skeptical of survey results in the first place.

This is not to discredit the conclusions of all of these polls. But it raises serious questions about the methodology and the intention of the designers to get crazy results.

***Update*** From ABC's polling expert Gary Langer:

The problems are fundamental. “Some people have said” is a biasing introductory phrase; it imbues the subsequent statements with an air of credibility – particularly when you don’t note that others say something else. (That approach can have problems of its own; the “some people” vs. “other people” format implies equivalence.)

The subsequent statements, for their part, are classically unbalanced – there’s no alternative proposition to consider. A wealth of academic literature, neatly summarized here, demonstrates that questions constructed in this fashion – true/false, agree/disagree – carry a heavy dose of what’s known as acquiescence bias. They overstate agreement with whatever’s been posited, often by a very substantial margin. (This reflects avoidance of cognitive burden, which tends to happen disproportionately with less-educated respondents, as is reflected in Harris’ results.)

Using all negative statements, rather than a mix of negative and positive ones, reflects another non-standard approach, one that can further bias responses. (The ordering of items, unclear in the Harris release, can be troublesome as well.)