"The world may have a polling problem." That's the headline on a blogpost by Nate Silver, the wunderkind founder of FiveThirthyEight. It was posted on 9:54 ET the night of May 7, as the counting in the British election was continuing in the small hours of May 8 UK Time.
That was an hour after the result in the constituency of Nuneaton made it clear that all the pre-election polls were wrong. Nuneaton, in the Midlands just east of Birmingham, was No. 28 on a list of 42 marginal two-party contests. Projections based on pre-election polls were that Labour would win 35 of these 42 seats. Instead, Conservatives won 34 of them.
Nationally, the pre-election polls predicted that Conservatives would win about 280 seats, barely ahead of Labour and far short of a 326-seat majority. The exit poll pegged them at 316. They ended up winning 331.
Something similar happened in 1992, when pre-election polls showed the two parties tied but Conservatives won by a 7.5-point margin. The most common explanation, advanced by Conservative analyst Rob Hayward: "Shy Tories" were unwilling to tell pollsters they favored the Conservative Party.
British pollsters made adjustments then, but as Hayward notes, they didn't work this year. Internal party polls apparently did better. American pollster Stanley Greenberg, working for Labour and using a longer questionnaire, found the party's numbers sagging. Australian consultant Lynton Crosby, running Conservatives' campaign, assured party leaders they would win 300 seats.
Silver correctly points out that Britain isn't the only country where recent polls have been off. Israeli polls projected that Benjamin Netanyahu's Likud Party would win 22 seats in the Knesset. Instead, it won 30 and Netanyahu formed a government.
And writing just after America's November 2014 off-year elections, Silver concluded that polls tilted toward Democrats. Reviewing the data then, I noted that in seriously contested statewide races, Democrats tended to get percentages of the vote close to their numbers in the polls. But many Republicans, notably in Kansas, Kentucky and Georgia, ran far ahead of the polls. "Shy Republicans," apparently.
Readers may have noticed that all these errors seem to come from one ideological direction. In nations where the dominant media -- the New York Times and the old-line TV networks here, the BBC in Britain, Haaretz in Israel -- lean left, opinion on the right has been understated in the polls. Evidently, some people don't want to identify themselves as troglodytes to telephone interviewers or even on robocalls.
But, as Silver notes, there are other, not necessarily ideological challenges facing pollsters. Current polling techniques were developed in countries with universal landline telephones and populations that answered the phone. Americans, Britons, Israelis and many others no longer live in such countries.
The Pew Research Center reported in 2012 that only 9 percent of its telephone calls resulted in completed interviews. Is that 9 percent an accurate representation of the larger population? Unclear; no poll can tell you the opinion of people who refuse to be polled.
Nor, despite the relative accuracy of Britain's exit poll this year, do exit polls always get it right. In American elections, the Edison/Mitofsky exit poll numbers are massaged by experts from its news organization clients gathered (and deprived of cellphones) in a room in Rockefeller Center at 5:00 p.m. ET Election Day.
Exit poll interviews have tended to tilt Democratic especially, according to a report by the late exit poll pioneer Warren Mitofsky, when the interviewer is a female graduate student. In the 2008 Democratic primaries, exit polls were so heavily tilted toward Barack Obama as to be useless.
The massaging gets the results closer to, but not quite on, the mark. On election night 2004 the exit poll showed 51-48 Kerry. The actual result was 51-48 Bush. On election night 2012, the evidently over-massaged exit poll came in 49-49. The actual result was 51-47 Obama.
Perhaps the next step is to go back toward the beginning, and to conduct interviews, as Gallup did in 1935, in person -- still fairly common practice when I started working for pollster Peter Hart in 1974. In-person interviews are standard practice in Mexico and other countries where telephones are not universal.
Polling provides useful information, but information whose reliability is often ephemeral and increasingly, it seems, limited. Good luck to British and other pollsters in the continuing struggle to get it right.
Join the conversation as a VIP Member