How reliable are opinion polls?

"Surveys have not become more inaccurate, but rather more elaborate," says Thomas Wind, founder and managing director of the Institute for Target Group Communication (IfZ), who has been researching opinion polls. He believes there is no problem with the actual surveys, just with the increasing number of them.

"Every day a new figure or a new survey is published, and this has a knock-on effect," says Wind.

More polls, fewer participants

Political scientist and publisher Frank Brettschneider at the University of Hohenheim has conducted several studies on opinion research and election polls and confirms these observations. He says both the number of election polls and the reporting on them have increased more than tenfold in Germany since 1980.

But fewer and fewer people are willing to participate in preelection polls. According to the international study "Election polling errors across time and space," 20 years ago, 30% of those asked participated in polls; today, the rate is less than 10%.

The undecided

Whether a poll is representative is determined by the quality of the samples. One major problem here is how to deal with the group of undecideds.

"The lessons learned from the polls on the election of US President Donald Trump or thereferendum on Brexit in 2016 showed us that we can't get to certain target groups," explains expert Wind. The undecideds are excluded from many surveys,including the ARD Deutschlandtrend, which DW publishes once a month.

"The undecided pose a problem for us because they can make up 20% or more of the people polled," says Wind.

Political scientist Brettschneider goes on to explain: "For example, in the US, many Trump supporters refused to participate in polls on principle because, in their eyes, they were part of the establishment, just like the 'fake news media.'"

"Since they didn't vote in past elections, they didn't matter," Brettschneider says. "Now they participate in elections — and that leads to discrepancies between preelection polls and election results."

The same might apply to the far-right populist Alternative for Germany (AfD) and their supporters in Germany.

Evaluation and transparency

How a survey is conducted can also affect how representative a poll really is. Surveys via landline telephone are considered particularly meaningful because they can be attributed to a geographic area, and the interviewees usually have more time to talk. In cell phone surveys, on the other hand, it is unclear where the person is at the moment and whether he or she is really approachable. Online surveys are easier to evaluate but are often anonymous and capture only people who are on the internet a lot.

To improve sample quality, polling institutes, therefore, give greater weight to underrepresented groups. For public opinion researcher Wind, this is when "things start to get mysterious," because the institutes rarely publish their weighting procedures.

The evaluation and analysis of preelection polls are also susceptible to error. This is because the statistically calculated deviation rate of 2% to 3% is often only mentioned in the small print — if at all.

With a possible deviation of 2.5%, a party that is polled at 24%, for example, could in reality have a vote share of only 21.5%.

The precision of German results

Despite the numerous factors of uncertainty, polls for federal elections in Germany have been very accurate over the past 20 years. As shown by data published on the platforms wahlrecht.de and dawum.de, the polls exceeded the usual margin of error of 3% only twice in the last two decades.

An analysis of polling data since 2001 by the German national Die Zeit website shows that the election results of the individual parties were predicted with an average accuracy of 1.74%. Looking only at the last 10 years, the deviation has on average increased by 0.41%.

The center-right Christian Democratic Union (CDU) and their Bavarian sister party, the Christian Social Union (CSU), were significantly overestimated by all polling institutes ahead of the 2017 federal elections.

Zeit data journalist Christian Endt suspects that the increase in deviation could be related to the success of the AfD and the increasing fragmentation of the German party spectrum.

The international meta-analysis "Election polling errors across time and space" sees "no evidence of an increase in errors." Between 1940 and 1950, and 1960 and 1970, the study found a 2.1% deviation rate; since 2000, the rate has remained constant at 2%.

The authors note, however, that "the larger the party, the larger the deviation." 

The bottom line is that even though election polls may overall be more accurate than many believe, there are often discrepancies that exceed the usual 2 to 3% rate.

As was the case in the state election of Saxony-Anhalt on June 6, 2021. The polls had not predicted the clear victory for the CDU, but rather a neck-and-neck race between the CDU and the AfD.

And any lack of precision in media reporting of opinion poll results might also amplify inaccuracies.

This article has been translated from German.

While you're here: Every Tuesday, DW editors round up what is happening in German politics and society, with an eye toward understanding this year's elections and beyond. You can sign up here for the weekly email newsletter Berlin Briefing, to stay on top of developments as Germany enters the post-Merkel era.