Why the Polls Were Wrong

It’s not good news. From Vox:

What the hell happened with the polls this year?

Yes, the polls correctly predicted that Joe Biden would win the presidency. But they got all kinds of details, and a number of Senate races, badly wrong . . .

To try to make sense of the massive failure of polling this year, I reached out to the smartest polling guy I know: David Shor, an independent data analyst who’s a veteran of the Obama presidential campaigns who formerly operated a massive web-based survey at Civis Analytics before leaving earlier this year. . . . Shor’s been trying to sell me, and basically anyone else who’ll listen, on a particular theory of what went wrong in polling that year, and what he thinks went wrong with polling in 2018 and 2020, too.

The theory is that the kind of people who answer polls are systematically different from the kind of people who refuse to answer polls — and that this has recently begun biasing the polls in a systematic way.

This challenges a core premise of polling, which is that you can use the responses of poll takers to infer the views of the population at large — and that if there are differences between poll takers and non-poll takers, they can be statistically “controlled” for by weighting according to race, education, gender, and so forth. . . . If these two groups do differ systematically, that means the results are biased.

The assumption that poll respondents and non-respondents are basically similar, once properly weighted, used to be roughly right — and then, starting in 2016, it became very, very wrong [note: of course, 2016 was when Txxxx began poisoning the political process]. People who don’t answer polls, Shor argues, tend to have low levels of trust in other people more generally. These low-trust folks used to vote similarly to everyone else. But as of 2016, they don’t: they tend to vote for Republicans.

Now, in 2020, Shor argues that the differences between poll respondents and non-respondents have gotten larger still. In part due to Covid-19 stir-craziness, Democrats, and particularly highly civically engaged Democrats who donate to and volunteer for campaigns, have become likelier to answer polls. It’s something to do when we’re all bored, and it feels civically useful. This biased the polls, Shor argues, in deep ways that even the best polls (including his own) struggled to account for.

Liberal Democrats answered more polls, so the polls overrepresented liberal Democrats and their views (even after weighting), and thus the polls gave Biden and Senate Democrats inflated odds of winning. . . .

Dylan Matthews

So, David: What the hell happened with the polls this year?

David Shor

So the basic story is that, particularly after Covid-19, Democrats got extremely excited and had very high rates of engagement. They were donating at higher rates, etc., and this translated to them also taking surveys, because they were locked at home and didn’t have anything else to do. There’s some pretty clear evidence that that’s nearly all of it: It was partisan non-response. Democrats just started taking a bunch of surveys [when they were called by pollsters, while Republicans did not].. . .

Dylan Matthews

You mentioned social trust. Walk me through your basic theory about how people who agree to take surveys have higher levels of social trust, and how that has biased the polls in recent years.

David Shor

For three cycles in a row, there’s been this consistent pattern of pollsters overestimating Democratic support in some states and underestimating support in other states. This has been pretty consistent. It happened in 2018. It happened in 2020. And the reason that’s happening is because the way that [pollsters] are doing polling right now just doesn’t work. . . .

Fundamentally, every “high-quality public pollster” does random digit dialing. They call a bunch of random numbers, roughly 1 percent of people pick up the phone, and then they ask stuff like education, and age, and race, and gender, sometimes household size. And then they weight it up to the census, because the census says how many adults do all of those things. That works if people who answer surveys are the same as people who don’t, once you control for age and race and gender and all this other stuff.

But it turns out that people who answer surveys are really weird. They’re considerably more politically engaged than normal. . . . [They] have much higher agreeableness [a measure of how cooperative and warm people are], which makes sense, if you think about literally what’s happening.

They also have higher levels of social trust. . . . It’s a pretty massive gap. [Sociologist] Robert Putnam actually did some research on this, but people who don’t trust people and don’t trust institutions are way less likely to answer phone surveys. Unsurprising! This has always been true. It just used to not matter.

It used to be that once you control for age and race and gender and education, that people who trusted their neighbors basically voted the same as people who didn’t trust their neighbors. But then, starting in 2016, suddenly that shifted. . . . These low-trust people still vote, even if they’re not answering these phone surveys.

Dylan Matthews

So that’s 2016. Same story in 2018 and 2020?

David Shor

The same biases happened again in 2018, which people didn’t notice because Democrats won anyway. What’s different about this cycle is that in 2016 and 2018, the national polls were basically right. This time, we’ll see when all the ballots get counted, but the national polls were pretty wrong. If you look at why, I think the answer is related, which is that people who answer phone surveys are considerably more politically engaged than the overall population. . . .

Normally that doesn’t matter, because political engagement is actually not super correlated with partisanship. That is normally true, and if it wasn’t, polling would totally break. In 2020, they broke. There were very, very high levels of political engagement by liberals during Covid. You can see in the data it really happened around March. Democrats’ public Senate polling started surging in March. Liberals were cooped up, because of Covid, and so they started answering surveys more and being more engaged.

This gets to something that’s really scary about polling, which is that polling is fundamentally built on this assumption that people who answer surveys are the same as people who don’t, once you condition on enough things. . . . But these things that we’re trying to measure are constantly changing. And so you can have a method that worked in past cycles suddenly break. . . .

There used to be a world where polling involved calling people, applying classical statistical adjustments, and putting most of the emphasis on interpretation. Now you need voter files and proprietary first-party data and teams of machine learning engineers. It’s become a much harder problem.

Dylan Matthews

. . . Pollsters need to get way more sophisticated in their quantitative methods to overcome the biases that wrecked the polls this year. Am I understanding that right?

David Shor

. . . A lot of people think that the reason why polls were wrong was because of “shy Txxxx voters.” You talk to someone, they say they’re undecided, or they say they’re gonna vote for Biden, but it wasn’t real. Then, maybe if you had a focus group, they’d say, “I’m voting for Biden, but I don’t know.” And then your ethnographer could read the uncertainty and decide, “Okay, this isn’t really a firm Biden voter.” That kind of thing is very trendy as an explanation.

But it’s not why the polls were wrong. It just isn’t. People tell the truth when you ask them who they’re voting for. They really do, on average. The reason why the polls are wrong is because the people who were answering these surveys were the wrong people. If you do your ethnographic research, if you try to recruit these focus groups, you’re going to have the same biases. They recruit focus groups by calling people! Survey takers are weird. People in focus groups are even weirder. Qualitative research doesn’t solve the problem of one group of people being really, really excited to share their opinions, while another group isn’t. As long as that bias exists, it’ll percolate down to whatever you do.

Unquote.

I think what this means is that even if you correct for the low number of Republicans who answer polls, you’re still in trouble, because you’re still polling the kind of Republicans who answer polls (the relatively nice ones). Your sample of Republican voters doesn’t represent Republicans who vote.

This does not bode well for the two Senate races in Georgia. Polls show the Democrats are a few points behind. When the run-off elections occur in January, the polls will probably be wrong again, unless the pollsters have quickly figured out how to solve this problem.

Conspiracy Theories, Plausible or Not

Last year, a company called Public Policy Polling asked 1,247 registered voters in the United States their opinions regarding what the company called “conspiracy theories” (although some of the questions, such as “Do you believe aliens exist, or not?” don’t necessarily refer to conspiracies). Here are some of the more interesting questions and answers, beginning with the least popular “theories”. The poll, which is described here, had a margin of error of 2.8%.

1) Do you believe that shape-shifting reptilian people control our world by taking on human form and gaining political power to manipulate our societies, or not?

11% of the respondents said Yes or weren’t sure (happily, that means 89% said No).

2) Do you believe that the exhaust seen in the sky behind airplanes is actually chemicals sprayed by the government for sinister reasons, or not?

13% said Yes or weren’t sure (not surprisingly, 87% said No).

3) Do you believe the moon landing was faked, or not?

16% said Yes or weren’t sure.

4) Do you believe Paul McCartney actually died in a car crash in 1966 and was secretly replaced by a lookalike so the Beatles could continue, or not?

19% said Yes or weren’t sure.

5) Do you believe the United States government knowingly allowed the attacks on September 11th, 2001, to happen, or not?

22% Yes or weren’t sure.

6) Do you believe media or the government adds secret mind-controlling technology to television broadcast signals, or not?

30% Yes or weren’t sure.

7) Do you believe global warming is a hoax, or not?

49% Yes or weren’t sure.

8) Do you believe a UFO crashed at Roswell, New Mexico, in 1947, and the US government covered it up, or not?

53% Yes or weren’t sure (21% said Yes).

9) Do you believe that a secretive power elite with a globalist agenda is conspiring to eventually rule the world through an authoritarian world government, or New World Order, or not?

53% Yes or weren’t sure (but 28% said Yes).

10)  Do you believe there is a link between childhood vaccines and autism, or not?

54% Yes or weren’t sure (20% said Yes).

11) Do you believe the Bush administration intentionally misled the public about the possibility of weapons of mass destruction in Iraq to promote the Iraq War, or not?

56% Yes or weren’t sure (44% said Yes).

12) Do you believe that there was a conspiracy (whether or not it included Lee Harvey Oswald) behind the assassination of President Kennedy?

75% Yes or weren’t sure (although I rephrased the question to make it consistent with the others).

One might conclude from some of these results that an uncomfortably large percentage of the American electorate is absolutely nuts. However, we should keep in mind what David Hume said about miracles. If someone claims to have seen a miracle, it’s much more likely that he or she is lying or confused than that a miracle actually occurred. Likewise, if roughly 10% of voters are open to the possibility that shape-shifting reptiles walk among us or that those vapor trails up in the sky are a government plot, we should conclude that many who gave those answers were either confused about the question or messing with the pollsters.

On the other hand, if shape-shifting reptiles do control many of the world’s governments, that would explain a lot. And I for one say “Welcome to our reptilian overlords!”.

(Note: that’s supposed to be a giant insect in the picture behind Kent Brockman, but somebody decided to add a guy’s face.)

A couple of these poll results are more troubling. Half of us think that global warming is a hoax or are open to that possibility, and a similar percentage think that vaccines do or may cause autism. It’s understandable why some might think that the experts are mistaken about global warming, but to believe that thousands of scientists are or could be conspiring to mislead the rest of us is incredibly dumb and also likely to impede efforts to address the problem. Similarly, one might wonder if there is a possible link between vaccines and autism, but to take that idea seriously enough to ignore the medical consensus and not vaccinate one’s children is both foolish and dangerous.

There’s a natural tendency to be skeptical about whatever the official story is. None of us want to be taken in by the powers that be. Governments, corporations and supposed experts lie more than they should and conspiracies do sometimes occur. There’s also nothing wrong with keeping an open mind on controversial topics when there is evidence on both sides.

So I’m comfortable being with the skeptical majority who think people in the Bush administration lied about those weapons of mass destruction or at least decided it wasn’t worth knowing the truth. I’m also comfortable saying that Lee Harvey Oswald may have participated in a conspiracy or been used by one. I think he acted alone but wouldn’t be surprised either way (unless Vice President Johnson had something to do with it – that would be a big surprise). The good news is that Bush, Cheney and Rumsfeld can’t do much damage anymore and anyone who was involved in the Kennedy assassination is probably gone or will be soon.

A probably unrelated note: Having been on the Central Coast of California for the past week or so, I can report that the state has not completely dried out. In fact, casual observation revealed very little evidence of the major drought they’re having. Shops in one small town were directing everyone to some new portable toilets on the main street, and the outdoor showers at one of the beaches were turned off. I wouldn’t go so far as to say that the drought is a hoax being carried out by conspiring reptilian shape-shifters, but you never know.