Are State Polls Any Better Than They Were in 2016?

To most, Joe Biden’s clear lead in recent state polls suggests that he has the early edge in the race for the presidency. To others, it’s not so meaningful. After all, Hillary Clinton also held a clear lead in the state polls, and yet Donald Trump won the election.

Four years later, it’s fair to wonder whether there is a serious risk of another systematic polling error. The answer is not cut and dried.

There is always a chance of a systematic polling error, even when the reasons aren’t evident in advance. This time, there are obvious causes for concern. Many state polls suffer from the same methodological issues that were partly or even largely responsible for the miss four years ago, despite many opportunities to improve. Yet at the same time, many of the major causes of error in 2016 seem somewhat less acute.

What’s better than in 2016: undecided voters

One major source of the 2016 polling error is much less of a factor than it was four years ago: voters who are undecided or say they will vote for a minor-party candidate. These voters appeared to break overwhelmingly toward Trump, especially in the relatively white, working-class battleground states.

This time, far fewer voters are telling pollsters they’re undecided, and that means less room for a late shift among these voters to cause a polling error. At this point in 2016, about 20% of voters either supported a minor-party candidate or said they were undecided. Today, the number is about half that level.

What’s a little better than in 2016: education weighting

Another source of polling error was the failure of many state pollsters to adjust their samples to adequately represent voters without a college degree. Voters with a college degree are far likelier to respond to telephone surveys than voters without one, and in 2016 the latter group was far likelier to support Trump. Overall, weighting by education shifted the typical national poll by around 4 percentage points toward Trump, helping explain why the national polls fared better than state polls.

Four years later, weighting by education remains just as important. The gap in the preference of white voters with or without a college degree is essentially unchanged, despite the appeal Biden was supposed to have with less educated white voters.

In The New York Times/Siena College surveys conducted in October, Biden’s combined lead over Trump in the core six battleground states — Wisconsin, Pennsylvania, Michigan, Arizona, Florida and North Carolina — was 2 percentage points. That lead would have been 6 percentage points had it not been weighted by education or turnout (which correlates with education).

More pollsters are weighting by education today than four years ago. It could still be better, but overall, 46% of the more than 30 pollsters who have released a state survey since March 1 appeared to weight by self-reported education, up from around 20% of battleground state pollsters in 2016.

Some of the increase is because a handful of pollsters have decided to start weighting by education, a prominent example being the Monmouth University poll. But more of the change is because of the high volume of state online polls, which have always been likelier than state telephone surveys to weight by education.

What could be worse than in 2016: new online polls

There has been an explosion of new online-only state polls. Over the past month, there have been 13 such surveys, representing nearly half of the pollsters who have conducted state polls over this period. In contrast, only 10 online-only pollsters conducted surveys over the final three weeks of the 2018 election, which was about 10% of all of the pollsters who conducted surveys in that period.

Online polling isn’t necessarily bad. Many are sophisticated and comparable in quality to a typical live-interview telephone survey. But most of these new state polls take a simple approach: Contact the members of a large online panel, then weight those respondents by standard census demographics and maybe recalled vote choice in 2016 (more on that later). This is inexpensive and easy, but most pollsters have concluded that it’s not great. The panels just aren’t sufficiently representative, especially in small states, to expect a simple methodology to yield a high-quality result.

Until recently, few pollsters have tried to use this approach in state polling (Morning Consult is the most prolific example of a pollster that has done it nationally). But the early evidence suggests that these kind of state polls might lean to the left.

Perhaps the best early data is the AP/NORC/VoteCast polling ahead of the midterms, which combined a traditional telephone survey of 40,000 respondents with a large nonprobability online sample of 110,000 respondents. The online-only element of the survey was fairly comparable to most of the online surveys released in recent weeks, and it wouldn’t have fared well without calibration using the live-interview surveys. It would have overestimated the Democratic result by an average of about 5 percentage points across 71 races. Similarly, the new online polls tend to lean to the left of the state telephone polls so far this cycle. In polls conducted since March 15, Biden has run 6.6 percentage points ahead of Clinton’s margin in online state polls, compared with a gain of 3.7 points in live-interview telephone surveys. Notably, the latter figure comes very close to Biden’s gain in national polls — about 4 points — over the same period.

Over the long run, these polls might make up a smaller share of the battleground polling than they have so far. But it seems inevitable that new online polls will represent a larger share of state polls this cycle, and so far the best indications are that they’ll lean to the left.

What could be worse than in 2016: recalled vote weighting

More and more, pollsters with fairly mediocre sampling methods are relying on a new tool to bring their results closer to reality: recalled 2016 vote. Here, the pollster asks respondents whether they voted for Clinton or Trump, then adjusts the sample so that the recalled vote choice matches the actual result of the 2016 election.

Weighting on recalled vote choice certainly has its advantages. You could probably hammer even the worst survey into the ballpark. You could probably get a plausible poll result for Wyoming using a sample of New Yorkers this way.

But although this is a surefire way to reduce error, it is very hard to execute without risking a modest systematic bias. And here again, the bias would tend to be toward the Democrats.

There’s a large body of evidence suggesting that people are likelier to recall voting for the winner and less likely to recall voting for the loser. If so, polls weighting on recalled past vote would tend to be biased toward the party that lost the prior election. It seems that this effect is, at the very least, substantially diminished compared with a decade ago. Even so, it might still be there. In the Times/Siena polls from October, for instance, 6% of 2016 voters refused to say whether they backed Clinton or Trump. Those voters backed Biden by 2-to-1, suggesting that they were probably likelier to have supported Clinton.

Another issue is that today’s registered voters aren’t exactly the same as those of four years ago: In the interim some people have either died, reached voting age, or moved elsewhere. So it is not appropriate to assume that people who voted in 2016, and are now registered to vote, backed Trump or Clinton by the same margin as in the 2016 result.

It is hard to say whether this is generalizable, but overall 13% of the voters who took the Times/Siena polls in 2016 are no longer on their state voter file, and those voters backed Clinton by a 6-point margin, compared with a 1-point lead for Trump among those who remain registered in the state. Here again, recalled vote weighting might bias a poll by only 1 percentage point in Biden’s favor, but the risks start to add up.