NYT poll out of line with all others in NC Senate race

The New York Times, in partnership with the Kaiser Foundation, recently conducted an election survey in four southern states (Arkansas, Louisiana, Kentucky and North Carolina) and found all four Democrats running for the U.S. Senate were doing well.

Posted 11:17 p.m. Apr 26, 2014 — Updated 6:44 p.m. Jul 13, 2018

RALEIGH, N.C. — The New York Times, in partnership with the Kaiser Foundation, recently conducted an election survey in four southern states (Arkansas, Louisiana, Kentucky and North Carolina). Interestingly, it was conducted specifically for “The Upshot,” a new “big data” type blog the Times has launched to replace “FiveThirtyEight,” run by Nate Silver.

As Margin or Error readers will come to appreciate, I strongly believe that, when survey data is reported, it should also explain in great detail how the poll was conducted. The American Association for Public Opinion Research also stresses the importance of transparency.

Surprisingly, the only information I could find about the Times poll was the total number of respondents in each state and the fact that respondents were registered voters. Thus, I don’t know, for instance, if live interviewers were employed or if cellphones were part of the sampling frame. How a survey is administered affects it accuracy. Surveys that fail to include cell phone sub-samples, for example, exclude younger adults and other groups, and that can lead to biased results.

This poll attracted attention, though, because it found all four Democrats running for the U.S. Senate were doing well, and in the case of Mark Pryor in Arkansas, very well (He sported a surprising 10-point lead). Since political scientists agree that national conditions favor Republicans in 2014, some saw the results as surprising.

Some thought the poll included too many Democrats, which would bias support upwards for the Democratic candidates. At first glance, there seems to be merit to this claim. In all four states, respondents were asked whom they had voted for in 2012. In each state, all of which went for Romney in 2012, more respondents said they voted for Obama. Yet, as the Upshot explained to my satisfaction, there are well-known biases in measuring recalled vote choice that favor the winner. In other words, these results don’t demonstrate the survey had a skewed sample.

Yet, as I delved deeper into the results, one finding stood out like a sore thumb. North Carolina Democrat Kay Hagan’s approval rating in this poll is incongruent with all other polling that has been conducted in North Carolina this year. The Times poll found Hagan’s job approval was even with her disapproval (44-44). This is the most positive job approval ratio reported this year.

Just a few weeks ago, Public Policy Polling reported that Hagan’s approval was 41 percent and her disapproval was 48 percent (net -7 points). In early March, PPP found that 41 percent approved of Hagan and 50 percent disapproved (net -9 points). In fact, for all polls I found since February, excluding the Times poll, Hagan averaged 38 percent approval and 50 percent disapproval.

Either the Times poll is an outlier, or perhaps opinions are shifting and the Times is the first to detect the shift. Making things more complicated, the horse-race question in the Times poll is consistent with all other polling. The Times poll found a statistical dead-heat between Hagan and her two most likely Republican rivals (Thom Tillis or Greg Brannon). Hagan has a 2 percentage-point lead in both match-ups, but that “lead” is well within the margin of sampling error.

Suggest A Correction