On right, left, polling data doesn't prop up partisan conclusions

The conclusions drawn from a pair of recent polls on attitudes toward teacher pay increases show the perils of how question order and wording can affect answers and the conclusions pollsters will draw.

Posted 3:18 p.m. Jun 2, 2014 — Updated 6:45 p.m. Jul 13, 2018

RALEIGH, N.C. — The Republican majority in the state Senate recently passed a budget that tied increasing K-12 teacher salaries to surrendering tenure rights. Groups on the left and the right of the political spectrum both released polling data suggesting their preferred policy position was supported by majorities of North Carolinians.

Is ideology influencing polling about teacher pay?

Let’s start with Civitas, an ideologically conservative organization that uses National Research Inc. to conduct some of its polls. Civitas issued a news release claiming that two of their polls, one from March of this year and another from March 2013 "show North Carolina voters support alternatives to the tenure system that pay larger raises to the best teachers."

Yet, when I found the survey and looked at all of the questions that were included, it was clear Civitas had cherry-picked a few questions with questionable wording and placement to reach that conclusion. For example, Civitas asked many other questions that undercut their press release. For one, respondents were asked if they thought teacher pay was too high, too low or about right. Just 2 percent said too high, and 79 percent said too low. Respondents were also asked whether North Carolina spent too much, too little or the right amount on public education, and 71 percent said not enough.

Nor did Civitas mention that they also asked respondents if they supported a new teacher pay increase to $35,000, noting this amounted to a 14 percent raise. Supporters of this pay increase outnumbered opponents, 74 percent to 21 percent. This result indicates an overwhelming majority support an unconditional pay raise for teachers.

In their very next question, Civitas introduced information about tenure as they asked people if they supported or opposed it. Although the balance of reactions was even – 45 percent supported and 45 percent opposed – I am opposed to feeding respondents information about a topic while they are being asked how they feel about it. In general, surveys should avoid giving respondents information because it helps them to answer it, even if they are uninformed, uninterested and did not hold any prior opinion to be measured.

A graphic from Pew Research shows how question order can influence survey respondent answers.

Giving respondents information means the choice of what information to include or exclude and can dramatically shape responses not just on that question but also to subsequent questions on the same topic. Question order can also influence respondent answers. For example, a 2003 Pew survey found varying degrees of support for gay marriage depending on which questions were asked and in what order.

As it turns out, the next question inquired about raises coupled with ending tenure, although it wasn’t identical to legislation just approved by the Senate. Civitas asked, "A change in the state law requires local school systems to end tenure and offer the top performing 25 percent of the system’s teachers four-year contracts with a guaranteed pay raise each year of the contract. Do you support or oppose this new system of improving teacher quality and increasing the salaries for the best teachers?"

That wording is not neutral. It encourages affirmative responses. First, it is hard for people who already said they approve of pay raises for all teachers to turn around and oppose pay raises for the "best teachers." Likewise, it is difficult to oppose a system that is said to improve teacher quality, even though that is a claim made by advocates of the policy and not an objective fact. Unsurprisingly, 64 percent supported the plan, while 29 percent were opposed.

So, what should we believe? The simple answer is that North Carolinians support pay raises for teachers, and their support does not depend on ending tenure.

After that, it is hard to say anything else because I’m concerned that question wording and question order influenced the attitudes measured about the idea of pay raises coupled with ending tenure. The ideal survey question would directly pose these options to respondents and have them choose whether pay raises should be dependent on ending tenure or be independent of that policy. Oddly, nobody has asked this.

PPP publicity raises similar questions

Meanwhile, Public Policy Polling, a local polling firm that works mainly for Democratic candidates and causes, claims nearly the opposite.

PPP argues "teacher pay raises are a top priority for voters in the state" and "they want to see teachers get more than Republicans in Raleigh are proposing." PPP, though, did not ask respondents if pay raises were a top priority. In fact, respondents provided contradictory answers about the scope of pay raises, so it isn’t accurate to say people want a bigger raise than Republicans’ proposed.

In their May 2014 poll, PPP asked three questions about teachers. The first one asked, "How much do you think the starting salary should be for a public school teacher in North Carolina: $30,000, $35,000, $40,000, $45,000, or more?" Currently, the starting salary is $30,800, and the Senate proposal would increase that to $35,000 within two years.

I would have preferred a version of this question that allowed respondents say any dollar amount, rather than giving them a narrowly, fixed range. Nevertheless, a total of 40 percent of respondents answered $35,000 or less, suggesting a large minority would find the Senate proposal acceptable. On the other hand, a majority – 56 percent – chose $40,000 or more, above what Republicans have proposed. It is also worth pointing out that just 14 percent selected $30,000 as the "right" starting salary, which is basically where its at now.

PPP then asked, "How much of a raise do you think all North Carolina teachers should receive in this year’s budget: a 1 percent raise, 2 percent raise, 3 percent raise, 4 percent raise, 5 percent raise, 6 percent raise, 7 percent raise, more than 7 percent, or no raise at all?" Like before, I dislike the answer options as opposed to an open-ended format, especially in this case, since the total raise offered amounts to about 14 percent, 7 percent which comes in the first year. Yet, 7 percent is the maximum of the range offered, other than to say "more." With so many choices, there was no consensus about the exact size any raise should be. While only 2 percent said there shouldn’t be any raise, 74 percent of respondents selected a 6 percent or less, percentages that are smaller than what the Senate approved.

A final PPP question asked people to choose between pay raises and a tax cut, which presumably is how they concluded pay raises were a priority. Respondents were asked, "With limited money in the budget, do you think  it is more important for the General Assembly  to raise teacher pay or cut taxes?" Pay raises were preferred by a 54 percent to 36 percent margin, with 10 percent unsure.

Most people probably don’t know the starting salary of teachers, so perhaps we should discount the fact only a slim majority preferred a base salary above the Republican Senate plan. On the other hand, most people were fine with a 6 percent raise or less, well below the actual proposal. And, finally, preferring teacher pay increases over tax cuts is not a direct indication that this issue is a priority to voters, although it might be.

In the end, my concerns with PPP have less to do with the polling than with their interpretation of it. This is different than suggesting they loaded the deck in their favor. In fact, this is largely my complaint with Civitas too. Both groups asked good questions, even if they both failed to directly ask about the Senate plan that just passed. Rather, I’ve concluded that both groups took liberties with the results that seem to correspond with their ideological convictions.

Suggest A Correction