They asked only 600 people?
Statisticians know they can get a pretty good idea about public opinion by asking questions of a small portion of the population. But they also know that method comes with room for error.Posted — Updated
When the polls diverge, which ones should you believe?
Although your first instinct might be to dismiss both surveys – or perhaps just the one showing your candidate faring worse – the truth is that both survey results are probably equally accurate. There are a few core factors common to all surveys that usually can explain their discrepancies, even significant ones.
Within the margin of error
Sampling error refers to the difference between our survey estimate and what we would have found had we been able to interview everyone in the population. When it is reported, it is described as “plus or minus” some percentage, for example: +/-3 percentage points. Larger samples have smaller margins of sampling error, and vice versa.
The graph below shows the relationship between the sample size and the margin of sampling error. Notice that increasing the size of the sample eventually leads to diminishing returns of increased precision. That is why most polls include either around 600 respondents or around 1,000 respondents. Polls are expensive, and if adding more respondents doesn’t significantly increase how accurate they are, then it is usually not worth the costs of doing so.
Sampling error can therefore explain how different survey results from two perfectly reputable firms are not actually distinct.
Imagine watching a news story on WRAL News where Poll X is reported to have found that Gov. Pat McCrory has a 51 percent approval rating. Later that day, you see another news story about Poll Y where McCrory is said to have a 46 percent approval rating. The likely headlines for each story might also contribute to the perceptions that these two polls are different, because a majority approves of McCrory in one case and a minority in the other.
If each poll included around 600 respondents, then the margin of sampling error for each estimate is about +/-4 percentage points. For Poll X, this means that anywhere between 47 percent and 55 percent of Americans approve of McCrory. Likewise, this means that anywhere between 42 percent and 50 percent of Americans approve of McCrory in Poll Y. Since these two estimates have ranges that overlap, they are, statistically speaking, indistinguishable.
Sampling error might be the only kind of error of survey error that can be precisely quantified, but it is far from the only type that affects the accuracy of surveys. Other critical factors include the method of contacting respondents, the timing of the polls, question wording, question order, answer options and the nature of the population being surveyed.
Copyright 2022 by Capitol Broadcasting Company. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.