Margin Of Error: Breaking down the polls

Margin Of Error: Breaking down the polls

Trust an Internet poll? The New York Times did

Posted August 4
Updated August 5

House Speaker Thom Tillis is running, and, polls show, trailing, in the race to represent North Carolina in the U.S. Senate.

— Only one poll that I know of since mid-May finds Republican Thom Tillis with a lead over incumbent Democrat Kay Hagan in North Carolina's U.S. Senate race.

That poll, conducted by “YouGov,” was commissioned by CBS/NY Times and reported a 1 percentage point lead for Tillis. Of course, a 1 point lead is really a dead heat (i.e., “too close to call”). Nevertheless, this poll has sparked controversy and soul-searching among industry professionals.

The poll is controversial because of how it was conducted, and the decision by The New York Times to stand behind it, not because of its results, although the data are clearly in dispute, too.

YouGov conducts its surveys online – completely. Adults are recruited to take the surveys, but the critical factor is that they are not chosen through probability methods where all persons in the defined population (say, in North Carolina) have an equal chance of being asked for their opinion.

It is an understatement to point out that probability-based sampling methods have enabled large-scale representative surveys to be successful for almost a century.

Nate Cohn recently explained, in a post for the Times' “Upshot” blog, the issues at stake. This is a big deal.

As reported in Politico, the American Association of Public Opinion Research (AAPOR) has issued a statement highly critical of The New York Times and CBS News for promoting survey results obtained through a methodology devoid of a theoretical underpinning.

I don’t have a lot to add to what has already been said, especially after the Pew Research Center did a great job of identifying the key aspects of the controversy and Upshot Editor David Leonhardt agreed to an interview with Chris Cillizza from The Washington Post. I thought this development was worthy of highlighting, though, because the decision for a major news organization to promote survey data collected via a non-probability sampling technique is, well, newsworthy.

If I had anything to contribute, it would be the following.

I used to be “old school” about sampling methods – until a few months ago. By that, I mean I would have dismissed the YouGov poll almost out of hand, as many others have done.

Yet, the truth of the matter is that probability sampling is itself broken, and polls that portend to use it depend on weighting their data afterwards. I am unaware of any national or state-level poll that is reported without weighting because the raw data are inaccurate. Response rates for phone surveys have dipped into the single digits. A growing segment of the population cannot be reached over a land line. It's not clear to me how entirely different a poll conducted over the phone where nearly half the population can’t be reached that way – or won’t answer – is from a poll where people opt in to the sample and are contacted only via email.

The only online panel that I personally have confidence in using is GFK, formally “Knowledge Networks.” The difference is that panelists are recruited via probability sampling, finding adults via land phones, cellphones and the Internet, who are representative of the population, and only then sampling from within this massively large panel. The debate about the validity of YouGov is far from settled, but this really is the cutting edge of either a brand new way to conduct reliable surveys or the tip of the iceberg about to sink the Titanic.

17 Comments

This blogpost is closed for comments.

Oldest First
View all
  • WralCensorsAreBias Aug 5, 3:59 p.m.

    Bill Brasky, we clearly know that's your mission in life. A lib is as a lib says.

    But I'm sure you'll blame it on Tillis.

    Or Bush.

  • Bill Brasky Aug 5, 2:12 p.m.

    What happened to my earlier posts WRAL? Did I offend a cry baby liberal? Are you censoring words... View More

    — Posted by foolmeonce

    Remember, when you don't like something or you don't understand it.. label it Liberal, Marxist, perhaps as you say fascist. Its much easier than looking up those words in the dictionary.

  • krimson Aug 5, 1:19 p.m.

    The significant difference between probability sampling methods and convenience sampling methods... View More

    — Posted by jurydoc

    5/3's of the commentators here don't understand fractions... How do you expect them to understand statistics???

  • jurydoc Aug 5, 11:56 a.m.

    I guess I shouldn't be surprised that in an article about statistical sampling the usual hackery... View More

    — Posted by NYtoNC81

    I tried. ;-)

  • Come On_Seriously Aug 5, 11:46 a.m.

    What happened to my earlier posts WRAL? Did I offend a cry baby liberal? Are you censoring words... View More

    — Posted by foolmeonce

    Try a little civility rather than continued name-calling. Maybe you'll have better luck.

  • foolmeonce Aug 5, 11:37 a.m.

    What happened to my earlier posts WRAL? Did I offend a cry baby liberal? Are you censoring words like communist, socialist, fascist and marxist?

  • NYtoNC81 Aug 5, 10:15 a.m.

    I guess I shouldn't be surprised that in an article about statistical sampling the usual hackery from both sides is the thrust of the comment section.

    We're hopeless.

  • Come On_Seriously Aug 5, 10:05 a.m.

    The new Hagan ad where she is portraying herself as a moderate is comical. Compared to the... View More

    — Posted by foolmeonce

    This is exactly why Hagan will win. There is so much more name-calling than substance to the negative comments from the right. People get tired of the mud and with the exception of the name-callers, are generally smart enough to realize that more names = less thought due to thought-clouding anger.

    Meanwhile, Tillis detractors have direct and recent facts to back them up and don't have to rely on sticking ridiculous and angry labels all over everyone.

  • European American Aug 5, 8:54 a.m.

    "After all the Lies come out from the Liberals, Hagan who can't tell the Truth will get all the... View More

    — Posted by WralCensorsAreBias

    Oh Snap! Isn't that the truth!

  • dollibug Aug 5, 8:47 a.m.

    The ads which have been seen on TV are a *waste* of time and money. I would hope most people would NOT believe them. It is sad when candidates blame *one* person for everything which went *wrong* when we all know that it takes more than just one lawmaker voting to pass whatever is trying to get approved. Only IF there is one vote difference which probably would not be accepted to begin with on getting an issue voted in. Hagan has shown what she is capable of doing OR not. Tillis has NOT served in Washington yet. So we, the people, must decide who is the lesser of 2 evils and cast a vote for whomever *seems * to be the one capable of representing NC the best. In my own personal opinion, our choices are quite slim. So do we continue to let Hagan stay or do we take a chance for Tillis to have his turn. It is a toss.

More...