Tips for evaluating polls
The tips below are from a January 17th, 2020 segment of Science Friday, The Science of Polling: 2020 and Beyond :
With the rise of the internet and DIY online surveys, anyone can conduct a poll, says Courtney Kennedy at Pew Research Center. While the increased access to polling technology has allowed for creativity and new perspectives, it has also made it difficult to trust the legitimacy of a poll. Before Friday’s show, Kennedy told SciFri three questions you should ask when you’re evaluating a poll:
1. Who created the poll?
“You need to know if there are any conflicts of interests,” Kennedy says. If you see a poll by a campaign or a lobby group, you’ll want to disregard it because these polls are released selectively and may have an agenda behind them, she says.
“Who did the poll is a key thing, always will be.”
2. How were people recruited?
“If you told me you went out and interviewed 2,000 people across the country, what’s the source? What database did you actually use to find them?” she says.
More traditional means of recruiting people include random telephone digit dialing and polling from a list of registered voters. However, the polling industry has dramatically shifted away from telephone methods and towards digital. Online polls pull from a wide array of sources. People may be solicited through pop-up ads, social media, or an emailed survey invitation by a corporate membership list. Other online polls recruit people by sending a random sample of notices by postal mail, while some use an opt-in sample, or what pollsters call “a convenient” sample, of people on the internet, Kennedy says.
You should ask yourself: “Was a poll done with a truly randomized sample of Americans from a list that covered everybody? Or was it done with more ad hoc means?”
3. Did the pollster account for population representation in the statistics?
Kennedy admits this statistical assessment can be tough to make, but she says you want to ask: Did the pollster weight the data to be representative of race, age, education, gender, geography, and other demographics?
“Good pollsters just list out the variables that they adjusted on to make the poll as representative as possible,” Kennedy says. “You want to see that all the major demographic variables are listed there.”
Additional readings
- Read a review from FiveThirtyEight on pollsters to trust, from 2018.
- See Pew Research’s field guide to polling for the 2020 election.
- Read Courtney Kennedy’s Q&A about the 2016 election on Pew Research.
- Watch a video about trust in polls on Pew Research.