7

A recent Gallup poll claims that 46% of Americans believe :

God created human beings pretty much in their present form within the last 10,000 years or so

It seems to me possible that, when asked about a matter of faith on a phone survey, people might exaggerate their position or, alternatively, they might provide the sarcastic answer that is the opposite of their actual feeling. I admit that I have answered polls in ways intended to confound the results.

One might expect that dishonest responses cancel each other out, but that is not necessarily the case.

Do polling companies account for exaggeration/dishonesty?

  • 2
  • When you ask someone their opinion about something that threatens their very soul you get the answer that their church tells them their god wants hear. Most polls I have seen on this topic tend to be slanted. I would submit that it was ~10k years ago that homo sapiens sapiens emerged as the dominant specie on the planent. If a gods design(evolution) intended this then that belief could be true. – Chad Jun 4 '12 at 12:57
7

Gallup publishes the data along with the methodology:

Results are based on telephone interviews conducted May 3-6, 2012 with a random sample of –1,024—adults, aged 18+, living in all 50 U.S. states and the District of Columbia.

For results based on the total sample of national adults, one can say with 95% confidence that the margin of error is ±4 percentage points.
For results based on the sample of –534—national adults in Form A and –490—national adults in Form B, the maximum margins of sampling error are ±5 percentage points.

Interviews are conducted with respondents on landline telephones and cellular phones, with interviews conducted in Spanish for respondents who are primarily Spanish-speaking. Each sample includes a minimum quota of 400 cell phone respondents and 600 landline respondents, with additional minimum quotas among landline respondents by region. Landline numbers are chosen at random among listed telephone numbers, cell phone numbers are selected using random-digit dial methods. Landline respondents are chosen at random within each household on the basis of which member had the most recent birthday.

Samples are weighted by gender, age, race, Hispanic ethnicity, education, region, adults in the household, and phone status (cell phone only/landline only/both, having an unlisted landline number, and being cell phone mostly). Demographic weighting targets are based on the March 2011 Current Population Survey figures for the age 18+ non-institutionalized population living in U.S. telephone households. All reported margins of sampling error include the computed design effects for weighting and sample design.

In addition to sampling error, question wording and practical difficulties in conducting surveys can introduce error or bias into the findings of public opinion polls.

So in other words there's about a 5-point margin of error on the results, and they may well be imprecise due to lying.

In other words, the results are way less precise than presented. It is assumed that there will be errors, but that the picture is substantially correct. I find no fault in the overall picture, but of course the exact numbers are probably different from what was presented.

In any case, what counts is the qualitative answer: most US residents are creationists. Quantitatively we can say it's the largest subset, although the number is only known to be between 41 and 51 percent of the population (within a 2σ — 95% — confidence interval)

Furthermore, the results are slightly different from what is presented on the main site, from the same pdf as above:

Gallup Poll

From the table above it is clear that, once we consider the 5 point uncertainty, nothing new has emerged from the poll: the fluctuations of the values across the years are consistent with the predicted errors.

Finally, there's a big manipulation here: the significant question is whether man evolved or not, not whether God assisted! Believing that God influences the world has nothing to do with either the scientific position of evolution and the charlatanerie of creationism.

If one discards that false dilemma, then the numbers look much more sane and believable: 47% of Americans accept evolution, and 46% believe in the superstition of creationism. It's still a frighteningly high percentage though.

TL;DR

The polls don't account directly for people being dishonest. However, that's not the big problem. The real sham is the misrepresentation of the results via omitting the error bars and introducing a false dilemma to make the position of evolution look much, much lower than it is.

  • 1
    I'm not so sure about the "manipulation": some people may find the question of evolution/creationism of minor importance compared to the belief whether god does exist or not (but would still answer the poll). Anyways, we just know that 46 % of those americans who are willing to discuss this question for a phone poll say they believe in creationism. The poll does not report how many people ended the call after learning what it is about. There may very well be a correlation of "strong" opinion (creationism) and the willingness to tell the world about this opinion, which could lead to bias. – cbeleites supports Monica Apr 21 '16 at 20:57
  • How does 41% to 51% translate to "most US residents"? For me, that phrase would imply the number is definitely above 50% at the least. – purposeful porpoise Apr 21 '16 at 21:57
7

Do polling companies account for exaggeration/dishonesty?

Yes they do (in a way). Polling methodology is a very well researched field. Mostly because of the spectacular failure of the 1936 Literary Digest presidential poll. Even if a poll is set up to take into account exaggeration, dishonesty, confusion, or the many other variables involved in these polls, there is still a lot of room for biased and flawed polls to get out there.

The main counter that polls have is sample size (you can get an idea on the sample size from the degree of confidence in the poll, i.e. the +- error bar). The larger (and more "controlled" random (1) ) the sample size, the more likely you are to get a good result. There are many, many books that go over the methodology, and why it works. Here is a small list of Scholarly books:

Of course, know how to conduct a survey still doesn't prevent people from intentionally skewing results to their favour, or against an opponent. Even amongst (political) polling organizations, some do have perceptions of being more or less reliable. One group suggest a series of 20 questions one should ask to determine if a poll is scientific. They are:

  1. Who did the poll?

  2. Who paid for the poll and why was it done?

  3. How many people were interviewed for the survey?

  4. How were those people chosen?

  5. What area (nation, state, or region) or what group (teachers,lawyers, Democratic voters, etc.) were these people chosen from?

  6. Are the results based on the answers of all the people interviewed?

  7. Who should have been interviewed and was not? Or do response rates matter?

  8. When was the poll done?

  9. How were the interviews conducted?

  10. What about polls on the Internet or World Wide Web?

  11. What is the sampling error for the poll results?

  12. Who’s on first?

  13. What other kinds of factors can skew poll results?

  14. What questions were asked?

  15. In what order were the questions asked?

  16. What about "push polls?"

  17. What other polls have been done on this topic? Do they say the same thing? If they are different, why are they different?

  18. What about exit polls?

  19. What else needs to be included in the report of the poll?

  20. So I've asked all the questions. The answers sound good. Should we report the results?

Even so, it takes a great deal of research on the part of anyone who is digesting poll results to really get an idea of what is truly happening. Sadly, the general public does not put forth that effort, nor does the media traditionally delve into polls beyond the surface results. Does that mean one should discount the results entirely? No, as previously stated, the science and methodology is actually pretty well understood. It's just what people do with the results that ends up making the results so difficult to parse. There is an actual quarterly publication journal for polling organizations. I suggest you read up in Public Opinion Quarterly if you desire to learn more.

(1) By controlled random is that the pollsters actually understand how their selection methodology may bias a "random" population. For instance, if they only call people with a land line, they are more apt to get older and rural respondents. If they conduct it via the internet, then that population also has a bias.

  • 1
    Comment (not an answer): Given what I see in the general public, I see no reason to particularly doubt that Gallup poll result. – Larian LeQuella Jun 2 '12 at 20:24
  • Based on that question list and info on a newer version of the poll (gallup.com/file/poll/170828/…) I'd put my finger on points 6 and 19: I'm missing the number of people who just ended the call after they heard that a poll is conducted and possibly on what subject it is (as opposed to: never answered the call). There might be a correlation of "strong" opinion (such as hard-core creationism) and the willingness to tell the world about this opinion (as this is about a certain flavor of religious belief, "testify" may be the term...). – cbeleites supports Monica Apr 21 '16 at 20:44
3

Pollsters are aware of a number of problems with polling. Some of the other answers address this, but they don't seem to focus on the core issue: respondents giving unreliable answers about their true intentions.

In election polling, one such suspected bias is known as the Bradley Effect.

The Bradley effect, less commonly called the Wilder effect, is a theory proposed to explain observed discrepancies between voter opinion polls and election outcomes in some United States government elections where a white candidate and a non-white candidate run against each other. The theory proposes that some voters will tell pollsters they are undecided or likely to vote for a black candidate, while on election day they vote for the white candidate.

The theory is controversial. I am not presenting it as true. I am presenting it as evidence that pollsters and analysts are aware of effects that might systematically skew results, and they don't assume that all respondents are honest.

(Other similar claims that social desirability bias affect the polls include the Shy Tory Factor.)

I wanted to show more evidence that analysts are openly concerned about the reliability of surveys, so I found this rather arbitrary research article looking at the problem in surveys in rural Malawi about HIV/AIDS, which started with a bit of a literature review. They looked at some US studies that attempted to detect inconsistencies by reinterviewing people.

These studies shared several conclusions about survey response error in demographic surveys carried out in developing countries. First, response error varies according to respondent characteristics, and to the content and psychological nature of the questions asked. More personal, sensitive questions tend to be less reliable than questions pertaining to relatively concrete factual matters (Knodel and Piampiti 1977). Second, although reporting at the aggregate level may appear to be relatively reliable, it often masks a high level of individual-level inconsistency. There is contradictory evidence, however, about whether these individual errors are randomly or systematically distributed (Coombs 1977: 255)

(Remember: This is not about voting preferences, but just demonstrates it is a known issue in the field.)

There are some techniques to address poll reliability:

  • Item Response Theory can look for questions that are not getting the expected distribution of responses.

  • Internal Consistency can look for answers that describe inconsistent positions within the same survey.

Conclusion

I haven't provided much solid evidence that particular polls are biases in particular way. I have provided evidence that the pollsters and analysts are not naive, and don't simply assume people are reliable.

(I have used Wikipedia throughout, as my goal is not to prove that the effects are real, but merely that the risks are well-known.)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .