In 2012 Australian psychologist Stephen Lewandowsky (and others) published a paper claiming to show that there was a significant correlation between beliefs in free market economics and the rejection of scientific claims (examples given include rejecting the reality of climate change or that smoking causes cancer).

The draft text of Lewandowsky's paper is here. And the results have been discussed in this skeptics question: Does being a strong free market supporter correlate with rejection of scientific claims?

Some critics have, however, questioned the quality of his results and the provenance of his data. For a review of such evidence see Steve McIntyre's commentary here.

For example, Lewandowsky claimed to have posted his survey on 8 "pro-science" blogs the most prominent of which was Skeptical Science. McIntyre claims the link never appeared but that Lewandowsky has continued to claim it did.

The most prominent of the eight were SKS [skeptical science], Deltoid, and Tamino. Chambers, Woods, and others sharply criticized the idea of trying a survey of skeptics at stridently anti-skeptic blogs, a criticism that has remains unrefuted. Chambers and Woods quickly located referring blogposts at all of the blogs except SKS. They were able to locate a contemporary tweet from John Cook, but no SKS blogpost. Close examination of the Wayback machine archive showed beyond any doubt that there was no SKS blogpost on the Lewandowsky survey in August/September 2010

So there are two allegations of a slapdash attitude to data. Surveying "skeptical" opinion (skeptical in the climate-skeptic sense) on a pro-science group of blogs and not keeping good enough track of the responses to know which blogs posted the link and therefore contributed to the results (made worse because the readership of SKS is discussed in the full paper).

A recent blog entry analyses some of the raw data used in the paper:

Let's look at the whole picture. This was a scattered online study posted at political climate-related websites. Here's the endorsement count for each of the conspiracies in their conspiracy variable, and the rejection count for the HIV and smoking facts. This is out of 1145 participants:

Oklahoma City bombing conspiracy: 289

JFK assassination conspiracy: 247

Coca-Cola conspiracy (don't ask): 151

Pearl Harbor attack conspiracy: 146

MLK assassination conspiracy: 90

New World Order conspiracy: 70

9-11 attacks conspiracy: 69

Roswell UFO conspiracy: 47

SARS disease conspiracy: 42

Area 51 UFO conspiracy: 35

Princess Diana assassination conspiracy: 25

Reject HIV-AIDS link: 16

Reject smoking-lung cancer link: 11

Moon landing hoax conspiracy: 10

Out of 1145 participants, only 11 reject the idea that smoking causes cancer. Out of 176 people who endorsed free markets, only 7 rejected the claim that smoking causes cancer. 96% of them agreed that smoking causes lung cancer.

...When only 11 out of 1145, and 7 out 176 in an online study rejected the item, and you know several of those have to be thrown out, it's just fraud to proclaim that a person's politics predicts rejection of these basic scientific facts. That's a scam. There's nobody there, no data.

The very small numbers of believers in the key conspiracies mentioned in the original article's headline should cause some twitching of people's statistical number sense. The vast majority of climate skeptics in the survey don't believe in those conspiracy theories and removing small numbers of implausible responses totally destroys the relationship.

Similar criticism of the original paper also appeared in SkepticalScience (a mainstream climate blog) from Tom Curtis:

Combined, these respondents account for 2 of the strongly agree results in almost every conspiracy theory question; and the other potential scammers also have a noticable number of strong agreements to conspiracy theories. For most conspiracy theory questions, "skeptics" only had two respondents that strongly agreed, the two scammed results. Given the low number of "skeptical" respondents overall; these two scammed responses significantly affect the results regarding conspiracy theory ideation. Indeed, given the dubious interpretation of weakly agreed responses (see previous post), this paper has no data worth interpreting with regard to conspiracy theory ideation.

McIntyre is a well know climate skeptic and therefore one of the group targeted by Lewandowsky's original analysis. So skepticism of his claims is likely warranted. On the other hand the evidence he assembles seems thorough. But has he (and the other associated critics he references) produced enough analysis to raise serious questions about Lewandowsky's methods and data quality?

NB McIntyre et. al. have also argued that Lewandowsky and his associates have lied to cover up their mistakes. This isn't what I'm asking. I want to focus on whether critics have demonstrated that the basic methods used to gather and process data for the paper were poor enough to cast doubt on its conclusions.

PS The analysis above is not from peer reviewed sources, but the statistical facts have not been challenged. They sound like entirely valid criticisms to me. That's why I'm including them in this question so, if there are strong, reliable sources that can rescue Lewandowski's big claims from what look to be really dubious statistics then let's get them in an answer.

  • So the claim here is that for one of the eight blog surveys we can't find the original data? Have we established whether removing these data points would make any difference to the results? – DJClayworth Apr 3 '13 at 0:01
  • 2
    I'm scratching my head here as to why a self-selected online survey is being given any credence at all. Shouldn't we dismiss this worth little more than an exploratory first attempt, whether the link was published or not? Won't the self-selection bias dominate over the results anyway? Or am I overly distrusting here? – Oddthinking Apr 3 '13 at 4:02
  • @Oddthinking I don’t see why an online survey is inherently untrustworthy, and every survey suffers from self-selection bias to some extent. Of course this must be taken into account and corrected for, to the extent possible. – Konrad Rudolph Apr 3 '13 at 7:17
  • 4
    @Konrad: These are good questions, and I don't think I have good answers for them. (For that reason, I am being cautious not to overstep here; I am trying to check where others draw the line, rather than pronouncing I know where it is.) My concern seems to be that only the strongly motivated people will respond; in the same way artists find YouTube comments are biased towards negative sentiment (happy consumers are much less likely to comment than unhappy ones), you will strongly oversample only motivated answerers, who are more likely to be homogenous in their view. – Oddthinking Apr 3 '13 at 12:54
  • 4
    Anyone too dumb to confuse " the rejection of scientific claims" with "rejection of two or three specific claims that are usually made by people with political goals and acceptance of 99.9999% of other scientific claims" shouldn't be posting research on the topic and pretending to be a scientist as opposed to a political hack. – user5341 Apr 17 '13 at 13:55

You must log in to answer this question.

Browse other questions tagged .