8

It is claimed by Rickova and Irsova, in a paper titled "Publication Bias in Measuring Climate Sensitivity" [A], that:

We present a meta-regression analysis of the relation between the concentration of carbon dioxide in the atmosphere and changes in global temperature. The relation is captured by “climate sensitivity”, which measures the response to a doubling of carbon dioxide concentrations compared to pre-industrial levels. Estimates of climate sensitivity play a crucial role in evaluating the impacts of climate change and constitute one of the most important inputs into the computation of the social cost of carbon, which reflects the socially optimal value of a carbon tax. Climate sensitivity has been estimated by many researchers, but their results vary significantly. We collect 48 estimates from 16 studies and analyze the literature quantitatively. We find evidence for publication selection bias: researchers tend to report preferentially large estimates of climate sensitivity. Corrected for publication bias, the bulk of the literature is consistent with climate sensitivity lying between 1.4 and 2.3C.

Is this true? A paper titled "No evidence of publication bias in climate change science", written by Christian Harlos, Tim C. Edgell & Johan Hollander [B], appears opposed, but finds that while

Our meta-analysis did not find evidence of small, statistically non-significant results being under-reported in our sample of climate change articles

it did find the following:

However, our meta-analysis did find multiple lines of evidence of biases within our sample of articles, which were perpetuated in journals of all impact factors and related largely to how science is communicated: The large, statistically significant effects were typically showcased in abstracts and summary paragraphs, whereas the lesser effects, especially those that were not statistically significant, were often buried in the main body of reports. Although the tendency to isolate large, significant results in abstracts has been noted elsewhere (Fanelli 2012), here we provide the first empirical evidence of such a trend across a large sample of literature.

We also discovered a temporal pattern to reporting biases, which appeared to be related to seminal events in the climate change community and may reflect a socio-economic driver in the publication record. First, there was a conspicuous rise in the number of climate change publications in the 2 years following IPCC 2007, which likely reflects the rise in popularity (among public and funding agencies) for this field of research and the increased appetite among journal editors to publish these articles. Concurrent with increased publication rates was an increase in reported effect sizes in abstracts.

Similar stylistic biases were found when comparing articles from journals with high impact factors to those with low impact factors. High impact factors were associated with significantly larger reported effect sizes (and lower sample sizes; see Fig. 4); these articles also had a significantly larger difference between effects reported in abstracts versus the main body of their reports (Fig. 3). This trend appears to be driven by a small number of journals with large impact factors; however, the result is consistent with those of supplementary studies.

and also shows in Figure 3: Figure displaying Climate Sensitivity estimates, grouped by Journal Impact Factor and Sample Size that when their sample includes all journals (which is required per their above finding that journals with a high impact factor selectively publish larger results), the Climate Sensitivity is estimated at 1.6C per doubling of atmospheric C02.

[A] https://ideas.repec.org/p/fau/wpaper/wp2015_14.html (2015)

[B] https://link.springer.com/article/10.1007/s10584-016-1880-1 (2016)

PS: I found this blog to give a better overview of the claim than I can: http://grokinfullness.blogspot.com/2017/04/publication-bias-in-climate-science.html

6
  • The second paper is looking at a different sample of papers to the first, and the 'cumulative effect size' column is not a measure of climate sensitivities found from a paper. – James Picone Oct 15 '20 at 2:03
  • 1
    The first paper is a working paper with no citations. It was later published in Energy and Environment (Reckova, Dominika, and Zuzana Irsova. "Publication bias in measuring anthropogenic climate change." Energy & Environment 26.5 (2015): 853-862.) E&E is a biased publication. – David Hammen Oct 15 '20 at 13:38
  • 2
    How is E&E biased? – Charlie Crown Oct 18 '20 at 9:09
  • 2
    @CharlieCrown The journal is well-known for publishing shoddy papers that take a contrarian position on climate change. For example, a paper by Ernst-Georg Beck claiming that CO2 was ~400ppm in the 1800s, and a paper by Oliver Manuel claiming that the sun is made of iron. While I think the purpose of Skeptics is to look at individual claims on their merits, it is worth knowing that being published in E&E isn't much better than being published on a random blog. – James Picone Oct 20 '20 at 2:44
  • 1
    @JamesPicone Oh, I don't actually know E&E... I was thinking of Energy and Environmental Science. That would be a different journal. Most journals (all journals) have retractions because garbage makes it in though. I wouldn't hold 2 papers against a journal. – Charlie Crown Oct 20 '20 at 2:52
8

Going to try to expand some things from the comments into a full answer.

The meta view

The paper in question was published in Energy & Environment in 2015. It's since been cited 7 times, mostly by a T Havránek who closely collaborates with the paper's authors. While obviously it's important for Skeptics to consider claims on the object level, on the meta level this is a single paper published in a small-time journal (Impact factor of ~1.7 according to SAGE) that has a reputation for publishing shoddy climate 'skeptic' papers, and the paper has since essentially been ignored.

Some examples of shoddy papers E&E has published are a paper that claims the sun is made out of iron, and a paper claiming wild variation in atmospheric CO2 concentrations just prior to the modern instrumental era. While almost every journal has bad papers get through from time to time, these specific papers are so far beyond the pale they shouldn't have passed any reputable peer review process, and there are a lot more where they came from. The short version is that there are very good reasons to be skeptical of the validity of the paper.

The Second Paper

The second paper the blog post notes - No evidence of publication bias in climate change science - is not addressing precisely the same question as the former paper. From the paper:

For the purpose of our meta-analysis, we sampled articles from the body of literature that explores the effects of climate change on marine organisms.

All the results included in the sample that paper took are about the response of marine organisms; not about climate sensitivity.

The biases that they found and you note are not the kinds of biases the initial paper is discussing:

...The large, statistically significant effects were typically showcased in abstracts and summary paragraphs, whereas the lesser effects, especially those that were not statistically significant, were often buried in the main body of reports.

That is, papers tended to report more significant results more prominently than less significant results - their abstract would say "We found increased temperature harmed coral growth (effect size Very Big, Quite Significant)", and then page 10 would say "We found increased CO2 concentration reduced shell growth rates (effect size small, not very significant)". This is a bias, but it isn't not reporting bad results, which is what the initial paper alleges.

First, there was a conspicuous rise in the number of climate change publications in the 2 years following IPCC 2007, which likely reflects the rise in popularity (among public and funding agencies) for this field of research and the increased appetite among journal editors to publish these articles. Concurrent with increased publication rates was an increase in reported effect sizes in abstracts.

So this is just saying that after 2007 there was a significant increase in the number of papers published addressing issues they sampled, and that effects discovered by those papers were larger. You could allege that scientists started hyping up their results to get kudos in the next big thing, but it's more plausible that increased funding and more people moving into the field allowed scientists to do better research so they got better results.

Similar stylistic biases were found when comparing articles from journals with high impact factors to those with low impact factors. High impact factors were associated with significantly larger reported effect sizes (and lower sample sizes; see Fig. 4); these articles also had a significantly larger difference between effects reported in abstracts versus the main body of their reports (Fig. 3). This trend appears to be driven by a small number of journals with large impact factors; however, the result is consistent with those of supplementary studies.

"Better results are published in more prominent journals". This is entirely normal and unsurprising. It'd be nice from a pure-empiricism perspective if negative results were more prominent, but in the meantime you don't get published in Nature unless you've got a big result.

The graph you post from the paper is not a graph of climate sensitivities in published papers. It's a graph of effect sizes found in papers looking at the impact of climate change on marine organisms. The measure of effect size they used is Hedge's d:

Hedges’ d was the mean of the control group (X C) subtracted from the mean of the experimental group (X E), divided by the pooled standard deviation (s) and multiplied by a correction factor for small sample sizes (J).

So a paper finding an effect size of 1.5 (sort of) found that the effect they were measuring was ~1.5 times the standard deviation in the property affected. If the paper was looking at coral bleaching events, it might find that normally the number of coral bleaching events in a year has a mean of 20 and a standard deviation of 5, but under elevated temperatures it has a mean of 28 and a standard deviation of 5. That would be an effect size of ~1.5. This is a hundred-foot view without statistical rigour, mind.

This graph is fundamentally just demonstrating that bigger effects get published in larger journals, but smaller effects are more common.

The First Paper

So is there publication bias in climate sensitivity estimates?

I'm skeptical, for a few reasons:

Very few studies

The appendix to the paper lists the papers used to construct the funnel plot; there are only 16! This is significant because it means that a few 'weird' papers can significantly skew the results. And weird papers we might have, because their sample includes Scafetta 2013a and 2013b, which are ECS estimates in much the same way saying "I guess ECS is about 3" is an ECS estimate. You can find some analysis of some of Scafetta's other work here and here; all of his papers on climate change are essentially the same. They're all numerology.

Lindzen and Choi 2011 is another paper included which is considered pretty questionable.

These three papers are notable for presenting extremely low climate sensitivity estimates (Lindzen and Choi 2011 estimates 0.7!), with inappropriately 'precise' results due to terrible methodology. In the small sample of papers, this has a significant effect on the observed funnel, making it look like there are very precise papers with very low sensitivities.

Climate sensitivities are not normally distributed

The funnel plot methodology implicitly assumes that the 'funnel' of results will be distributed normally around the true result. As the paper discusses:

In the absence of publication bias these figures should look like an inverted funnel. However, Figure 3 depicts only the right-hand side of the inverted funnel and the left-hand side is completely missing, indicating publication selectivity bias.

But climate sensitivity is bounded from below by physics; there is extremely strong agreement that the no-feedback sensitivity is about 1c. A sensitivity below 1c would then imply a stable climate, which is extremely physically implausible in light of, for example, ice ages. A more quantitative version of that argument implies sensitivities below ~1.5 are extremely unlikely. This explanation by SkepticalScience includes a number of estimates of climate sensitivity; you can see that essentially none have a lowerbound below 1.5c. This pretty much requires the left edge of the funnel to be missing. There's some discussion about this issue here.

Low sensitivity is incompatible with observed warming

Since 1970 we've seen ~0.8c of warming:Graph of global temperature since 1970.

Since 1970, CO2 concentration has gone from ~325ppmv, to ~415ppmv: Graph of CO2 concentration since 1970

This increase in CO2 is ~35% of the effect of a CO2 doubling (ln(415/325) / ln(2) ~= 0.35). The implied ECS is ~2.3c, and the actual equilibrium response is expected to be larger than the transient response. It is difficult to justify ECS estimates lower than what we've actually seen!

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .