Did the IPCC obfuscate the evidence in a way that obscures the discrepancy between previous climate models and recent observations?

Question

Just before the recent IPPC meeting which was due to publish new analysis on climate change, the Financial Times published an excellent summary of the difference between predictions made in the last IPCC report and actual observations. The chart shows what looks like a clear presentation of the discrepancy between observations and predictions (see the FT article here: note the dataviz is interactive and shows more than just temperature. The dataviz is accessible here to those who can't access the (paywalled) FT original). A static picture of the temperature chart is below:

Climate skeptic Steve Mcintyre (I think "skeptic" is a better term for him than denier as he doesn't deny climate change; other opinions on him are available) argued that the IPCC were originally going to use a similar chart but changed their mind during the review process.

He argues that the chart below was present in early drafts:

but was changed to this chart in later drafts and the final report:

Mcintyre argues:

Figure 1.4 of the Second Order Draft clearly showed the discrepancy between models and observations, though IPCC’s covering text reported otherwise. ... Needless to say, this diagram did not survive. Instead, IPCC replaced the damning (but accurate) diagram with a new diagram in which the inconsistency has been disappeared.

There are a number of technical issues here (even in the first chart: eg what does the grey bad mean and what is its derivation?). But the thrust of his argument seems to be that the IPCC have deliberately chosen a way of presenting the data that obscures what seems like an obvious problem: model skill doesn't seem to be very good in the last decade or so.

I think answers to the questions of whether there is a problem with model skill would be better addressed in this question: Are the IPCC climate change models overestimating sensitivity to carbon dioxide? so please don't make them in answers here.

My question is simpler: is Mcintyre correct to argue that the IPCC have modified their presentation of the data in a way that obscures the discrepancy? Supplementary question: what is a good and unbiased way to present this sort of comparison in a way that allows a neutral skeptic to make a judgement?

Answer 1

No, IPCC did not obfuscate evidence.

The early draft version of the graph was flawed. That's why it was in an early draft and has been removed subsequently.

This question has been adressed in an article at skepticalscience.com, which draws mainly from an excellent blog post by climate blogger and statistician Grant Foster (a.k.a. Tamino). Quoting from the skepticalscience article, which in turn quotes from the blog-post:

Late last year, an early draft of the IPCC report was leaked, including the first draft version of the figure shown above. The first version of the graph had some flaws, including a significant one immediately noted by statistician and climate blogger Tamino.

"The flaw is this: all the series (both projections and observations) are aligned at 1990. But observations include random year-to-year fluctuations, whereas the projections do not because the average of multiple models averages those out ... the projections should be aligned to the value due to the existing trend in observations at 1990.

Aligning the projections with a single extra-hot year makes the projections seem too hot, so observations are too cool by comparison."

In the draft version of the IPCC figure, it was simply a visual illusion that the surface temperature data appeared to be warming less slowly than the model projections, even though the measured temperature trend fell within the range of model simulations. Obviously this mistake was subsequently corrected.

This illustrates why it's a bad idea to publicize material in draft form, which by definition is a work in progress.

Note that neither skepticalscience nor the linked blog are peer-reviewed publications. I recommend reading both articles in their entirely and then forming your own opinion.

Answer 2

The first version of the IPCC diagram is not a very good representation of what the models actually say. Note that the diagram suggests that the models were all in exact agreement on global temperatures in 1990. Was this the case? No, as we can see if we look at the outputs of the model using ClimateExplorer (which is a great tool for checking out what the models actually say):

These are the anomalies (i.e. the mean seasonal cycle has been subtracted out) for all model runs for RCP8.5 for the temperature at surface (TAS). It is quite clear that the models are not all in perfect agreement about 1990. The revised IPCC diagram faithfully reflects the variability in actual model output, so rather than obfuscating the model-observation comparsion, it has made it more fair.

The difference between the two plots is due to baselining. Climate models are better at modelling the changes in temperature resulting from changes in forcings than they are at accurately modelling absolute temperature (some of the variability in the model runs shown above is due to differences in absolute temperature between model runs). This is well known to those who work with models (although it is such common practice it is difficult to find a paper explaining why this is necessary - it is apparently part of the scientific paradigm). However these differences in absolute temperature are essentially irrelevant to the question of the response of the climate to increasing CO2 radiative forcing. The simple solution is to subtract a constant from each model run so that they agree on the mean temperature over some agreed baseline period.

Here is what the IPCC say about baseline periods (from the TAR):

"A popular climatological baseline period is a 30-year "normal" period, as defined by the WMO. The current WMO normal period is 1961-1990, which provides a standard reference for many impact studies."

Long baseline periods are a good idea as it means that the projections are not very sensitive to the effects of internal variability in the observations and in the models (as it is the average offset over an extended period that e.g. contains several cycles of ENSO). If we use just one year and align all the models and observations to that one year, then if it is a particular warm year in the observations e.g. 1990, the models also start hot and this increases the apparent discrepancy later. If we use a cool year, e.g. 1992, the models start cooler as well, and this would decrease the apparent discrepancy later. However the difference between the two is entirely meaningless as it depends on the noise in the observations. That is why the longer baseline used in the revised diagram gives a more accurate depiction of the difference between models and observations.

So why did the IPCC use a single year baseline in the original diagram? I suspect it is because it makes the trends (rate of warming) easier to see visially if the models and projections all pass through some common point. HOWEVER this does mean that there is a spurious offset between models and observations, that depends on which year in which the data are made to agree.

Far from obfuscating the difference between models and observation, the new diagram presents it more clearly.

Answer 3

Reading the three charts:

• The first chart joins the observed values with lines, emphasising volatility in observed values. It has a shorter y-axis scale than the others, making things look more dramatic than the other two charts. The four 2007 scenarios are each shown as a single line rather than a range. The visual suggestion is that observed values are a long way below the scenarios A1B, A2 and B1 but are reasonably close to the Commit scenario.

• The second chart gives coloured bands for the different reports, with AR4 visible in the middle of these and apparently narrower than TAR ranges. The grey band is not explained in the key. AR4 scenario A1B is emphasised with the yellow double headed arrows and is in the bottom three-quarters of the AR4 band. The x-axis scale from 1990 to 2015 emphasises observations since FAR, SAR ,TAR and AR4. 2010 observations seem to be below the AR4 and A1B ranges but 2011 observations (without an uncertainty bar) below them.

• The third chart shows a lot of lines presumably of what was in AR4 CMIP3. It also gives a variety of observed points and smoothed lines; the y-axis scale is longer than the others, making observed changes seem smaller. The extension of the x-axis scale to 1950 to 2030 substantially reduces the emphasis on observations since AR4. The background shading seems to emphasise FAR, SAR and TAR rather than AR4. The bars at the right suggest that the individual scenarios A1B, A2 and B1 each had wider uncertainty than TAR. Recent observed values appear to be near the bottom of the very spiky AR4 CMIP3 lines.

To me, the biggest difference between the three graphs is how much uncertainty was associated the three AR4 scenarios A1B, A2 and B1 and how this has been displayed. The first chart shows no uncertainty in each, and has the three very close together to 2020; the second chart suggests a narrower uncertainty for one of them AR4 as a whole and AR4 scenarios together less uncertain than TAR; and the third chart only shows anything about them at all off to the side of the chart, and suggests that each was extremely uncertain, which is why the y-axis scale has to be so long.

The change in x-axis scale between the second and third charts in fact changes the message from the charts: the second appears to say it is a comparison between forecasts and observations with a scale covering observations since the forecasts were made, while the third seems to be saying it was uncertain before the initial forecasts and will be highly uncertain into the future beyond current observations. Compared with the second chart, the third chart de-emphasises the comparison between forecasts and observations and this may be what is underlying McIntyre's point.

If the charts have been modified, it has been with the effect of suggesting increased uncertainty in the past and so subsequent observations appearing more within the (now wider) uncertainty bands, and great uncertainty beyond the current observations. This is presumably deliberate. The real question is whether it is honest history: the IPCC believes that it is.

Answer 4

The data visualization choices made by the IPCC are poor and have the effect of obscuring the differences between model forecasts and observed temperatures.

Note: this answer assesses whether the IPCC have done a good job of presenting the comparison of observed temperatures and model projection. This, I think, is what a good skeptic would demand if the purpose of presenting the data is to assess whether the models have any predictive skill. This may not be the intention of the IPCC presentation, of course. Also, I am not assessing the IPCCs conclusions, merely their choices of presentation methods.

The key problem the IPCC or other climate modellers have to face is how to present the results of climate projections which contain a great deal of uncertainty. Models don't predict a single path of future temperatures: the short term climate is chaotic and, even if we know all the inputs with precision (which we don't) there are many factors that create many paths into the future. So even a single model will give an ensemble of possible paths into the future. So one key question is how to describe that uncertainty in a visual way that does not misrepresent the nature of the uncertainty.

Another problem is that there are multiple scenarios for what will happen in the future. The key ones use different assumptions about how the world will respond to climate change (will we constrain the output of CO₂ or just keep producing increasing amounts each year?

The initial IPCC chart does a half-assed job of the first problem and a poor job dealing with the second. The chart presents the envelopes of forecasts produced by particular models, which is commendable, but incomplete and it fails to account for the shape of the probability distribution of possible temperature paths (examples of better approaches below). The second problem is that the chart tries to show multiple envelopes from multiple model versions. This just confuses as the envelopes overlap and it is hard to get a clear visual impression of the particular envelope for any one model. In fact the choice of models to show is particularly confusing as the envelopes are shown for the key models from previous IPCC rounds despite the obsolescence of the old models and their irrelevance to whether current models are doing a good job. A better approach, if the goal would be to show one chart for each group of models showing the projections made at each previous report with a comparison of all subsequent actual observations. Small Multiples of this sort are a very effective visualisation technique (see this excellent discussion with many examples at Junk Charts). The IPCC even used this approach when they presented the original AR4 forecasts (why not, it seems reasonable to ask, just repeat these charts in the current report with observed temperatures overlayed?):

There are good techniques for unambiguously presenting statistical ranges on forecasts. One of the best is used by the Bank of England (BOE) for its inflation forecasts. Inflation forecasts have a lot in common with climate forecasts: they are based on complex models of the behaviour of the economy, projected from historic observations but containing chaotic noise. They, again like climate models, produce a range of possible paths for future inflation. An example is given below:

The economist article the chart is derived from also discusses the technique explaining it like this:

In 1996, the Bank introduced the innovation of presenting the forecast as a probability distribution, in order to highlight the risks to the central forecast. These forecasts, which are agreed by the Monetary Policy Committee (MPC), are called “fan” charts because the expected outcomes spread out in bands from the most likely path of inflation to cover 90% of the probability distribution.

The advantage of the BOE approach is that it gives the right visual impression that paths near the edge of the envelope are less probable than those in the middle. The IPCC envelopes in the original chart equally weight the whole envelope of forecasts giving the (incorrect) impression that edge paths are just as likely as central paths.

The final version of the IPCC charts compounds this problem by also adding many individual paths to the plot. Visually this overweights the extreme paths at the edge of the probability distribution far too heavily compared to the much larger number of paths near the middle of the distribution.

The final presentation also botches the second task even more thoroughly by including multiple AR4 scenarios despite the fact that only one of those comes close to representing the observations (such as CO₂ emissions) which were uncertain at the time of the projection but are now history (why include scenarios assuming tight emission constraints when we know that didn't happen?)

The IPCC could have done a much better job. For example, the chart below shows a much clearer view (I'm not judging the data here, just the presentation format):

This chart is from Judith Curry's blog but is initially derived from Jeff Hawkings here. Surprigingly, there have been very few, if any, attempts to either do this comparative analysis or to present the results well. As Judith Curry observes:

In the midst of substantial public interest on this issue, there is no published analysis that I know of that compares CMIP5 simulations to observations, although it looks like Ed Hawkins’ analysis is heading towards publication.

An alternative and simpler view would show a snapshot of the average performance of the models up to now versus the actual temperature record (this view simplifies the paths taken over time so loses some information). An example of such an analysis is shown below:

The original source for this is a recent paper in Nature (paywalled). The chart makes a direct comparisons among CMIP5 forecasts (the grey distribution) and the observed path of temperature since the forecasts were made. (Note this was published after the Curry comment quoted above).

Summary

The effective visualization of complex data is difficult. But good advice exists (for example: see the guru of the field Edward Tufte's work; the Junk Charts readable and topical blog; or the usually excellent examples at Flowingdata. Unfortunately this advice is mostly ignored, especially by scientists who often seem to believe "the data in the chart was fine; what's the problem?"

In pure visualization terms the IPCC's changes made an average chart much worse and obscured things a reasonable skeptic would have preferred to be clarified. The visualization changes made any comparison between model projections and observations much harder and much less statistically sound. In their defence, perhaps clarity was never their goal and they didn't have any competent visualization advice.

Answer 5

Evaluating information visualizations

To determine whether or not the published chart "obscures the discrepancy" compared to the draft versions of the chart, we'd need to conduct an empirical evaluation of the visualizations. (Chen, 2000)

Visualizations are commonly measured against accuracy and efficiency metrics. "Accuracy measures typically include precision, error rate, the average number of incorrect answers and the number of correct document retrieved. Efficiency measures typically include the average time to completion and the performance time." (Chen, 2000)

Identification of the task to be performed is a required step in the experimental design. (Card, 1997; Chen, 2000; Schulz, 2013) Here is just a small sample of the range of tasks that have been proposed in the information visualization community: exploring the data, presenting data, finding similarities/differences, visualizing parameter distributions, visualizing parameter variations, visualizing outliers, gaining overview over a dataset, finding correlations, creating a high quality presentation, searching for characteristic features, and finding relations/phenomena/effects. (Schulz, 2013)

What task is this chart made for?

Even if it did "obscure the discrepancy" in a way that slows down a viewer's analysis, perhaps the chart was chosen for improvement in other metrics, such as accuracy of that analysis, or ability to answer questions about a wider range of projections. As much as we don't know the effect of this choice of visualization, we don't even know the intent.

My impression

If you want a sample size of 1 from a bad experiment, I can still see the discrepancy in the published chart.

References

Card, Stuart K., and Jock Mackinlay. "The structure of the information visualization design space." In Information Visualization, 1997. Proceedings., IEEE Symposium on, pp. 92-99. IEEE, 1997.

Chen, Chaomei, and Yue Yu. "Empirical studies of information visualization: a meta-analysis." International Journal of Human-Computer Studies 53, no. 5 (2000): 851-866.

Schulz, Hans-Jörg, Thomas Nocke, Magnus Heitzler, and Heidrun Schumann. "A Design Space of Visualization Tasks." IEEE transactions on visualization and computer graphics 19, no. 12 (2013): 2366-2375.