9

Just before the recent IPPC meeting which was due to publish new analysis on climate change, the Financial Times published an excellent summary of the difference between predictions made in the last IPCC report and actual observations. The chart shows what looks like a clear presentation of the discrepancy between observations and predictions (see the FT article here: note the dataviz is interactive and shows more than just temperature. The dataviz is accessible here to those who can't access the (paywalled) FT original). A static picture of the temperature chart is below:

enter image description here

Climate skeptic Steve Mcintyre (I think "skeptic" is a better term for him than denier as he doesn't deny climate change; other opinions on him are available) argued that the IPCC were originally going to use a similar chart but changed their mind during the review process.

He argues that the chart below was present in early drafts:

Mcintyre's copy of early draft models vs observations

but was changed to this chart in later drafts and the final report:

Mcintyre's version of final ipcc chart

Mcintyre argues:

Figure 1.4 of the Second Order Draft clearly showed the discrepancy between models and observations, though IPCC’s covering text reported otherwise. ... Needless to say, this diagram did not survive. Instead, IPCC replaced the damning (but accurate) diagram with a new diagram in which the inconsistency has been disappeared.

There are a number of technical issues here (even in the first chart: eg what does the grey bad mean and what is its derivation?). But the thrust of his argument seems to be that the IPCC have deliberately chosen a way of presenting the data that obscures what seems like an obvious problem: model skill doesn't seem to be very good in the last decade or so.

I think answers to the questions of whether there is a problem with model skill would be better addressed in this question: Are the IPCC climate change models overestimating sensitivity to carbon dioxide? so please don't make them in answers here.

My question is simpler: is Mcintyre correct to argue that the IPCC have modified their presentation of the data in a way that obscures the discrepancy? Supplementary question: what is a good and unbiased way to present this sort of comparison in a way that allows a neutral skeptic to make a judgement?

  • 3
    I think that asking about intentions & motivation is usually considered off-topic here, isn't it? – EnergyNumbers Oct 6 '13 at 13:18
  • 1
    I'm happy to clarify the language to clear up the question. Obviously Mcintyre believes they were motivated to do it, but we should be able to discuss the effect without accepting his opinion. – matt_black Oct 6 '13 at 13:29
  • 2
    Changing the presentation of graphs back and forth during the publication process is routine. The data presented here is complex and good graphs intentionally focus on one particular feature. This makes good graphs always biased, but it’s a different thing entirely to suppose that the IPCC intentionally tried to obfuscate any discrepancies. – Konrad Rudolph Oct 6 '13 at 17:05
  • 3
    It is worth noting that the FT diagram appears to be plotting monthly anomalies for the observations against annual means for the projections. This is clearly bogus as (even if they had included the uncertainty in the projections - which they didn't, only the range of scenarios) the variability of monthly anomalies is much higher than that of annual anomalies, so you would expect them to be regularly outside the projections. Conclusion: be very skeptical of any presentation of scientific information provided by the media. – Dikran Marsupial Oct 9 '13 at 7:18
  • 3
    Essentially the problem here is that McIntyre doesn't understand what baselining is for and why it it necessary in model-observation comparison. – Dikran Marsupial Oct 9 '13 at 7:21
10

No, IPCC did not obfuscate evidence.

The early draft version of the graph was flawed. That's why it was in an early draft and has been removed subsequently.

This question has been adressed in an article at skepticalscience.com, which draws mainly from an excellent blog post by climate blogger and statistician Grant Foster (a.k.a. Tamino). Quoting from the skepticalscience article, which in turn quotes from the blog-post:

Late last year, an early draft of the IPCC report was leaked, including the first draft version of the figure shown above. The first version of the graph had some flaws, including a significant one immediately noted by statistician and climate blogger Tamino.

"The flaw is this: all the series (both projections and observations) are aligned at 1990. But observations include random year-to-year fluctuations, whereas the projections do not because the average of multiple models averages those out ... the projections should be aligned to the value due to the existing trend in observations at 1990.

Aligning the projections with a single extra-hot year makes the projections seem too hot, so observations are too cool by comparison."

In the draft version of the IPCC figure, it was simply a visual illusion that the surface temperature data appeared to be warming less slowly than the model projections, even though the measured temperature trend fell within the range of model simulations. Obviously this mistake was subsequently corrected.

This illustrates why it's a bad idea to publicize material in draft form, which by definition is a work in progress.

Note that neither skepticalscience nor the linked blog are peer-reviewed publications. I recommend reading both articles in their entirely and then forming your own opinion.

  • 2
    @matt_black I don't know if skepticalscience.com and Grant Foster are biased. I couldn't find a discussion on these graphs in the peer-reviewed literature. The explanation makes perfect sense and explains that the way results were represented in the draft were wrong, thus answering the question by that there is no obfuscating going on. If you already state that something has clearly obfuscated the key messages, you seem to already have an answer? Then why do you ask? Or are you going to answer your own question? – gerrit Oct 9 '13 at 14:06
  • 2
    @matt_black GCMs are able to reproduce change in temperature in response to changes in forcings better than they are able to reproduce well calibrated absolute temperatures. Thus you need to baseline them before performing comparisons to eliminate the meaningless variation in absolute temperature between model runs. This is well known (except perhaps to McIntyre) and entirely uncontroversial. The original IPCC used a single year baseline, which is sensitive to the internal variability in the model runs. The new version uses a longer period, and is more in keeping with best practice. – Dikran Marsupial Oct 9 '13 at 15:11
  • 3
    Essentially the only thing that the new diagram hides is the error in the version found in the draft report. That is the very purpose of having a draft report, which is that it gives an opportunity to fix any errors before the document is made public (unless of course some unscrupulous person leaks the draft so they could get their knickers in a twist about any changes they find in the final version). – Dikran Marsupial Oct 9 '13 at 15:14
  • 2
    @gerrit Two issues: I still don't understand the explanation and am looking for a clearly argued way to show model predictions vs outcomes; second, the choice of how do display data is far more confusing on the later IPCC chart (aside from any changes to the data) and that was my question's focus. If the IPCC wanted to adjust baselines etc, the format of the early chart would still have worked (or, even better, the format of the FT version). – matt_black Oct 9 '13 at 18:50
  • 2
    @matt_black If you want to show model predictions vs outcomes, then the obvious way to do so is to plot the model predictions directly (rather than forcing them to agree with the observations in an arbitrary year) and the observations on one diagram. That is just what the second version of the IPCC diagram does. The first version is (I suspect) intended for visualisation of the trends, not visualisation of the difference between the models and the observations. See my answer for details. – Dikran Marsupial Oct 9 '13 at 19:15
7

The first version of the IPCC diagram is not a very good representation of what the models actually say. Note that the diagram suggests that the models were all in exact agreement on global temperatures in 1990. Was this the case? No, as we can see if we look at the outputs of the model using ClimateExplorer (which is a great tool for checking out what the models actually say):

enter image description here

These are the anomalies (i.e. the mean seasonal cycle has been subtracted out) for all model runs for RCP8.5 for the temperature at surface (TAS). It is quite clear that the models are not all in perfect agreement about 1990. The revised IPCC diagram faithfully reflects the variability in actual model output, so rather than obfuscating the model-observation comparsion, it has made it more fair.

The difference between the two plots is due to baselining. Climate models are better at modelling the changes in temperature resulting from changes in forcings than they are at accurately modelling absolute temperature (some of the variability in the model runs shown above is due to differences in absolute temperature between model runs). This is well known to those who work with models (although it is such common practice it is difficult to find a paper explaining why this is necessary - it is apparently part of the scientific paradigm). However these differences in absolute temperature are essentially irrelevant to the question of the response of the climate to increasing CO2 radiative forcing. The simple solution is to subtract a constant from each model run so that they agree on the mean temperature over some agreed baseline period.

Here is what the IPCC say about baseline periods (from the TAR):

"A popular climatological baseline period is a 30-year "normal" period, as defined by the WMO. The current WMO normal period is 1961-1990, which provides a standard reference for many impact studies."

Long baseline periods are a good idea as it means that the projections are not very sensitive to the effects of internal variability in the observations and in the models (as it is the average offset over an extended period that e.g. contains several cycles of ENSO). If we use just one year and align all the models and observations to that one year, then if it is a particular warm year in the observations e.g. 1990, the models also start hot and this increases the apparent discrepancy later. If we use a cool year, e.g. 1992, the models start cooler as well, and this would decrease the apparent discrepancy later. However the difference between the two is entirely meaningless as it depends on the noise in the observations. That is why the longer baseline used in the revised diagram gives a more accurate depiction of the difference between models and observations.

So why did the IPCC use a single year baseline in the original diagram? I suspect it is because it makes the trends (rate of warming) easier to see visially if the models and projections all pass through some common point. HOWEVER this does mean that there is a spurious offset between models and observations, that depends on which year in which the data are made to agree.

Far from obfuscating the difference between models and observation, the new diagram presents it more clearly.

  • 3
    It is quite clear that the models are not all in perfect agreement about 1990. For an outsider, it may be confusing why models do not agree about years where we have plenty of measurements. It may be worth noting that climate models do not use measurements (as opposed to weather models and closely related reanalysis models). – gerrit Oct 9 '13 at 16:47
  • 4
    Yes, it is hard for an outsider to understand these issues without first learning what a climate model does and how it operates. They are essentially simulations of the plausible weather that we could observe for a given set of forcings (e.g. changes in atmospheric CO2, solar output, etc.). Just as in real life there will be warm years and cool years within the general long term trends. There is no reason to expect these variations to be syncronised between model runs or between models and observations, hence they don't agree exactly at any point. There is always some uncertainty. – Dikran Marsupial Oct 9 '13 at 17:23
  • 2
    @DikranMarsupial In any other sphere a model of the future would be initiated with actual historic observations and one of the ways of judging the model's quality would be whether it self-corrected from a starting point that was a random high or low. Switching from a start point that is fact to one that is partly opinion (which years feed the trend?) seems like allowing arbitrary after-the-fact judgements to post-adjust the forecast. – matt_black Oct 9 '13 at 21:53
  • 3
    @matt_black The climate system is chaotic. That means that its exact behaviour fundamentally cannot be predicted, because we cannot measure the initial historic observations with perfect accuracy and resolution. That means you can only simulate weather, not predict it. This means that there will be different cycles of internal variability in each run. Adding an offset to bring them into agreement does not initialise them with actual historical observation, and will not synchronise the cycles of internal variability within the models. – Dikran Marsupial Oct 10 '13 at 6:44
  • 2
    As I said in my previous comment, you need to understand the basic principles of the models operation in order to know how to interpret the output. This is not a good forum for an in-depth discussion of this, but I would be happy to discuss this with you at a more suitable forum, such as skeptical science.com, perhaps this would be an appropriate thread skepticalscience.com/climate-models.htm – Dikran Marsupial Oct 10 '13 at 6:46
2

Reading the three charts:

  • The first chart joins the observed values with lines, emphasising volatility in observed values. It has a shorter y-axis scale than the others, making things look more dramatic than the other two charts. The four 2007 scenarios are each shown as a single line rather than a range. The visual suggestion is that observed values are a long way below the scenarios A1B, A2 and B1 but are reasonably close to the Commit scenario.

  • The second chart gives coloured bands for the different reports, with AR4 visible in the middle of these and apparently narrower than TAR ranges. The grey band is not explained in the key. AR4 scenario A1B is emphasised with the yellow double headed arrows and is in the bottom three-quarters of the AR4 band. The x-axis scale from 1990 to 2015 emphasises observations since FAR, SAR ,TAR and AR4. 2010 observations seem to be below the AR4 and A1B ranges but 2011 observations (without an uncertainty bar) below them.

  • The third chart shows a lot of lines presumably of what was in AR4 CMIP3. It also gives a variety of observed points and smoothed lines; the y-axis scale is longer than the others, making observed changes seem smaller. The extension of the x-axis scale to 1950 to 2030 substantially reduces the emphasis on observations since AR4. The background shading seems to emphasise FAR, SAR and TAR rather than AR4. The bars at the right suggest that the individual scenarios A1B, A2 and B1 each had wider uncertainty than TAR. Recent observed values appear to be near the bottom of the very spiky AR4 CMIP3 lines.

To me, the biggest difference between the three graphs is how much uncertainty was associated the three AR4 scenarios A1B, A2 and B1 and how this has been displayed. The first chart shows no uncertainty in each, and has the three very close together to 2020; the second chart suggests a narrower uncertainty for one of them AR4 as a whole and AR4 scenarios together less uncertain than TAR; and the third chart only shows anything about them at all off to the side of the chart, and suggests that each was extremely uncertain, which is why the y-axis scale has to be so long.

The change in x-axis scale between the second and third charts in fact changes the message from the charts: the second appears to say it is a comparison between forecasts and observations with a scale covering observations since the forecasts were made, while the third seems to be saying it was uncertain before the initial forecasts and will be highly uncertain into the future beyond current observations. Compared with the second chart, the third chart de-emphasises the comparison between forecasts and observations and this may be what is underlying McIntyre's point.

If the charts have been modified, it has been with the effect of suggesting increased uncertainty in the past and so subsequent observations appearing more within the (now wider) uncertainty bands, and great uncertainty beyond the current observations. This is presumably deliberate. The real question is whether it is honest history: the IPCC believes that it is.

  • Have you any idea what the commit scenario is? It isn't explained anywhere I can find easily. – matt_black Oct 6 '13 at 23:29
  • Has the IPCC modified their presentation of the data in a way that obscures the discrepancy? – user5582 Oct 7 '13 at 0:07
  • @matt_black: ipcc-data.org/ar4/scenario-COMMIT.html says "An idealised scenario in which the atmospheric burdens of long-lived greenhouse gasses are held fixed at AD2000 levels" so not what has actually happened in the last 13 years. – Henry Oct 7 '13 at 7:26
  • @Sancho An animated gif in the comments on MacIntyre's website suggests that the starting point of the scenarios has been changed. It also illustrates the change in scales. The Skeptical Science site (an anti-climate sceptic blog) explains this as moving the starting point of the scenarios from the actual temperature anomaly in 1990 to its trend value in 1990. – Henry Oct 7 '13 at 7:45
1

The data visualization choices made by the IPCC are poor and have the effect of obscuring the differences between model forecasts and observed temperatures.

Note: this answer assesses whether the IPCC have done a good job of presenting the comparison of observed temperatures and model projection. This, I think, is what a good skeptic would demand if the purpose of presenting the data is to assess whether the models have any predictive skill. This may not be the intention of the IPCC presentation, of course. Also, I am not assessing the IPCCs conclusions, merely their choices of presentation methods.

The key problem the IPCC or other climate modellers have to face is how to present the results of climate projections which contain a great deal of uncertainty. Models don't predict a single path of future temperatures: the short term climate is chaotic and, even if we know all the inputs with precision (which we don't) there are many factors that create many paths into the future. So even a single model will give an ensemble of possible paths into the future. So one key question is how to describe that uncertainty in a visual way that does not misrepresent the nature of the uncertainty.

Another problem is that there are multiple scenarios for what will happen in the future. The key ones use different assumptions about how the world will respond to climate change (will we constrain the output of CO2 or just keep producing increasing amounts each year?

The initial IPCC chart does a half-assed job of the first problem and a poor job dealing with the second. The chart presents the envelopes of forecasts produced by particular models, which is commendable, but incomplete and it fails to account for the shape of the probability distribution of possible temperature paths (examples of better approaches below). The second problem is that the chart tries to show multiple envelopes from multiple model versions. This just confuses as the envelopes overlap and it is hard to get a clear visual impression of the particular envelope for any one model. In fact the choice of models to show is particularly confusing as the envelopes are shown for the key models from previous IPCC rounds despite the obsolescence of the old models and their irrelevance to whether current models are doing a good job. A better approach, if the goal would be to show one chart for each group of models showing the projections made at each previous report with a comparison of all subsequent actual observations. Small Multiples of this sort are a very effective visualisation technique (see this excellent discussion with many examples at Junk Charts). The IPCC even used this approach when they presented the original AR4 forecasts (why not, it seems reasonable to ask, just repeat these charts in the current report with observed temperatures overlayed?):

IPCC model forecasts from AR4

There are good techniques for unambiguously presenting statistical ranges on forecasts. One of the best is used by the Bank of England (BOE) for its inflation forecasts. Inflation forecasts have a lot in common with climate forecasts: they are based on complex models of the behaviour of the economy, projected from historic observations but containing chaotic noise. They, again like climate models, produce a range of possible paths for future inflation. An example is given below:

economist version of BOE forecasts

The economist article the chart is derived from also discusses the technique explaining it like this:

In 1996, the Bank introduced the innovation of presenting the forecast as a probability distribution, in order to highlight the risks to the central forecast. These forecasts, which are agreed by the Monetary Policy Committee (MPC), are called “fan” charts because the expected outcomes spread out in bands from the most likely path of inflation to cover 90% of the probability distribution.

The advantage of the BOE approach is that it gives the right visual impression that paths near the edge of the envelope are less probable than those in the middle. The IPCC envelopes in the original chart equally weight the whole envelope of forecasts giving the (incorrect) impression that edge paths are just as likely as central paths.

The final version of the IPCC charts compounds this problem by also adding many individual paths to the plot. Visually this overweights the extreme paths at the edge of the probability distribution far too heavily compared to the much larger number of paths near the middle of the distribution.

The final presentation also botches the second task even more thoroughly by including multiple AR4 scenarios despite the fact that only one of those comes close to representing the observations (such as CO2 emissions) which were uncertain at the time of the projection but are now history (why include scenarios assuming tight emission constraints when we know that didn't happen?)

The IPCC could have done a much better job. For example, the chart below shows a much clearer view (I'm not judging the data here, just the presentation format):

hawkins comparison chart

This chart is from Judith Curry's blog but is initially derived from Jeff Hawkings here. Surprigingly, there have been very few, if any, attempts to either do this comparative analysis or to present the results well. As Judith Curry observes:

In the midst of substantial public interest on this issue, there is no published analysis that I know of that compares CMIP5 simulations to observations, although it looks like Ed Hawkins’ analysis is heading towards publication.

An alternative and simpler view would show a snapshot of the average performance of the models up to now versus the actual temperature record (this view simplifies the paths taken over time so loses some information). An example of such an analysis is shown below:

pielke version of nature chart

The original source for this is a recent paper in Nature (paywalled). The chart makes a direct comparisons among CMIP5 forecasts (the grey distribution) and the observed path of temperature since the forecasts were made. (Note this was published after the Curry comment quoted above).

Summary

The effective visualization of complex data is difficult. But good advice exists (for example: see the guru of the field Edward Tufte's work; the Junk Charts readable and topical blog; or the usually excellent examples at Flowingdata. Unfortunately this advice is mostly ignored, especially by scientists who often seem to believe "the data in the chart was fine; what's the problem?"

In pure visualization terms the IPCC's changes made an average chart much worse and obscured things a reasonable skeptic would have preferred to be clarified. The visualization changes made any comparison between model projections and observations much harder and much less statistically sound. In their defence, perhaps clarity was never their goal and they didn't have any competent visualization advice.

  • 1
    I think you need references to support strong claims like "The initial IPCC chart does a half-assed job of the first problem and a poor job dealing with the second", "it is hard to get a clear visual impression", "it gives the right visual impression that paths near the edge of the envelope are less probable than those in the middle", "[t]he visualization changes made any comparison between model projections and observations much harder", etc. Evaluation of the presentation methods through user studies could support such statements. – user5582 Oct 26 '13 at 17:35
  • 1
    You also assume that "[t]he key problem the IPCC or other climate modellers have to face is how to present the results of climate projections which contain a great deal of uncertainty". How do you know this? How do you know that they didn't have other problems they were trying to solve with their choice of presentation? – user5582 Oct 26 '13 at 17:37
  • 1
    @Articuno We don't need new user studies to confirm the experience of decades data visualization experience. I illustrated by example how a better job could have been done. If you think they don't work as examples, explain why (after reading some of the visualization links I provided.) – matt_black Oct 26 '13 at 17:48
  • 1
    I've studied information visualization. In this field, nobody publishes a claim of reduced or improved performance by simply referring to the chart's use of standard techniques or lack thereof. Sure, they motivate their design choices by reference to standard techniques, but performance claims are always supported by a user study, and they don't always come out as expected. – user5582 Oct 26 '13 at 17:49
  • 1
    Love the fact that this has 4 downvotes but only one person has told me why they disagree with my analysis. – matt_black Oct 28 '13 at 14:09
0

Evaluating information visualizations

To determine whether or not the published chart "obscures the discrepancy" compared to the draft versions of the chart, we'd need to conduct an empirical evaluation of the visualizations. (Chen, 2000)

Visualizations are commonly measured against accuracy and efficiency metrics. "Accuracy measures typically include precision, error rate, the average number of incorrect answers and the number of correct document retrieved. Efficiency measures typically include the average time to completion and the performance time." (Chen, 2000)

Identification of the task to be performed is a required step in the experimental design. (Card, 1997; Chen, 2000; Schulz, 2013) Here is just a small sample of the range of tasks that have been proposed in the information visualization community: exploring the data, presenting data, finding similarities/differences, visualizing parameter distributions, visualizing parameter variations, visualizing outliers, gaining overview over a dataset, finding correlations, creating a high quality presentation, searching for characteristic features, and finding relations/phenomena/effects. (Schulz, 2013)

What task is this chart made for?

Even if it did "obscure the discrepancy" in a way that slows down a viewer's analysis, perhaps the chart was chosen for improvement in other metrics, such as accuracy of that analysis, or ability to answer questions about a wider range of projections. As much as we don't know the effect of this choice of visualization, we don't even know the intent.

My impression

If you want a sample size of 1 from a bad experiment, I can still see the discrepancy in the published chart.

References

Card, Stuart K., and Jock Mackinlay. "The structure of the information visualization design space." In Information Visualization, 1997. Proceedings., IEEE Symposium on, pp. 92-99. IEEE, 1997.

Chen, Chaomei, and Yue Yu. "Empirical studies of information visualization: a meta-analysis." International Journal of Human-Computer Studies 53, no. 5 (2000): 851-866.

Schulz, Hans-Jörg, Thomas Nocke, Magnus Heitzler, and Heidrun Schumann. "A Design Space of Visualization Tasks." IEEE transactions on visualization and computer graphics 19, no. 12 (2013): 2366-2375.

  • The only problem with this answer is that the principles of good visualization are well know from repeated observations so the IPCC's chart must be judged against that and is not a single experiment. It isn't good by those standards by any metric. – matt_black Oct 26 '13 at 17:29
  • You have no idea what the subjective experience a typical user will get from a particular chart without doing user studies. What are the time constraints that a typical viewer has when looking at the presentation? Do they have the benefit of the text? What is the task being judged? Are they trying to extract trend differences? Or absolute values? It is false to claim that the IPCC's chart must be judged against principles of good visualization. It can be judged against them, but any results of that judgement is only opinion without user studies to back them up. – user5582 Oct 26 '13 at 17:43
  • At best you can say that the visualization failed to use standard techniques X, Y, and Z, but you can't conclude that the visualization resulted in reduced performance on any particular task. – user5582 Oct 26 '13 at 17:46
  • I've studied information visualization. In this field, nobody publishes a claim of reduced or improved performance by simply referring to the chart's use of standard techniques or lack thereof. Sure, they motivate their design choices by reference to standard techniques, but performance claims are always supported by a user study. – user5582 Oct 26 '13 at 17:48
  • I'm a practical dataviz guy not an academic. And I just checked one of my Tufte books and he doesn't often refer to empirical evidence. Nor does he need to: rubbish dataviz usually violates core principles built from years of experience. Perhaps for new types of dataviz we need to test what works, but this isn't some new innovation; clearly better approaches already exist for similar types of data. – matt_black Oct 26 '13 at 17:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .