A recent post on the climate skeptic blog climateaudit.org showed some comparisons of the relationship between world temperature and CO2 over the 20th century from a simple model originally published in the 1930s and a recent complex model from the UK Met Office. The simple model seemed to do a better job of predicting temperature given the CO2 level.

The comparison chart is shown below:

climateaudit comparison chart

Climateaudit then compared the results from many other models using a measure of predictive skill and concluded:

In addition to calculating the skill score of HadGEM2, I also calculated skill scores for the 12 CMIP5 RCP4.5 averages on file at KNMI. ... Remarkably, none of the 12 CMIP5 have any “skill” in reconstructing GLB temperature relative to the simple GCM-Q formula. Indeed, 10 of 12 do dramatically worse....

Note that this comparison is not being used here to deny climate change: the simple model predicts significant warming in the world climate (though with a lower sensitivity than typical modern models). The issue is whether complicated models of climate change do a better job than simple models.

So, is climateaudit right? Are, as many (non-climate) modellers might intuit, simple models better than complex ones in predicting average climate?

  • 9
    You have to be very careful with McIntrye's ClimateAudit - he's very adept at seeming to state one thing, leading readers down the wrong path, while maintaining plausible deniability and actually subtly stating something different. Contrary to this question, he's not claiming that "simple, old models of the effect of CO2 predict temperature better than complex modern climate simulations" He's claiming that an old model recalibrated with new coefficients gives a closer fit (according to his cherry-picked metric) to a cherry-picked data set than some cherry-picked more recent models.
    – 410 gone
    Jul 30 '13 at 7:29
  • 1
    @EnergyNumbers that doesn't seem to be all that different than any scientists claim about their preferred model
    – Ryathal
    Jul 30 '13 at 12:15
  • 1
    @EnergyNumbers That's a rather ad hominem attack for a skeptic. I was hoping for an insightful analysis about whether what he says is reasonable on the basis of the science used. I thought he was arguing that the old model does a better job given observed CO2 levels, which doesn't sound like an unreasonable test or cherry-picking. If you are right, show me where i'm wrong.
    – matt_black
    Jul 30 '13 at 19:28
  • 6
    @Ryathal: I beg to differ. Any scientist with some sense understands what models are (simplifications of reality under certain defined assumptions) and what their favourite model's weaknesses are. Also, overfitting is often not a good thing. Rather than checking whether a model fits the data well one should check whether it can be applied to other data sets that differ from the one on which the model was generated and -in case of predictive models- whether it can actually predict stuff...
    – nico
    Jul 30 '13 at 20:14
  • 3
    Also "I hope to provide further details on the model in the future. In the meantime" (also from climateaudit.org/2013/07/21/…). This is a recipe for misleading the reader, if the model is sound, the skeptical reader should be provided with the means of checking the analysis. Sadly CA is so full of snide remarks that it puts me off reading any further. There is also the question of why pick a particular GCM (HadGEM2), the reason CMIP5 has a multi-model ensemble is to capture the uncertainty in the form of the model.
    – user18604
    Sep 16 '13 at 9:31

To begin with, the question ought to be reworded as "Can simple, old models of the effect of CO₂ predict global mean surface temperature better than complex modern climate simulations?". GMSTs can be modelled quite well by comparatively simple models of global climate, using physics that has been well understood for many years (see the primers by Pierrehumbert and McGuffie and Henderson-Sellers). The reason that climatologists use GCMs rather than these simple climate models is GCMs can model regional climate, so that for instance we can project the effect of increasing GHGs on Europe, or Australasia or the Arctic. The simple climate models cannot do this as they have no concept of the spatial element of climate. So even if simple, old models can provide a better prediction of observed GMSTs, that doesn't mean that they are more useful than GCMs, as they don't provide the level of spatial detail necessary.

The next important part is that there is a big difference between the questions "Can simple, old models ... ?" and "Do simple, old models ... ?" Callendars simple, old model is only one of many, there were others with higher climate sensitivity than Callendars (there is a nice compilation of classic papers on climate edited by Archer and Pierrehumbert). For instance, Gilbert Plass suggested a climate sensitivity of 3.6ºC per doubling of CO2. Had ClimateAudit used that figure instead of Callendar's more modest 1.67 deg, I suspect that the GCM-Q model would have shown much more warming than from HadGEM2 and the conclusion would have been the exact opposite.

The important point here is that ClimateAudit isn't using GCM-Q as a genuine prediction as Callendar's low climate sensitivity was selected in the knowledge that a low climate sensitivity seems to match the observed climate better than a high one, having already seen the observations to be predicted. A skeptic ought to ask why ClimateAudit chose Calendar's value of climate sensitivity, rather than any of the higher historical estimates that could have been selected? A better analysis would have been to perform a survey of historical estimates of climate sensitivity and plot the results for each estimate using GCM-Q. This would basically show that modern GCMs lie well within the span of historical estimates, but that if you were so inclined, you could select a historical model that was closer to the observations than the modern models. Would that tell us anything surprising or that we didn't already know? I would say "no, not really".

So, does the fact that a lower climate sensitivity seems to fit the observations mean that the modern GCMs with higher climate sensitivities are wrong? No, sadly it isn't as simple as that. The observed climate is a combination of the forced response (i.e. the response of the climate system to a change in the forcings, such as CO2 or solar) and the unforced response (a.k.a. "natural variability", "weather noise" etc., which is changes in the climate that are not directly due to the forcings, such as oscillations in ocean currents, such as ENSO etc.). Now the unforced response is chaotic, which means that it is deterministic, but cannot be predicted a long way into the future because it is extremely sensitive to the initial conditions. This means that GCMs can only simulate the effects of the unforced response that are statistically plausible, but cannot predict them as we don't have sufficient information regarding the initial conditions. The best we can do is to form an ensemble of model runs and take the average. The unforced responses in individual runs will not be coherent, and thus will largely cancel out, leaving us with just an estimate of the forced response (which is also what GCM-Q gives us). However, in comparing with the observations, we need to bear in mind that we are not comparing apples with apples, but apples with oranges. The models give us an estimate of the forced response only, but the observations are a combination of forced and unforced response. So the difference between the two may be due to the models being systematically wrong (i.e. their climate sensitivity is too high) or because the effects of the unforced response has been cooling, rather than warming, which makes climate sensitivity over the period of observation look lower than it actually is, or a bit of both. We only have one observed climate, so we can't work out from the observations which is which, the best we can do is to look at the spread of the model runs (which gives us an idea of the plausible variability due to the unforced response) and see if the observations lie in the spread of the runs. This is as accurate as the GCMs can plausibly be expected to be, and this is pretty much what climatologists actually do (see below).

enter image description here

Edit: Found a corresponding figure for the 20th century. Note the spread of the model runs (which is our best estimate of the plausible variability due to the unforced response) is pretty broad. There is no good reason to expect the observed climate to lie any closer to the modern GCM ensemble mean (or indeed GCM-Q) than that.

enter image description here

So all in all, there is nothing really surprising here, at least not to anyone familiar with the operation of climate models and aware of the existence of other "old, simple models" (on this case estimates of climate sensitivity) that could equally have been discussed, but which were not.

  • Surely the key point of more complex models is to do a better job of estimating the underlying climate sensitivity? If they don't achieve that, whether because climate is too noisy or because their internal assumptions are wrong, isn't that a problem? The comparison between an old model's and a complex model's predictions are merely designed to highlight relative predictive value, which may well come down to which has the better sensitivity estimate.
    – matt_black
    Sep 18 '13 at 13:17
  • 1
    As I point out in the last paragraph of my answer, comparing model output with observations of 20th century climate is not a good indicator of the accuracy of an estimate of climate sensitivity. Paleoclimate data seems to provide a more reliable constraint on climate sensitivity (e.g. nature.com/nature/journal/v491/n7426/full/nature11574.html). Also the CA post does not highlight predictive value of the old models as the old model was selected a-posteriori, whilst ignoring the range of other values of CS that had been proposed in the journals of the time.
    – user18604
    Sep 18 '13 at 14:56
  • 1
    Just to add, the advantage of looking at paleoclimate is that it is looking at climate over a timescale long enough for the effects of unforced variability to be small (as they are largely cyclical, so average out in the long run) in comparison with the effects of the forced response. Sadly model-observation comparison is not as straightforward as it seems, the key mistake is to think that the model output is a direct prediction of the observed climate, rather than only of the forced component.
    – user18604
    Sep 18 '13 at 15:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .