Air pollution is a major health concern worldwide. One of the major problems is the emissions caused by traffic such as nitrogen dioxide and the small particulates known as PM 2.5 (which are very small carbon particles associated with vehicle engines, especially diesels).
The UK, for example, recently estimated that the combined effect of vehicle emissions could be causing more than 30,000 early deaths per year (see this report, for example).
There have been some questions about the accuracy of some of these estimates (see this question Does particulate pollution in outdoor air kill tens of thousands every year? ).
But statistician William "Matt" Briggs suggests in this video on the ecological fallacy (see his specific comments using PM 2.5 studies to illustrate the fallacy at about 1:34 in the video) that essentially none of the major studies have actually measured exposure to PM 2.5s in the populations they study, instead they have all used proxies for that exposure (so we could be seeing effects that are ecological not actually caused by the pollution and so overestimating or underestimating their harms). See also his blog entry about the video which has some references such as this one, a major study on mortality vs PM 2.5s which he claims contains a double ecological fallacy with both the outcome and the exposure been measured by proxy models.
For those wo don't want to watch his video here is a quote from one of his blogs describing his interaction (presumably as a statistical advisor) with the California Air Resources board:
Yours Truly was involved in a critique of a study submitted to the California Air Resources Board (CARB) which claimed to have discovered a correlation between air pollution (X; particulates of a certain size) and heart disease (Y). A weak, barely there finding of statistical “significance” was enough to embolden CARB to create new and enhance old air pollution regulations in order to “save lives.” Yet X was never measured.
At a very few places, particulate measures were taken for a limited time. The air pollution in these places was then crudely extrapolated to areas in which it was not measured. Finally, the extrapolated air pollution nearest the address of the study participants (where they lived at one time, ignoring moves) was taken as the exposure; this was their W. Nobody knows how much air pollution anybody was actually exposed to.
The sequel to this story is fascinating. I submitted written critiques where they were discussed at a CARB meeting. One panel member thanked me, called me learned, and took my criticisms of the epidemiologist fallacy seriously. But it was judged that—and here you must laugh—because the fallacy was so common that it led to many results referenced by CARB, that this current study was no different. And therefore acceptable.
Given the perceived importance of air pollution to health it seems to be a fairly significant claim that we might not be measuring the true effects but a false ecological effect instead. This makes his claim an important one to test: Have few if any studies on the health effects of PM 2.5 particulates directly measured the exposure to the particles?
Clarification of the claim and the evidence required to refute it
It is clear from the comments that this question has been severely misinterpreted by many readers, often for reasons that have nothing to with the claim itself and everything to do with who made it. Briggs has heretical views on climate science (which seems to be the source of much commentator ire) but is also a professional statistician with published textbooks like this one, so dismissing his complaint because he doesn't know anything about the topic is an extremely weak skeptical response.
And a little clarification (from the Encyclopaedia of Epidemiology) of why the ecological fallacy matters in epidemiology (my emphasis):
Exposure assessment is a critical component of epidemiologic research, and unfortunately for many studies, it is this component that introduces many limitations. Exposure assessment has been defined by Last (2001) as the ‘process of estimating concentration or intensity, duration, and frequency of exposure to an agent that can affect health’. It involves preferably quantitative, but often qualitative, procedures to estimate and assign an individual’s past or current exposures...
...Ecological fallacy: "The ecological fallacies in the three examples above arise from assuming that all individuals in each ecological group have the same summary measure (the mean value) of the group without accounting for possible confounding by other variables and for the unobserved heterogeneity of individuals in each group.
In view of this, when statistically significant association is found between exposure and health outcome at group level (usually aggregate data are easily assessable as they already exist, having been previously collected for other purposes), then individual-level data should be collected to obtain the joint distributions of exposures and outcomes. This would make it possible to test the ecological hypothesis thus generated so as to corroborate or refute the putative ecological association at the individual level. This is because for causal inference, individual data are required to account for population heterogeneity and confounding bias"
The Briggs argument is, essentially, that what is recommended here has not, usually, been done for epidemiological studies on particulates.
More importantly, this claim is, in principle, easy to refute (despite repeated claims in the comments that it can't be).
The claim is this. Many studies exist that try to address the relationship between particulate exposure and health. We can observe population health. And we can compare that to exposure to pollution. So far, so good. The problem is that, he alleges, many of these studies use data on exposure that is unreliable. He claims few if any studies use better exposure data.
Often these use models that estimate that exposure based on zip code of resident or even, county-level averages. He says we need better measurements of the real exposure. For example, a persons' exposure might be much worse if they travel to work via a busy road. Or far lower if they work in an air conditioned office 8 hours a day. Or the indoor exposure could be far lower than the typical outdoor exposure in their neighbourhood.
Some commentators claim that we simply can't use better exposure measurements. But portable meters exist so could easily be used to validate the models. Briggs is not demanding that we run unethical randomised controlled trials exposing people to different levels of pollution: he is questioning whether we have measured the actual individual exposures to the pollutants reliably. The issue is whether the input data on current epidemiological studies is reliable.
And this is, or should be, easy to refute. If there are good studies showing that a particular model of exposure to PMs shows a strong relationship to measured individual exposures, then his claim can be dismissed. If there are strong, convincing reasons why this is impossible to do, that would be a weaker, but still useful, challenge to his claim.
In short, he seems to be questioning whether we have reliable data about exposure in current studies on the effect of PM 2.5 particulates. This seems to be a perfectly reasonable question and one that should be easy to test.