In Last Week Tonight of June 8th, 2020, John Oliver mentions that 1 in 1,000 black men can expect to be killed by the police. At 5:05 : https://www.youtube.com/watch?v=Wf4cea5oObY

I have tracked the source to Edwards, Lee & Esposito, 2019, "Risk of being killed by police use of force in the United States by age, race–ethnicity, and sex", PNAS, https://www.pnas.org/content/116/34/16793

This numbers seems surprisingly high, but I am not well equipped to assess the validity of the method used in the research cited above.

Is this statistic in line with reality?

  • Comments are not for extended discussion; this conversation has been moved to chat. – Jamiec Jun 11 '20 at 8:22
  • 1
    Please explain exactly why you are skeptical about the research or conclusions of that peer-reviewed article? Otherwise what is the basis for this question? – curiousdannii Jul 16 '20 at 2:24

The statement by John Oliver that "1 in 1,000 black men can expect to be killed by the police" is supported by research of the highest calibre. There is a peer reviewed paper in one of the most reputable journals in the world, The Proceedings of the National Academy of Sciences that states exactly that.

We use data on police-involved deaths to estimate how the risk of being killed by police use of force in the United States varies across social groups. We estimate the lifetime and age-specific risks of being killed by police by race and sex. We also provide estimates of the proportion of all deaths accounted for by police use of force. Risk is highest for black men, who (at current levels of risk) face about a 1 in 1,000 chance of being killed by police over the life course. The average lifetime odds of being killed by police are about 1 in 2,000 for men and about 1 in 33,000 for women.

I am aware that the study was mentioned in the question. That does not change its quality or reliability.

  • 1
    Comments are not for extended discussion; this conversation has been moved to chat. – Oddthinking Jun 11 '20 at 5:12
  • I don't have rep here to answer, but: if we assume a 60 year lifetime, a person must survive 60 trials to not be killed by police. If there is a 0.001 chance of death, there is a 0.999 chance of survival. e.g.: 0.999 = p^60. The probability of not being killed each year is p=0.9999835. (1-p)*20 million black males is 330 expected deaths per year. This is only a little higher than reality. – Ron Jensen - We are all Monica Jun 11 '20 at 17:16

Putting aside the reputation of the journal for a second, it is nevertheless possible to examine carefully the citation and methodology of the paper and find it to be a good representation of reality. Sadly, as is often the case with research, it requires leaping down rabbit-holes of citations upon citations, but the entire point is that for anybody interested in doing so (such as a skeptic), it is possible to do so. For the sake of brevity, I'll only be examining 1-layer deep (i.e. i'll be looking at sources that this paper cites, but I won't look too deeply into the citations present in those cited works.

The veracity of this study is reliant on two things being valid:

  1. The data that this study uses must be reliable.
  2. The methodology used in analysing this data must be sensible and reproducible.

In the first place, the paper cites that it obtains its data chiefly from https://fatalencounters.org/ -- it then goes on to cite three separate studies that show that this data is more reliable than official data from the government, which is shown to often undercount police-killings. These studies are here, here and here

Let's first note that fatalencounters.org specifically only contains documented deaths that occurred with police involvement, and admits to undercounting in certain specific cases, e.g.

Government data also suggests that vehicle-pursuit deaths are often not reported in news media, so our data certainly understates that total (outside of 2013-14, which include the National Highway Traffic Safety Administration data from USA Today).

In its methodology, it lists where it obtains its data from:

1) Paid researchers; 2) Public records requests; 3) Crowdsourced data.

Out of the 6,900 documents we have on June 15, 2015, around 85 percent have been submitted by researchers we pay to log data.

Their methodology for collecting and aggregating data seems reasonable, as is their verification process for crowdsourced data. I don't want to bog this post down with blockquotes; the information is readily accessible on the website itself.

So where does this leave us? Well, we have a database with a clear objective and methodology, one that goes through great lengths to prevent duplication of data (i.e. a database that prevents overcounting), and one that also cannot guarantee a lack of undercounting. It is well maintained and actively used. Additionally, we have 3 studies, indicating

  1. Law-enforcement related deaths in the US are indeed countable, as evidenced by the fact that even international bodies are capable of keeping a database of them.
  2. Law-enforcement related deaths are underreported in official statistics, as evidenced by a pretty solid capture-recapture analysis (note if you want to know more about this, the stats stackexchange is the place to ask about it!)
  3. Media-reported law-enforcement-related deaths (which is where fatalencounters.org and the like obtain and/or verify the vast majority of their data) are indeed able to be corroborated with medical records, indicating that they are generally a reliable source of this kind of data.

Additionally, the study itself has removed a lot of data that one might consider to actually be a law-enforcement related death:

We focus exclusively on police use-of-force deaths and exclude cases from the analysis that police described as a suicide, that involved a vehicular collision, or that involved an accident such as an overdose or a fall.

So all-in-all, I would say that the data source is pretty reliable. It is sourced in a sensible manner and studies have shown the data to be reliable.

So what's next? We need to analyze the methodology of the study. Again, for specific knowledge related to this it's best to head to https://stats.stackexchange.com/ and ask specific questions about things you may not understand.

To deal with missing data, they employ Multivariate Imputation by Chained Equations (MICE), which is considered a fairly standard method for dealing with missing data in psychology (and I imagine, in sociology and criminology and other social sciences in general). So the overall methodology is at least popularly used and not something out of left field.

They even provide their data and code, allowing anybody to independently run their analyses and/or verify the accuracy of that code! The codebase itself is not exactly million-dollar-production-code quality, but it's written easily enough to be followed by anybody comfortable with R, and contains brief but decent-enough comments that motivate the rationale of certain sections of code. All in all, the code quality is actually surprisingly readable for academic code. Considering the isolated goal of this code (to produce this particular study), the code appears to do exactly what its supposed to do. If I wanted to be more skeptical I'd run the code myself in R, but I presume that people in the peer-review process should've done this already.

Admittedly there's a lot of sensitive statistical analysis happening in this paper (such as using synthetic cohorts as a replacement for tracking a cohort over their entire life) but nothing here suggests that they've wrongfully used some incorrect statistical methodology; rather it just means that they're trying to engage in a very difficult problem to solve, which can be error-prone so actually running the numbers is generally a good idea -- perhaps when I get access to a computer with R I shall run these numbers myself and do some sanity checks.

It's worth pointing out that whether or not the study has been executed properly is, again, an issue for statisticians rather than skeptics. The skeptic (in my opinion) chiefly is concerned with the design of the study (does it use the correct tools?) and the veracity of the data used. And on these two points, I think it's fair to say that the study has succeeded in capturing reality, at least so long as the execution was correctly done. Presumably, peer-review is meant to help address that, but the fact that they've openly shared their data and code is a big boon, indicating that they are committed to capturing reality with their statements.

  • 11
    This is the best answer, almost a peer review of the paper containing a claim. Thanks Marko, and welcome to Skeptics! – gerrit Jun 10 '20 at 7:16
  • The statistic may be "correct" - but excludes far too many factors: including the likelihood of even having a police interaction in the first place – warren Jun 12 '20 at 18:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .