Putting aside the reputation of the journal for a second, it is nevertheless possible to examine carefully the citation and methodology of the paper and find it to be a good representation of reality. Sadly, as is often the case with research, it requires leaping down rabbit-holes of citations upon citations, but the entire point is that for anybody interested in doing so (such as a skeptic), it is possible to do so. For the sake of brevity, I'll only be examining 1-layer deep (i.e. i'll be looking at sources that this paper cites, but I won't look too deeply into the citations present in those cited works.
The veracity of this study is reliant on two things being valid:
- The data that this study uses must be reliable.
- The methodology used in analysing this data must be sensible and reproducible.
In the first place, the paper cites that it obtains its data chiefly from https://fatalencounters.org/ -- it then goes on to cite three separate studies that show that this data is more reliable than official data from the government, which is shown to often undercount police-killings. These studies are here, here and here
Let's first note that fatalencounters.org specifically only contains documented deaths that occurred with police involvement, and admits to undercounting in certain specific cases, e.g.
Government data also suggests that vehicle-pursuit deaths are often not reported in news media, so our data certainly understates that total (outside of 2013-14, which include the National Highway Traffic Safety Administration data from USA Today).
In its methodology, it lists where it obtains its data from:
1) Paid researchers; 2) Public records requests; 3) Crowdsourced data.
Out of the 6,900 documents we have on June 15, 2015, around 85 percent
have been submitted by researchers we pay to log data.
Their methodology for collecting and aggregating data seems reasonable, as is their verification process for crowdsourced data. I don't want to bog this post down with blockquotes; the information is readily accessible on the website itself.
So where does this leave us? Well, we have a database with a clear objective and methodology, one that goes through great lengths to prevent duplication of data (i.e. a database that prevents overcounting), and one that also cannot guarantee a lack of undercounting. It is well maintained and actively used. Additionally, we have 3 studies, indicating
- Law-enforcement related deaths in the US are indeed countable, as evidenced by the fact that even international bodies are capable of keeping a database of them.
- Law-enforcement related deaths are underreported in official statistics, as evidenced by a pretty solid capture-recapture analysis (note if you want to know more about this, the stats stackexchange is the place to ask about it!)
- Media-reported law-enforcement-related deaths (which is where fatalencounters.org and the like obtain and/or verify the vast majority of their data) are indeed able to be corroborated with medical records, indicating that they are generally a reliable source of this kind of data.
Additionally, the study itself has removed a lot of data that one might consider to actually be a law-enforcement related death:
We focus exclusively on police use-of-force deaths and exclude cases from the analysis that police described as a suicide, that involved a vehicular collision, or that involved an accident such as an overdose or a fall.
So all-in-all, I would say that the data source is pretty reliable. It is sourced in a sensible manner and studies have shown the data to be reliable.
So what's next? We need to analyze the methodology of the study. Again, for specific knowledge related to this it's best to head to https://stats.stackexchange.com/ and ask specific questions about things you may not understand.
To deal with missing data, they employ Multivariate Imputation by Chained Equations (MICE), which is considered a fairly standard method for dealing with missing data in psychology (and I imagine, in sociology and criminology and other social sciences in general). So the overall methodology is at least popularly used and not something out of left field.
They even provide their data and code, allowing anybody to independently run their analyses and/or verify the accuracy of that code! The codebase itself is not exactly million-dollar-production-code quality, but it's written easily enough to be followed by anybody comfortable with R, and contains brief but decent-enough comments that motivate the rationale of certain sections of code. All in all, the code quality is actually surprisingly readable for academic code. Considering the isolated goal of this code (to produce this particular study), the code appears to do exactly what its supposed to do. If I wanted to be more skeptical I'd run the code myself in R, but I presume that people in the peer-review process should've done this already.
Admittedly there's a lot of sensitive statistical analysis happening in this paper (such as using synthetic cohorts as a replacement for tracking a cohort over their entire life) but nothing here suggests that they've wrongfully used some incorrect statistical methodology; rather it just means that they're trying to engage in a very difficult problem to solve, which can be error-prone so actually running the numbers is generally a good idea -- perhaps when I get access to a computer with R I shall run these numbers myself and do some sanity checks.
It's worth pointing out that whether or not the study has been executed properly is, again, an issue for statisticians rather than skeptics. The skeptic (in my opinion) chiefly is concerned with the design of the study (does it use the correct tools?) and the veracity of the data used. And on these two points, I think it's fair to say that the study has succeeded in capturing reality, at least so long as the execution was correctly done. Presumably, peer-review is meant to help address that, but the fact that they've openly shared their data and code is a big boon, indicating that they are committed to capturing reality with their statements.
