Not peer reviewed
The researchers from the University of Denver currently have their paper titled Black lives matter protests, social distancing and COVID-19 posted online as an NBER Working Paper. This means the paper HAS NOT gone through peer review and carries the same scientific weight as a blog.
To reiterate, while this work may be valid, it has not been cross-examined and the media SHOULD NOT have picked it up and sensationalized it!
The authors conjecture that the number of COVID-19 cases may have gone down due to people wishing to avoid the protests
For example, other individuals who did not wish to
participate in the protests, perhaps due to fear of violence from police clashes or general unrest,
may have chosen to avoid public spaces while protests were underway. This could have an
offsetting effect, increasing social distancing behavior in other parts of the population. The net
effect, on both social distancing and on the spread of COVID-19 is thus an empirical question,
and the focus of this study.
Of concern to myself is that the protests are ongoing to which the authors point out
One concern regarding the lack of any strong effects for COVID-19 case growth is that
the post-protest sample period might not be sufficiently long enough as of yet to detect a
resurgence or increase in the infection rates. While this is a possibility, we also note that our
sample includes at least 21 days of data following the early protests that took place in 154 cities
(during the first five days following George Floyd’s death), at least 18 days of data following
protests in 242 cities, and at least 16 days of data for 257 cities that experienced protests (during
the first week following George Floyd’s death).
The author use a 21 day lag-time as their window of safety for evaluating if a city sees a rise in cases. They offer literature sources using this criteria. This does not make it valid, but, it appears to be the norm - insomuch as COVID-19 has norms.
In the body of the results/discussion the authors say that their findings show, with 95% confidence, that mass protest cities did not see a rise in COVID-19. However, in their conclusion they say
Likewise, while it
is possible that the protests caused an increase in the spread of COVID-19 among those who
attended the protests, we demonstrate that the protests had little effect on the spread of COVID19 for the entire population of the counties with protests during the more than three weeks
following protest onset. In most cases, the estimated longer-run effect (post-21 days) was
negative, though not statistically distinguishable from zero
So I am a bit confused. Are the 21 day results passing or not passing the hypothesis test that there is no rise in COVID-19 due to mass protests. They cannot conclude that COVID-19 decreased from the protests if the confidence interval contains 0. In fact, if it even contains 0.0000001, they cannot conclude that it didn't cause a rise in covid-19.
Edit: I previously made it sound like the paper was loaded with unsubstantiated jargon, and this is not accurate, as was pointed out in the comments. My opinion is that there are several areas that need tightening up.
Overall I will say that the work appears for the most part well written and of an academic level. This manuscript has potential but I believe the peer review process is required before I take it seriously. Good reviewers will make them substantiate all discussions involving statistics and will hopefully make their conclusions exactly match their discussion. Also, reviewers will go over their statistical methods to ensure they are acceptable. It is unfortunately the case that given a desired outcome, there is a way of evaluating the data to get that outcome. Reviewers will ensure this has not been done. I am not qualified to grade their metrics and choices.
Finally, the peer review process should take time. It is fine for the authors to publish a working manuscript, I do it myself, but it is absolutely not okay for the media to propagate it before it has been peer reviewed! If this is the case scientists can just start publishing things on blogs.