13

This article in The Register claims that Pokémon Go may have a bad side effect on road safety. The headline is:

Pokémon GO caused hundreds of deaths, increased crashes

And the article summarises the research:

...the game caused “a disproportionate increase in vehicular crashes and associated vehicular damage, personal injuries, and fatalities in the vicinity of locations, called PokéStops, where users can play the game while driving."

The results appear to be based on observation in a limited area extrapolated to the USA.

Are the claimed results and their extrapolation credible?

9
  • 3
    Well you have the research paper that makes the claim linked directly from the article. Have you tried reading it and doing your own peer review of it? As far as I can tell though they extrapolated rather wildly. They looked at one county ("Tippecanoe County") and from that extrapolated to the rest of the nation. Does not sound very solid to me.
    – user32299
    Commented Nov 27, 2017 at 14:27
  • 4
    Bear in mind that just one fatality difference in Tippecanoe County would result in a hugely different assessment for the country as a whole. So I would say that this extrapolation is not enough to say "Pokemon GO caused hundreds of deaths". You would have to qualify that statement by giving the full picture and say "Ok, if we extrapolated the result from one county onto the whole union of states... then we would get such a result". But it is a fictional number, not one borne from actual observation of the country as a whole.
    – user32299
    Commented Nov 27, 2017 at 14:50
  • 1
    I don't know about the death figure, but obviously there is a hidden government subsidy on Pokemon Go. Commented Nov 27, 2017 at 17:42
  • 3
    What is with the posting of "is this paper true" when the paper hasn't even gone through a peer review? Peer reviews are the best answer for such things, but it seems like these papers are getting noticed the day they hit the web sites. That's a dangerous thing in Science, because it means that it gets gamed like news: the more sensational the topic, the more it leaves a footprint in the reader's mind, regardless if it is a strong paper or rubbish.
    – Edwin Buck
    Commented Nov 27, 2017 at 19:08
  • 3
    @EdwinBuck The problem isn't skeptics.SE asking the question, it is newspapers putting unverified science in their headlines. This claims was widely reported. Part of this site's role is to provide thoughtful challenge to those headlines.
    – matt_black
    Commented Nov 27, 2017 at 19:12

2 Answers 2

17

This is the research paper in question.

The research paper was just released Nov 18th, 2017. Typically it takes some time for a proper peer review, and with a proper peer review, the next step for notable research papers is generally publication.

Compared to many other eye raising claims, this one has the form of an authentic claim.

  • The writers are associated with a prestigious university (Perdue)
  • The data covered is specific and detailed (accident reports from Tippecanoe County, Indiana)
  • The paper claims a small increase in accidents (25%) correlated to the locality of Pokemon stops.
  • The paper contains a respectable number of references.

That said, the paper also contains a number of hints that it might not be a groundbreaking, or even fully meaningful finding

  • The accidents are weighted by cost, which confounds understanding of incidence
  • The costs are low, with the delta weighing in just under $500,000 USD, with little indication if this is a statically significant number.
  • 85% of their baseline data was used (because it could be mapped to an intersection), with the rest of the data discarded. No comment was made on the protections against cherry-picking data, which could influence the outcome.
  • They make assumptions about traffic patterns during school breaks, altering their analysis according to their assumptions, which might be in error. They probably should have just discarded
  • Their alternate interpretations sections are victim to "begging the question" Rather than assuming that increase in accidents was due to non-Pokemon activity, they assume that it was due to Pokemon activity of people with parked cars. This assumes that Pokemon activity was present in the alternative interpretations, meaning that alternatives haven't really been explored.
  • Their sample size is embarrassingly low. For example, they do sub-analysis on pedestrian crashes, detailing the 36 pedestrian crashes that were noted, 4 of which occurred during their "with Pokemon" natural testing timeframe. An error of a single person (quite possible with a Poisson distribution) is a 20% to 25% error, which matches too closely to their 25% increase claim.
  • Their data, which is naturally Poisson distributed, is being evaluated with Gaussian Statistical tools.
  • They make speculative projections, without hint to the logic behind the projections.

I'm decent at reading research papers, and the real proof won't come until it is properly peer reviewed; but, really it won't even come then. Many papers which are peer reviewed should never have been passed, and sometimes even junk gets published.

To me, this seems to be an interesting observation, but one that doesn't survive the slant of self-bias. It might be true, but the paper doesn't seem to prove it. To prove it, I would prefer:

  • A paper that detailed accidents at Pokemon stops by modeling a Poisson distribution of the patterns of accident occurrence for that intersection. This is not a trivial thing, because the number of incidents is so low the error on the model would be quite high.
  • A paper that showed the two (before game release, and after game release) accident distributions at those intersections changed in a statistically significant manner, for Poisson distributions.

If their claim is that accidents increased, then it would be a supportable argument that the time to the next accident decreased, over the periods they wished to compare.

I do believe they found a difference, but without proper attention to the domain of the sample space, with so few data points being covered, and with the large number of caveats and structured reasonings about data to be included and excluded, I'm not sure that their differences can be attributed to the control variable (which this being a natural experiment, binding it to a control variable is the hard part).

The paper probably will be cited by laymen, and rejected as "possibly true, but we can't tell" by the scientific community.

6

I think the Science take on this is noteworthy in that it didn't mention the extrapolation, but only

The gaming-related accidents, which increased by 26.5% near Pokéstops over the first 5 months following the game's release, cost the county up to $25.5 million in damages, including the loss of two lives,

And New Scientist solicited another academic's opinion, who said:

“The statistical analyses they performed appear to be sound, correctly applied and actually statistically significant,” says Brandon Schoettle at the University of Michigan.

However, he is sceptical about how applicable the findings are for a wider geography or population. “I believe these results for Tippecanoe county, Indiana, are accurate and probably reliable. But I am not so sure that this county is very representative of the rest of the country,” he says. For example, the county is mostly rural with a major urban area and a major university that has many young, inexperienced drivers. “To claim to be able to estimate larger financial effects for the country is not necessarily supported by the data they analysed,” says Schoettle.

The only thing (nearly) certain is that the amount of press coverage this paper received is going to result in more papers on the topic. As an aside, before this study was puplished one economist argued on this blog that Pokemon Go should be subsidised, because it has positive externalities in fighting crime and obesity. I guess now they'll have to argue/decide on the balance. Aside to the aside, the effectiveness of AVGs (active video games) in reducing obesity isn't very clear so far ...although some physicians have argued that Pokemon Go is different enough to have more potential in this respect.

Coming back to the study in question here, it's a good thing they didn't extrapolate to worldwide deaths, because in a commentary on two other US deaths by Pokemon Go, we find that

From a legal standpoint, the use of a mobile phone for any purpose, including the use of applications, is illegal in the UK and the majority of Western European countries including France, Germany, Spain and Italy. In Tucson, Arizona, from where this case reports comes, there is a restriction on text messaging whilst driving, but no specific restriction on the use of smartphone games. Therefore, it might not be considered illegal to use an app whilst driving even it has been shown to affect concentration.

... but that's less reassuring prevention-wise given that some Pokemon Go accidents involved drivers swerving to avoid pedestrians who were playing the game.

It turns out however that there was one brief report (in a JAMA journal) on nationwide US incidents, to the extent that Twitter and the news are representative:

Thirty-three percent (95%CI, 31%-34%) of tweets indicated that a driver, passenger, or pedestrian was distracted by Pokémon GO, suggesting there were 113 993 (95% CI, 107 084-117 447) total incidences reported on Twitter in just 10 days. In contrast, safety messages were less common (13%; 95% CI, 12%-16%). The remainder of postings (54%) were hypothetical, unclear, or unrelated (Figure). Eighteen percent (95% CI, 17%-19%) of tweets indicated a person was playing and driving (“omg I’m catching Pokémon and driving”) and 11% (95% CI, 10%-11%) indicated a passenger was playing (“just made sis drive me around to find Pokémon”). Four percent (95%, CI, 3%-4%) indicated a pedestrian was distracted (“almost got hit by a car playing Pokémon GO”). There were 14 unique crashes—1 player drove his car into a tree—attributed to Pokémon GO in news reports during the same period.

This was for the July 10-20, 2016 period. Another aside: 110,000 Twitter incidents became 110,000 road accidents when the Daily Mail reported on this study.

Of course extrapolating from that to other time periods would be problematic, and wasn't done by the authors. However, it seems safe to assume this period was during the peak Pokem GO, so if we use this study to extrapolate an upper bound for the number of Pokemon crashes in the US, we get 1.4/day x 148 (the number of days from the Indiana study) = 207 crashes.

In contrast, the extrapolation based on the Indiana study was an "the increase in crashes attributable to the introduction of Pokémon GO is 145,632 with an associated increase in the number of injuries of 29,370 [...] over the period of July 6, 2016, through November 30, 2016" (nationwide). So this extrapolation is two or three orders of magnitude higher than I extrapolated from the JAMA study, depending which class of crashes we consider; perhaps fender-benders without any bodily injury were completely non-newsworthy. Of course, the tough question in this comparison is how well reported by the press were Pokemon crashes in that time period. Were they really underreported by a factor of 100? During a time when Pokemon GO was "hot" in the news?

Perhaps we could use the UK for information/extrapolation, which apparently had tallied such Pokemon incidents nationwide "Robberies, thefts, assaults and driving offences were among 290 incidents recorded across England and Wales throughout July [2016]." Since the US population is about 5 times larger, that would give 1,450 incidents per month... or about 7,250 incidents in ~150 days, again as an upper bound (given that the game waned in popularity after launch). But this latter estimate would include much more than car crashes. (And I know I've underestimated this by 15% or so for the sake of a quick calculation, because the UK is more than England and Wales, so a more correct population factor would have been 5.76, giving an estimate of 8,363 US incidents. But at this level of ballpark estimates, it doesn't matter too much.)

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .