24

One example of this claim from the Statistics How To website:

Abraham Wald’s Naval Work

In World War II, statistician Abraham Wald tried to determine how to minimize bomber losses. Prior to Wald’s work, researchers from the Center for Naval Analysis analyzed bombers that came back with damage and recommended that the damaged areas be reinforced on all bombers. However, they didn’t take into account that only the surviving aircraft came back; the bombers that did not survive likely had damage to other, more critical areas. Wald recognized that survivorship bias played a part in the Center for Naval Analysis’s decision and recommended basically the opposite–the reinforcement of areas that had not been hit on the surviving aircraft.

Is there proof that this historical incident actually happened? The question is specifically about Abraham Wald overriding a previous report from the Center for Naval Analysis.

5
  • 3
    For the record, it seems undeniable that Wald did work in this area. I can't find the report it supposedly "overrides".
    – Is Begot
    Sep 17, 2015 at 16:15
  • 1
    Much the same claim is made in the UK for Patrick Blackett and his operational research team, recommending armour-plating of undamaged parts of aircraft
    – Henry
    Sep 18, 2015 at 1:07
  • 1
    I was in the middle of writing a comment on the just-deleted answer: pages 63-65 of the linked reprint of 1943 memoranda give an example similar to the OP's of using probability/statistics to determine which areas of an aircraft are more vulnerable. Nowhere can I find a mention of whatever previous ideas these formulas might have replaced. Could still be a response to them, of course.
    – Dan Getz
    Jan 30, 2016 at 15:46
  • @DanGetz I'll restore the answer, I'm adding more info. It was the military generally that reinforced areas where returning planes were damaged, not SRG which wasn't part of the military, according to SRG's founder Wallis.
    – DavePhD
    Jan 30, 2016 at 16:17

2 Answers 2

13
+50

Firstly, Wald did not work for the Center for Naval Analyses or its predecessor organizations Antisubmarine Warfare Operations Research Group (ASWORG), Operations Research Group (ORG) or Operations Evaluation Group (OEG). Instead, he worked for the Statistical Research Group at Columbia University. The Center for Naval Analyses merely published his work many years later, and didn't exist as such until 1962.

A good history of what occurred is found in Breakthroughs in Statistics: Foundations and Basic Theory, which is also consistent with Abraham Wald's own description, as published in his book Sequential Analysis.

Quoting from pages 249-250 of "Breakthroughs in Statistics..."

the Applied Mathematical Panel of the U.S. Office of Scientific Research and Development established a Statistical Research Group at Columbia University on July 1, 1942. The purpose of the Group was to advise the Defense Department on statistical methods for experiments that were being conducted on a vast scale of war production. Allen Wallis headed the group with Harold Hotelling and Jacob Wolfowitz as its founding members. Wald, Harold Freeman, and Milton Friedman, among others, joined a few months later. In late March 1943, Wallis discussed the following problem with Friedman [presents highly mathematical problem]

This lead Wallis and Freidman to the following conjecture [more mathematics] Wallis and Friedman approached Wald in early April 1943. Wald was initially unenthusiastic, but phoned 2 days later to admit that their conjecture was, indeed, correct.

So, basically the OP is saying other members of the group erred, when in fact they formed the basis of Wald's work.

More particularly, concerning planes, Willis, the head of the group stated:

"The military was inclined to provide protection for those parts that on returning aircraft showed the most hits" Quoting "The Statistical Research Group, 1942-1945" Journal of the American Statistical Association, Volume 75, pages 320-330 (alternative link).

So, yes Wald was going against the earlier convention, but not against a previous recommendation of the group.

For additional information, in 1980 a reprint of Wald's original 1943 work was republished and the full 100 page text is available A REPRINT OF "A METHOD OF ESTIMATING PLANE VULNERABILITY BASED ON DAMAGE OF SURVIVORS"

5
  • I'm not sure if this answers the question of whether there existed a recommendation by the Center for Naval Analysis (or a similar organization) to reinforce the damaged areas on bombers. Jan 30, 2016 at 14:28
  • 1
    @JonathanReez The problem was originally presented to the group in the context of ordnance testing (shells). The group was recommending wastefully large sample sizes in ordnance testing. Captain G. L. Schuyler pointed out that this was flawed. Wallis and Freidman came up with their conjecture, and Wald further developed the idea and applied it to other areas. The statement in the OP would be correct if it referred to the group making wasteful recommendations about ordnance testing. See letter quoted on page 324 of Decision Theory: Principles and Approaches
    – DavePhD
    Jan 30, 2016 at 14:54
  • @DavePhD: That NCSU link doesn't work for me, I get a "page no longer available" error.
    – paddyr
    Jun 20 at 14:38
  • 1
    @paddyr ok, try this, I will change the link: apps.dtic.mil/sti/citations/ADA091073
    – DavePhD
    Jun 21 at 1:12
  • @DavePhD: Works for me, thanks!
    – paddyr
    Jun 23 at 10:29
0

Not quite a direct answer, but the story of Abraham Wald and the bullet holes is mentioned at length in Jordan Ellenberg's book How Not to Be Wrong: The Hidden Mathematics of Everyday Life (Allen Lane, UK, 2014, 1st ed/hardback). There are some interesting details not mentioned in other answers.

"The [military] officers saw an opportunity, you can get the same protection with the same armor if you concentrate the armor on the places with the greatest need, where the planes are getting hit the most [since armor adds weight and makes planes less maneuverable and less fuel efficient]. But exactly how much armor belonged on those parts of the plane? That was the answer they came to Wald for. It wasn't the answer they got.

The armor, said Wald, doesn't go where the bullet holes are. It goes where the bullet holes aren't: on the engines.

Wald's insight was simply to ask: where are the missing holes? The ones that would have been all over the engine casing, if the damage had been spread equally all over the place? Wald was pretty sure he knew. The missing holes were on the missing planes."

And...

"Wald's recommendations were quickly put into effect, and were still being used by the navy and the air force through the wars in Korea and Vietnam."

Lastly (and I'm not a mathematician!)...

"Why did Wald see what the officers, who had vastly more knowledge and understanding of aerial combat, couldn't? It comes back to his math-trained habits of thought... To a mathematician, the structure underlying the bullet hole problem is a phenomenon called survivorship bias. It arises again and again, in all kinds of contexts. And once you're familiar with it, as Wald was, you're primed to notice it wherever it's hiding."

Ellenberg also gives a few references not mentioned in other answers, including:

  1. Howard Wainer, Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies (Princeton University Press, 2011); mentions the bullet holes story.
  2. Marc Mangel and Francisco J. Samaniego, "Abraham Wald's Work on Aircraft Survivability", Journal of the American Statistical Association 79, no. 386 (June 1984): 259-67.
  3. Jacob Wolfowitz, "Abraham Wald, 1092-1950", Annals of Mathematical Statistics 23, no. 1 (Mar. 1952): 1-13.
3
  • 1
    This seems to be a repeat of the claim, rather than an answer. What do the references say?
    – Oddthinking
    Jun 20 at 23:55
  • @OddThinking: (2) and (3) say nothing on the specific question of whether Wald "overrode" anyone. I don't have access to (1).
    – paddyr
    Jun 21 at 9:25
  • -1: You have no evidence here that the claim is correct.
    – Oddthinking
    Jun 21 at 11:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .