Did the Center for Naval Analysis recommend that the damaged areas be reinforced on all bombers, creating a famous case of survivorship bias?

Question

One example of this claim from the Statistics How To website:

Abraham Wald’s Naval Work

In World War II, statistician Abraham Wald tried to determine how to minimize bomber losses. Prior to Wald’s work, researchers from the Center for Naval Analysis analyzed bombers that came back with damage and recommended that the damaged areas be reinforced on all bombers. However, they didn’t take into account that only the surviving aircraft came back; the bombers that did not survive likely had damage to other, more critical areas. Wald recognized that survivorship bias played a part in the Center for Naval Analysis’s decision and recommended basically the opposite–the reinforcement of areas that had not been hit on the surviving aircraft.

Is there proof that this historical incident actually happened? The question is specifically about Abraham Wald overriding a previous report from the Center for Naval Analysis.

Answer 1

Firstly, Wald did not work for the Center for Naval Analyses or its predecessor organizations Antisubmarine Warfare Operations Research Group (ASWORG), Operations Research Group (ORG) or Operations Evaluation Group (OEG). Instead, he worked for the Statistical Research Group at Columbia University. The Center for Naval Analyses merely published his work many years later, and didn't exist as such until 1962.

A good history of what occurred is found in Breakthroughs in Statistics: Foundations and Basic Theory, which is also consistent with Abraham Wald's own description, as published in his book Sequential Analysis.

Quoting from pages 249-250 of "Breakthroughs in Statistics..."

the Applied Mathematical Panel of the U.S. Office of Scientific Research and Development established a Statistical Research Group at Columbia University on July 1, 1942. The purpose of the Group was to advise the Defense Department on statistical methods for experiments that were being conducted on a vast scale of war production. Allen Wallis headed the group with Harold Hotelling and Jacob Wolfowitz as its founding members. Wald, Harold Freeman, and Milton Friedman, among others, joined a few months later. In late March 1943, Wallis discussed the following problem with Friedman [presents highly mathematical problem]

This lead Wallis and Freidman to the following conjecture [more mathematics] Wallis and Friedman approached Wald in early April 1943. Wald was initially unenthusiastic, but phoned 2 days later to admit that their conjecture was, indeed, correct.

So, basically the OP is saying other members of the group erred, when in fact they formed the basis of Wald's work.

More particularly, concerning planes, Willis, the head of the group stated:

"The military was inclined to provide protection for those parts that on returning aircraft showed the most hits" Quoting "The Statistical Research Group, 1942-1945" Journal of the American Statistical Association, Volume 75, pages 320-330 (alternative link).

So, yes Wald was going against the earlier convention, but not against a previous recommendation of the group.

For additional information, in 1980 a reprint of Wald's original 1943 work was republished and the full 100 page text is available A REPRINT OF "A METHOD OF ESTIMATING PLANE VULNERABILITY BASED ON DAMAGE OF SURVIVORS"

Answer 2

Not quite a direct answer, but the story of Abraham Wald and the bullet holes is mentioned at length in Jordan Ellenberg's book How Not to Be Wrong: The Hidden Mathematics of Everyday Life (Allen Lane, UK, 2014, 1st ed/hardback). There are some interesting details not mentioned in other answers.

"The [military] officers saw an opportunity, you can get the same protection with the same armor if you concentrate the armor on the places with the greatest need, where the planes are getting hit the most [since armor adds weight and makes planes less maneuverable and less fuel efficient]. But exactly how much armor belonged on those parts of the plane? That was the answer they came to Wald for. It wasn't the answer they got.

The armor, said Wald, doesn't go where the bullet holes are. It goes where the bullet holes aren't: on the engines.

Wald's insight was simply to ask: where are the missing holes? The ones that would have been all over the engine casing, if the damage had been spread equally all over the place? Wald was pretty sure he knew. The missing holes were on the missing planes."

And...

"Wald's recommendations were quickly put into effect, and were still being used by the navy and the air force through the wars in Korea and Vietnam."

Lastly (and I'm not a mathematician!)...

"Why did Wald see what the officers, who had vastly more knowledge and understanding of aerial combat, couldn't? It comes back to his math-trained habits of thought... To a mathematician, the structure underlying the bullet hole problem is a phenomenon called survivorship bias. It arises again and again, in all kinds of contexts. And once you're familiar with it, as Wald was, you're primed to notice it wherever it's hiding."

Ellenberg also gives a few references not mentioned in other answers, including:

1. Howard Wainer, Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies (Princeton University Press, 2011); mentions the bullet holes story.
2. Marc Mangel and Francisco J. Samaniego, "Abraham Wald's Work on Aircraft Survivability", Journal of the American Statistical Association 79, no. 386 (June 1984): 259-67.
3. Jacob Wolfowitz, "Abraham Wald, 1092-1950", Annals of Mathematical Statistics 23, no. 1 (Mar. 1952): 1-13.