Does "Unskilled and Unaware of it" prove that incompetents have a higher self-appraisal than competents?

Question

Does the 1999 paper by Kruger and Dunning "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments" reliably prove that incompetents have a consistently higher self-appraisal than competents?

I'm accepting that the gap between self-appraisal and reality is greater for incompetents than for competents (the paper discusses the potential for a regression effect in their statistical analysis), and that most people think they are above average ("Illusory superiority" or the "Lake Woebegone effect"), but the paper says that more than that is going on. It describes Charles Darwin as saying

"ignorance more frequently begets confidence than does knowledge"

and this study has been cited by the general public:

@gilles that's kind of intentional; we basically don't want these users. en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect – Jeff Atwood♦ May 15 at 20:03

(As far as I can tell, Atwood was implying that incompetent users are more likely to think that there's an error on the Stack Exchange servers than competent users)

Has the methodological rigour, and the relevance, of the paper been generally accepted?

Also, is their claim that incompetents have a higher self-appraisal than competents been treated by the scientific community as strongly confirmed, or is there still doubt about this claim within the mainstream?

If the claims are true, how come there wasn't a negative correlation between actual ability and perceived ability in figures 1, 3 and 4 in the 1999 paper? ^(PDF)

Figure 1

Figure 3

Figure 4

Answer 1

Since Kruger and Dunning (K&D)'s paper came out, a number of papers and articles have claimed that the statistical analysis method used by K&D is fatally flawed. (See the bibliography below.) As far as I can tell, the critics are correct, and the data in the K&D paper provides no support for the existence of the effect that is now named after it. A recent (2020) study by Gignaca and Zajenkowski, using a different analysis method, concluded that the effect "is (mostly) a statistical artefact".

(I should clarify that there does seem to be evidence in K&D's paper that the subjects tended to overestimate their skill level relative to their peers. What is unsupported is the claim that there is a correlation between skill level and the tendency to overestimate, with the least skilled subjects overestimating the most. I think the latter is what is normally called the "Dunning-Kruger effect".)

Ackerman et al (2002) demonstrated the problem with K&D's analysis by applying it to simulated data that, by construction, contains no Dunning-Kruger effect. The result is a graph similar to those that appear in K&D:

The authors wrote:

As can be seen in Fig. 1, the plotting of simulated data for 500 subjects resulted in exactly the same phenomenon reported by Kruger and Dunning (1999)—an overestimation for those in the lowest quartile and an underestimation for those in the top quartile. Further analysis comparing the means of self-report and objective knowledge for each quartile revealed that the difference between the simulated self-reported (M = −0.21) and objective (M = −1.22) scores for the bottom quartile was significant t (124) = −10.09, P < 0.001 (which would be “interpreted” as overestimation of performance). [and likewise for the top quartile]

It may be worth pointing out that their synthetic data was symmetric in the two variables, so binning it by "self-reported" instead of "objective" skill would have led to a similar plot, from which one might conclude that there is a D-K-like effect in the opposite direction (i.e., subjects who believe they are least competent underestimate their competence the most), with the same apparent statistical significance.

The reason this happens is a form of regression toward the mean. The top quartile of test results includes not only subjects in the top quartile of skill (granting for the sake of argument that such a thing exists), but also less skilled subjects who got lucky. The bottom quartile includes subjects who got unlucky or were just having a bad day. This mixing of skill levels tends to pull the average of the other plotted variable toward the overall mean.

K&D actually mention the problem, saying:

Despite the inevitability of the regression effect, we believe that the overestimation we observed was more psychological than artifactual. For one, if regression alone were to blame for our results, then the magnitude of miscalibration among the bottom quartile would be comparable with that of the top quartile.

As Ackerman et al point out, this is wrong. If a constant positive bias is added to all "self-reported" scores in Ackerman et al's synthetic data, the effect is to shift the "self-reported" curve upward relative to the "objective" curve without changing the shape of either, with a result nearly identical to K&D's figure 1, even though the introduced bias is independent of true or measured ability.

(As I said, there is evidence in K&D's data for an overall upward bias in self-assessed ability, but K&D claim throughout the paper that the bias is correlated with ability, and I think "the overestimation we observed" is meant to include that.)

As the variance in the true skill of the subjects decreases, the real D-K effect should decrease, while the D-K-mimicking artifact of the analysis increases, since noise introduced by the testing process will cause more mixing of the quartiles when there is less real difference between them. K&D's subjects were all undergraduates from a single university, and in at least 3 of the 4 studies were from a single department (psychology). Since K&D repeatedly referred to the lowest quartile of subjects as "incompetent", it's worth adding that the school in question is Cornell, which is generally ranked among the top 10-20 universities in the world. According to Wikipedia (just linked), the 25th-percentile SAT score of Cornell admittees is 1450, which according to the College Board is the 96th percentile of all SAT takers and the 99th percentile of U.S. 11th- and 12th-graders. I don't know how the psychology department compares to that university-wide average. Still, I can't help thinking that there is probably a large gulf between the "incompetents" in this study and what the average layperson who has heard of the D-K effect imagines when they hear "incompetent". Even if something like the D-K effect is real, I wouldn't expect to see it in this rather narrow slice of the spectrum of academic achievement.

---

Rex Kerr's answer mentions another study by Burson et al which replicates the findings of K&D. The replication uses the same analysis method, and is invalidated by the same argument.

I find it odd that neither K&D nor Burson et al presented their data in what would seem to be the obvious way: a scatterplot. I have a hypothesis that would explain that: their scatterplots showed no obvious effect, so they left them out of the paper, and went with another analysis that seemed to show the effect that they were expecting to find. It's not a very charitable hypothesis, but it seems plausible.

---

Papers critical of K&D:

P. L. Ackerman, M. E. Beier, and K. R. Bowen, "What we really know about our abilities and our knowledge." Personality and Individual Differences 33 (2002) 587–605. DOI, PDF

J. Krueger and R. Mueller, "Unskilled, unaware, or both? The better-than-average heuristic and statistical regression predict errors in estimates of own performance." Journal of Personality and Social Psychology 2002 Feb;82(2):189-92. DOI, PDF

E. Nuhfer et al, "Random Number Simulations Reveal How Random Noise Affects the Measurements and Graphical Portrayals of Self-Assessed Competency." Numeracy 9, Iss. 1 (2016). DOI, PDF

G. Gignaca and M. Zajenkowski, "The Dunning-Kruger effect is (mostly) a statistical artefact: Valid approaches to testing the hypothesis with individual differences data." Intelligence 80, May–June 2020, 101449. DOI, PDF

Articles/blog posts critical of K&D:

T. Yarkoni, "What the Dunning-Kruger effect is and isn’t" (personal blog, 2010)

A. Danvers, "Dunning-Kruger isn't real: The least knowledgeable people are not the most overconfident" (psychologytoday.com, 2020)

J. Jarry, "The Dunning-Kruger Effect Is Probably Not Real" (McGill University Office for Science and Society, 2020)

Both of these lists are probably very incomplete.

Answer 2

I think the Wikipedia article covers the support for this phenomenon adequately in its "supporting studies" section. There's no reason to doubt that it's a real effect given the number of confirmations.

As an example, Burson, Larrick, and Klayman (2006) set out to test the effect, asking whether people were judging task difficulty instead of their own ability. They performed similar tests but with varying difficulties and reproduced the original results, i.e., that people are poor at estimating their relative ability (Figure 4):

The gold standard of rigor in sciences is ability to replicate findings (that's the point of being rigorous!), so in this regard the paper passes: even if there were some methodological problems (it's often really hard to tell), the results seem sound.

With regard to confidence vs. ability, the Burson paper shows conditions (such as the "Hard Trivia" line above where the best performers have the most inaccurate estimate of confidence. This argues against Kruger & Dunning's hypothesis that higher ability is (necessarily) linked to more accurate judgments of self-ability.

Finally, you ask whether that the authors' claim that "incompetents have a higher self-appraisal than competents" is tested and accepted. That claim is tested and refuted by the authors, by Burson et. al, and others: correlations with performance are typically positive or non-existent, not negative. But the authors do not make the claim (quote from Darwin notwithstanding)! Here are a selection of quotes from Kruger and Dunning about what they do claim:

Prediction 1 Incompetent individuals, compared with their more competent peers, with dramatically overestimate their ability and performance compared to objective criteria. (p. 1112)

Emphasis mine: the claim is that they overestimate the most, not that their estimates are even higher than the estimates of high-performers!

To be sure, they [the poor-performers] had an inkling that they were not as talented in this domain as were participants in the top quartile, as evidenced by the significant correlation between perceived and actual ability.

Again, they agree that poor performers do not give estimates that outstrip the estimates of top performers.

Bottom-quartile participants were less successful than were top-quartile participants in the metacognitive tasks of discerning what one has answered correctly vs. incorrectly

Language like the above is seen throughout: not a statement that bottom-quartile estimates themselves higher than top-quartile estimates themselves, but that bottom-quartile has got it really badly wrong.

Other studies (e.g. Burson et al.) use similar language and show similar effects (though there is not solid agreement on the causes of the effects).

So that claim isn't made by the authors (at least in this paper), and if someone in the general public were to make such a claim they should be aware that it has been (weakly) refuted by (at least some) studies of these effects.