Since Kruger and Dunning (K&D)'s paper came out, a number of papers and articles have claimed that the statistical analysis method used by K&D is fatally flawed. (See the bibliography below.) As far as I can tell, the critics are correct, and the data in the K&D paper provides no support for the existence of the effect that is now named after it. A recent (2020) study by Gignaca and Zajenkowski, using a different analysis method, concluded that the effect "is (mostly) a statistical artefact".
(I should clarify that there does seem to be evidence in K&D's paper that the subjects tended to overestimate their skill level relative to their peers. What is unsupported is the claim that there is a correlation between skill level and the tendency to overestimate, with the least skilled subjects overestimating the most. I think the latter is what is normally called the "Dunning-Kruger effect".)
Ackerman et al (2002) demonstrated the problem with K&D's analysis by applying it to simulated data that, by construction, contains no Dunning-Kruger effect. The result is a graph similar to those that appear in K&D:
The authors wrote:
As can be seen in Fig. 1, the plotting of simulated data for 500 subjects resulted in exactly the same phenomenon reported by Kruger and Dunning (1999)—an overestimation for those in the lowest quartile and an underestimation for those in the top quartile. Further analysis comparing the means of self-report and objective knowledge for each quartile revealed that the difference between the simulated self-reported (M = −0.21) and objective (M = −1.22) scores for the bottom quartile was significant t (124) = −10.09, P < 0.001 (which would be “interpreted” as overestimation of performance). [and likewise for the top quartile]
It may be worth pointing out that their synthetic data was symmetric in the two variables, so binning it by "self-reported" instead of "objective" skill would have led to a similar plot, from which one might conclude that there is a D-K-like effect in the opposite direction (i.e., subjects who believe they are least competent underestimate their competence the most), with the same apparent statistical significance.
The reason this happens is a form of regression toward the mean. The top quartile of test results includes not only subjects in the top quartile of skill (granting for the sake of argument that such a thing exists), but also less skilled subjects who got lucky. The bottom quartile includes subjects who got unlucky or were just having a bad day. This mixing of skill levels tends to pull the average of the other plotted variable toward the overall mean.
K&D actually mention the problem, saying:
Despite the inevitability of the regression effect, we believe that the overestimation we observed was more psychological than artifactual. For one, if regression alone were to blame for our results, then the magnitude of miscalibration among the bottom quartile would be comparable with that of the top quartile.
As Ackerman et al point out, this is wrong. If a constant positive bias is added to all "self-reported" scores in Ackerman et al's synthetic data, the effect is to shift the "self-reported" curve upward relative to the "objective" curve without changing the shape of either, with a result nearly identical to K&D's figure 1, even though the introduced bias is independent of true or measured ability.
(As I said, there is evidence in K&D's data for an overall upward bias in self-assessed ability, but K&D claim throughout the paper that the bias is correlated with ability, and I think "the overestimation we observed" is meant to include that.)
As the variance in the true skill of the subjects decreases, the real D-K effect should decrease, while the D-K-mimicking artifact of the analysis increases, since noise introduced by the testing process will cause more mixing of the quartiles when there is less real difference between them. K&D's subjects were all undergraduates from a single university, and in at least 3 of the 4 studies were from a single department (psychology). Since K&D repeatedly referred to the lowest quartile of subjects as "incompetent", it's worth adding that the school in question is Cornell, which is generally ranked among the top 10-20 universities in the world. According to Wikipedia (just linked), the 25th-percentile SAT score of Cornell admittees is 1450, which according to the College Board is the 96th percentile of all SAT takers and the 99th percentile of U.S. 11th- and 12th-graders. I don't know how the psychology department compares to that university-wide average. Still, I can't help thinking that there is probably a large gulf between the "incompetents" in this study and what the average layperson who has heard of the D-K effect imagines when they hear "incompetent". Even if something like the D-K effect is real, I wouldn't expect to see it in this rather narrow slice of the spectrum of academic achievement.
Rex Kerr's answer mentions another study by Burson et al which replicates the findings of K&D. The replication uses the same analysis method, and is invalidated by the same argument.
I find it odd that neither K&D nor Burson et al presented their data in what would seem to be the obvious way: a scatterplot. I have a hypothesis that would explain that: their scatterplots showed no obvious effect, so they left them out of the paper, and went with another analysis that seemed to show the effect that they were expecting to find. It's not a very charitable hypothesis, but it seems plausible.
Papers critical of K&D:
P. L. Ackerman, M. E. Beier, and K. R. Bowen, "What we really know about our abilities and our knowledge." Personality and Individual Differences 33 (2002) 587–605. DOI, PDF
J. Krueger and R. Mueller, "Unskilled, unaware, or both? The better-than-average heuristic and statistical regression predict errors in estimates of own performance." Journal of Personality and Social Psychology 2002 Feb;82(2):189-92. DOI, PDF
E. Nuhfer et al, "Random Number Simulations Reveal How Random Noise Affects the Measurements and Graphical Portrayals of Self-Assessed Competency." Numeracy 9, Iss. 1 (2016). DOI, PDF
G. Gignaca and M. Zajenkowski, "The Dunning-Kruger effect is (mostly) a statistical artefact: Valid approaches to testing the hypothesis with individual differences data." Intelligence 80, May–June 2020, 101449. DOI, PDF
Articles/blog posts critical of K&D:
T. Yarkoni, "What the Dunning-Kruger effect is and isn’t" (personal blog, 2010)
A. Danvers, "Dunning-Kruger isn't real: The least knowledgeable people are not the most overconfident" (psychologytoday.com, 2020)
J. Jarry, "The Dunning-Kruger Effect Is Probably Not Real" (McGill University Office for Science and Society, 2020)
Both of these lists are probably very incomplete.