16

In QI Series L, Espisode 9, Stephen Fry says (source):

If there was a maths test between men and women it will be more likely that the men will do better because women are constantly told that women are worse than men at maths. Similarly, Asian people tend to do better at maths because they told that Asian people are good at it. Tests have shown that if you take a group of Asian women and tell them they are women who will be competing in a maths test against some men then the women will get around 60%. If you then take a group of Asian women and tell them that they are Asian women who will be competing in a maths test against European men, then the women will get around 80-90%.

Is this really true? Do Asian women (or anyone) really do better depending on whom they think they are competing against?

  • 4
    This is an extensively researched phenomenon in sociology. You can find further information by searching for "stereotype threat". – Kilian Foth Jan 17 '17 at 7:34
  • Beer tastes different depending on which brand you think your are drinking. – Trilarion Jan 18 '17 at 8:50
  • 1
    I wonder how many percent the European men would get in this test. – Trilarion Jan 18 '17 at 8:53
14

tl;dr; There are indications for this effect, but the research is still unconclusive. The claim as stated is probably false.

The evidence

This statement refers to the widely studied Stereotype threat effect, which is often described as the risk of conforming to stereotypes of one's social group.

In particular, it refers to the study STEREOTYPE SUSCEPTIBILITY: Identity Salience and Shifts in Quantitative Performance by Shih et al.

In this study they tested the math performance of 46 Asian-American women, divided into three groups. The first group (n=14) was primed with questions about their female identity, the second (n=16) with questions about their Asian identity and the third (n=16) with neutral questions.

They found that

Participants in the Asian-identity-salient condition answered an average of 54% of the questions that they attempted correctly, participants in the control condition answered an average of 49% correctly, and participants in the female-identity-salient condition answered an average of 43% correctly.

In the second part of the study, they did the same experiment in Vancouver, where the asian stereotype is less prevalent with a total of 19 Asian-American female high school students.

They state that

In this population, the stereotype that Asians possess superior quantitative skills is not prevalent. As predicted (see Table 1), participants in the female-identity-salient condition performed the worst (28% accuracy) of the three groups. Participants in the Asian-identity-salient condition (44% accuracy) also performed worse than a control group (59% accuracy), contrast t(16) = 4.55, p < .0005, r = .75.

Note, that in this study there was no mentioning of European men. While there are indications, that a comparison against another gender/ethnicity may invoke a stereotype threat, this was not studied directly.

In Competition in stereotyped domains: Competition, social comparison, and stereotype threat by Van Loo et al. evidence for a link between competition and the stereotype threat effect was found.

Finally note, that the numbers Fry stated do not fit the numbers from the study.

The criticism

Study design

In both experiments the number of participants was very small (n=46 and n=19), which raises questions about the study's reliability.

Replication issues

In a follow-up study WHEN POSITIVE STEREOTYPES THREATEN INTELLECTUAL PERFORMANCE: The Psychological Hazards of “Model Minority” Status by Cheryan and Bodenhausen they failed to reproduce the effect (again, with a very small sample size, n=49).

In the control, personal-identity condition (n = 16), students correctly answered an average proportion of .83 of the items (SD = 0.09). In line with the possibility that positive expectations for performance could cause choking under pressure when the positively stereotyped Asian identity was salient, performance in the ethnicity condition (n = 16) was markedly lower, M = .71, SD = 0.17. In contrast, performance in the gender condition (n = 17) was comparable to performance in the control condition, M = .81, SD = 0.14.

However, they reported that the ethnicity-salient group had more problems to concentrate and conclude

The present findings should therefore certainly not be taken as evidence against the more general existence of gender-based stereotype-threat effects in the domain of math performance.

Another replication study by Gibson et al. with a higher number of participants (n=164) found a significant effect after excluding 32 participants, which were not aware of at least one of the stereotypes. Before excluding those participants, the effect was not significant. See A Replication Attempt of Stereotype Susceptibility

A third replication study with 139 participants by Moon and Roeder did not observe significant differences between the groups. See A Secondary Replication Attempt of Stereotype Susceptibility

Publication bias and validity of the stereotype threat

In the 2014 meta analysis Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis by Flore and Wicherts they confirmed the stereotype effect on girls in standardized math tests based on the available publications

Analyzing 15 years of stereotype threat literature with children or adolescents as test-takers, we found indications that girls underperform on MSSS tests due to stereotype threat. Consistent with findings by Nguyen and Ryan (2008), Picho et al. (2013), Walton and Cohen (2003), and Walton and Spencer (2009), we estimated a small effect of −0.22.

However, they also found indications for publication bias in the data

Unfortunately the robustness of the stereotype threat effect can be questioned by the presence of publication bias. All three tests based on funnel plot asymmetry—trim and fill (Duval & Tweedie, 2000), Egger's test (Sterne & Egger, 2005), and Begg and Mazumdar's rank correlation test (Begg & Mazumdar, 1994)—indicated that publication bias was present. Additionally Ioannidis and Trikalinos's (2007) exploratory test highlighted an excess of significant findings, which can be due to publication bias.

They conclude

To conclude, we estimated a small average effect of stereotype threat on the MSSS test-performance of school-aged girls; however, the studies show large variation in outcomes, and it is likely that the effect is inflated due to publication bias. This finding leads us to conclude that we should be cautious when interpreting the effects of stereotype threat on children and adolescents in the STEM realm. To be more explicit, based on the small average effect size in our meta-analysis, which is most likely inflated due to publication bias, we would not feel confident to proclaim that stereotype threat manipulations will harm mathematical performance of girls in a systematic way or lead women to stay clear from occupations in the STEM domain.

Conclusion

While there are some indications that the claim might be true, it is still unconclusive if and to what extent the specific claim about Asian females or even the underlying Stereotype threat effect exists.

Since the numbers stated by Fry do not match any results from the studies, it is likely that he invented those numbers to underline his point. Also there does not seem to be direct evidence regarding the claim that Asian women are affected by a stereotype threat due to competition with European men.

There is weak evidence that Asian women perform worse when confronted with the "female" stereotype, than when confronted with the "Asian" stereotype, but these results fail to consistently reproduce.

So given its level of precision, the claim is almost certainly fabricated. Additionally, what studies there are suggest that any effect in that direction is weaker than the claim pretends, and certain difficulties in replication imply that there might be not much, if any, such effect at all.

  • Your final statement suffers from lack of clarity. The specific claim given by the quote (with percentages) is likely false because it was not supported by the available experiments, and was probably fabricated - but it's the overprecision rather than the failed replication that makes it so. The specific question asked in the title and question body (do test results vary with expected competition) was not addressed specifically in any of your referenced experiments (and may not have been addressed specifically by any experiments ever). – Ben Barden Aug 4 '17 at 13:24
  • The murkiness in replication casts some doubt, but none of these had a sample size that could be considered particularly large. I wouldn't say that any of them provide enough support for "probably false". "Weak", perhaps. Other than that, +1 for really digging into the issue. – Ben Barden Aug 4 '17 at 13:28
  • @BenBarden - It's still the most useful answer I've received. I doubt anyone has done a rigorously controlled assessment of the precise scenario that Stephen Fry mentioned, so any reasonable assessment will need to consider evidence like this. – paj28 Aug 5 '17 at 7:56
  • @BenBarden I agree that "probably false" may be a too strong assertion. However, I would rather say there is weak evidence against the claim than in favor of it, but I am not sure how to express this. – cero Aug 6 '17 at 6:03
  • 1
    @BenBarden I made some changes, adding a reference for the link between competition and stereotype threat as well as revising the last paragraph where I incorporated your suggestion. I hope it is more clear now. Thanks for your help! – cero Aug 8 '17 at 13:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .