Is the Bechdel Test an accurate critique of the film industry?

Question

The Bechdel Test is named for the author of a comic that popularized the following three "rules" that must be fulfilled before a movie passes the test:

1. The film must have at least two women in it

2. The two women must talk to each other during the film

3. At least one of these discussions must be about something besides a man

The premise of the comic (and the proponents of the rule) is that a striking majority of films fail this test. The implication is that the film industry has a strong gender bias toward male characters or male-centric plots such that the women are only really there to talk about the men.

The relevant questions are such:

• Do a strong majority of modern films regularly fail this test? (Say, at least 75%.)

• Does flipping the gender of the test result a drastically different outcome? (Say, a difference of 33% or more.)

These percentages are arbitrary, but they provide a starting point. A clarification of the last question: If the female Bechdel test fails 75% but the male Bechdel test fails 25% of the time, the difference would be 50%.

To help restrict the data set if a larger one is not practical, feel free to concentrate on extremely popular or critically acclaimed films from the previous decade (2000-2010). This, again, only serves to provide a starting point. (An alternative would be the decade surrounding the comic's printing: 1980-1990.)

Answer 1

According to this site, which uses a community effort to rank movies, 50% of all movies pass the test, and only 10% fail on all three points. So no a strong majority does not fail the test.

However the validity of the measure at all is questioned, for instance tvtropes

• a fair number of top-notch works have legitimate reasons for including no women (e.g. ones set in a men's prison or on a WorldWarTwo military submarine or back when only men were on juries or with no conversations at all, or with only one character).
• A movie can easily pass the Bechdel Test and still be incredibly misogynistic.
• it's also possible for a story to fail the test and still be strongly feminist in other ways

It would appear that the only thing the Bechdel test is good at measuring is whether or not something passes the Bechdel test. If it misidentifies mysogenist movies as feminist and vice-versa then it cannot be said to be a valid measure or critique of the industry.

Answer 2

I am answering the second point of the question. The first point has already been answered.

Does flipping the gender of the test result a drastically different outcome? (Say, a difference of 33% or more.)

This Reverse Bechdel Test (I'll denote RBT) is surprisingly difficult to find information about. Many people post their own independently result for a particular work online, but common problems (in addition to scientific controls on objectivity and consistency) include a lack of a meaningful sample size, and a lack of any attempt to apply the RBT to a meaningfully random sample. I only found three references that applied to RBT to 10 or more titles. Here they are.

comicbookGRRRL blog

http://www.comicbookgrrrl.com/2012/05/30/women-in-comics-dc-vs-marvel-the-bechdel-test/

This blogger applied the BT and the RBT individually to 6 episodes of 19 comic book series. This falls short of what we wanted because these are comics, not movies, but it satisfies other criteria. Mainly, it has a large amount of volume, with 6x19= 144 tests in total, looking for both BT and RBT. Nothing else I've seen comes anywhere close to these numbers.

Selection:

Given that the Bechdel Test requires reading each comic thoroughly, I have had to limit this test to my own 'big publisher' reading list (along with some borrowed from generous friends) - 8 DC titles, 8 Marvel titles, alongside Walking Dead, Chew and Buffy the Vampire Slayer. So please consider this a Bechdel Test on a sample that this woman in particular likes to read (the DC picks are my regulars, alogng with the Best of the Rest titles at the bottom).

I have sampled the most recent 6 issues of each title (issue numbers detailed below) that I have to hand, all of them as recent as possible. If anyone would like me to expand my sample, I'm happy for someone else to buy them!

Note that "titles" do not correlate directly with sales, but a similar criticism can be waged for any Bechdel Test.

Reporting:

The blogger reported results in batches of 6, one for each series. In each batch, the number of BT and RBT passes are reported. Now, this is a problem if you want cross-correlations. For instance, you can't definitively put a number on the number of episodes that failed both the BT and RBT. Nonetheless, we don't need that information anyway. Instead, I summed up all the BT and RBT passes, and I report them here:

• BT passes: 66
• RBT passes: 91
• total tests: 114

This translates to a pass rate of 57.9% for BT and 80.0% for RBT. With a significant amount of data, this supports the hypothesis that comic books are not gender-symmetric because BT passes are lower than RBT passes. The most controversial outcome is that it suggests that the RBT fails are not negligible, at 20%. One disconnect between this data and the claims we are debunking is that this is for comic books and the claims are for movies. Without evidence to the contrary, we must admit the possibility that comic books could be less male-centric than movies, or that just her selection of comic books are more female-centric. After all, she is female.

bechdeltest.com tells us that 53% of 3,300 movies passed BT by their method, which is about 5% lower than comicbookGRRRL. From her data, I calculate a standard deviation in counts of BT passes for a series (out of 6) to be 1.93. That allows me to calculate a predicted error in the mean of BT passes, which is the standard deviation divided by the number of trials. I find the error in the mean to be 5.2%, which implies that comicbookGRRRL's BT pass rate can be reported as 57.9% +/- 3.0%, which means that it's within 2 standard deviations of the rate from bechdeltest.com, which means that the statistics alone do not show a meaningful difference in the gender representation in major motion picture movies and this sample selection of comic books.

Reverse Bechdel blog

http://reversebechdel.blogspot.com/2011/02/first-ten-statistics.html

This was an independent blogger who posted about their own selection of major motion pictures. Other than their attempt to be non-biased in the selection there seems to be no controls on how the sample was formed. This person reported individual movie results for some time through Jan-Feb 2011. They made one blog post summarizing the numbers for the first 10 tests. They subsequently conducted more, but I have not taken the time to aggregate them.

• BT passes: 9
• RBT passes: 10
• total tests: 10

I should note that their pass rate of 90% isn't even close to other sources. This shows that someone managed to find 10 titles that passed the RBT.

Mild Concern Blog

http://mildconcern.com/2012/03/08/the-bechdel-test/

This is a site that assessed 21 movies. I can't find where they mention anything about how they selected them. They don't given any information about their methods beyond the the definition of the Bechdel Test.

• BT passes: 6
• RBT passes: 21* (see note)
• total tests: 21

note: the wording of the site makes the exact number of RBT passes somewhat unclear. They state that they have yet to find a movie that fails (although one came close), which suggests that they didn't actually test all the 21 movies for RBT. Their pictogram list adds to the confusion where they list 15 movies that passed the RBT... even though the sample size was 21. I think that 15/21 = 71% RBT pass rate would misinterpreting their data. My best attempt to interpret their result is to say RBT passed 15 of 15 times.

Their BT pass rate 28.6% is very low compared to the bechdeltest.com average of 53% and outside the range of expected statistical error had the method been consistent and the sample random. While few conclusions can be drawn from these data, it at least shows 15 RBT tests with all failing.

Summary

A quick survey of internet sources found over 3,300 applications of BT to motion pictures, at least 25 applications of RBT to motion pictures, and 114 applications of both to comics. For the applications of RBT to motion pictures the combination of sample selection and conflicting rates leaves an unreliable picture, although it does substantiate the claim that movies will generally pass the RBT. Elsewhere, there are plenty of examples of RBT fails, although I can not find any data to put this into a statistical framework. It is possible that a meaningful RBT survey has yet to be done for movies. In the survey of comic books, some top contributors to RBT fails were Batgirl, Batwoman, Catwoman, and Buffy. These can account for 15 of the 23 RBT fails.

Examples of RBT fail examples in movies include Black Swan, Bridesmaids, and All About Eve, but these are just anecdotal examples, the overall prevalence of these is unclear.

http://thisrecording.com/today/2011/10/26/in-which-we-mandate-a-reverse-bechdel-test.html

Everything found did agree with the hypothesis that BT pass rates are much lower than RBT pass rates, indicating that movies are not gender-symmetric.

To answer the question

Without a reverse test baseline to compare against, the Bechdel Test falls short of even making the point it was intended to make because it is not statistically meaningful.