# AI can guess whether you're gay or straight – how do they get 91% success rate?

This article claims there is a software, that can say if a person is homosexual, based only on their facial images. The other article calls it "pseudoscience" though:

critics say we’re revisiting pseudoscience

Is it true or fake? How did they measure 91%? If 7-11% of people are homosexual, simply by saying "straight" all the time the software would have 89%-93% rate of success.

• I won't be surprised if statistically speaking gay and straight people use different kinds of photos on dating websites. It probably won't work with standardized mugshots. Sep 24, 2017 at 9:44
• @JonathanReez They demonstrated that it was in fact the facial features, not the 'kind' of photo, that the algorithm relied upon for the classification. Sep 24, 2017 at 12:49
• I ran my face through their system - their system is a darned liar! jk. Artifical Neural Networks are fascinating. Could they could be used to detect other hidden 'traits', such as terrorists, spies, nationality, religion? Who knows what small differences humans can't observe. Sep 25, 2017 at 17:46
• Along the lines of what @GordonM said - if there is about a ten percent prevalence of homosexuality in society, and I program an AI to just always say "straight," then it should have about that success rate. Polling suggests that the rate is actually lower than former estimates of near 10% (one Gallup poll puts it at about 4%), so the AI performs worse than not bothering to guess. Sep 26, 2017 at 17:07
• This is a problem about understanding statistics, the paper never says 91% accuracy so I wish the comments and article didn't either. "The AUC = .91 does not imply that 91% of gay men in a given population can be identified, or that the classification results are correct 91% of the time. The performance of the classifier depends on the desired trade-off..." Sep 28, 2017 at 13:49

## 2 Answers

Is it true or fake?

The research paper in question has passed peer review, suggesting that the methodology was deemed to be sound by other experts in the field. It is conceivable that the results were faked (as occasionally happens in research) but this is very unlikely given how easy it would be to reproduce and verify the findings. In fact, there's nothing stopping you from reproducing their work yourself.

Could the results be wrong, though? Well, the authors of the study themselves wrote the following:

Our findings could be wrong. In fact, despite evidence to the contrary, we hope that we are wrong. However, scientific findings can only be debunked by scientific data and replication, not by well-meaning lawyers and communication officers lacking scientific training.

in a response to critiques from HRC and GLAAS (source).

How did they measure 91%? If 7-11% of people are homosexual, simply by saying "straight" all the time the software would have 89%-93% rate of success.

As explained in the paper, the algorithm was presented with an equal number of photos of heterosexual and homosexual people. In other words, the a priori probability of the person in each image being homosexual was 50%, not the 7-11% of people you pass on the street.

• You could add also that the 91% is an ROC AUC, I totally don't follow the papers statistics but i never liked stats... "in other words, the classifier provided for a nearly seven-fold improvement in precision over a random draw(47/7 = 6.71)." Sep 24, 2017 at 13:37
• "As explained in the paper, the algorithm was presented with an equal number of photos of heterosexual and homosexual people" - but the method still matters. Imagine you have a bunch of pictures and you need to divide them in two groups A and B. Some are definitely look like A, and some are more enigmatic. Let's say you found about 30% obvious ones and you're sure they have type A. Now you know that only 20% "unidentified" A remains. So you say "it is B" for other 70% and magically "guess" all the B pictures with the total success of 70%. Sep 25, 2017 at 12:30
• The NN, once trained, is essentially just a mathematical function which could be reported, just like any other correlation. Dunno if they've chosen to report the NN's result, but if so, replication studies wouldn't need to mess with AI stuff.
– Nat
Sep 25, 2017 at 13:04
• @enkryptor: Rather than speculate that they may have used a biased method, read the paper, and tell us how they actually did it. Sep 26, 2017 at 9:37
• One thing that nobody seems to have mentioned is that all the AI actually needs to decide is if person A seems more gay than person B. They already know that one person is gay, so it's all about relative differences. Thus, such a system might not hold up at actually deciding if any random person is gay (which seems to be a main fear among those interpreting these results).
– Kat
Sep 29, 2017 at 18:46

Summary: This paper is a real scientific paper that has passed peer review. The real judge of whether their findings are true or not is whether they can be replicated. There simply has not been enough time since the study came out for a replication to be published. Additionally there are questions about what exactly their results mean. Artifical Neural Networks do not explain to the user why they made their predictions; they are a black box.

Artificial neural networks (ANNs) are a machine learning tool that can be trained to recognize patterns in data and make predictions based on those patterns. Their structure was inspired by the structure of natural neural networks, brains. Like a natural neural network, their decision making processes are a black box. ANNs make a prediction (whether the individual is gay or straight) based on input data (a set of photographs), but they do not explain their reasoning. The researchers speculate on what the ANN might have been picking up on, but they don't really know.

What facial features were employed by the algorithm to detect sexual orientation? The average faces most likely to belong to gay men (see Figure 1) were more feminine, while the faces most likely to belong to lesbians were more masculine. Typically, men have larger jaws, shorter noses, and smaller foreheads. Gay men, however, tended to have narrower jaws, longer noses, larger foreheads, and less facial hair. Conversely, lesbians tended to have more masculine faces (larger jaws and smaller foreheads) than heterosexual women.

The gender atypicality of gay faces extended beyond morphology. Lesbians tended to use less eye makeup, had darker hair, and wore less revealing clothes (note the higher neckline)—indicating less feminine grooming and style. Furthermore, although women tend to smile more in general, lesbians smiled less than their heterosexual counterparts.

Additionally, consistent with the association between baseball caps and masculinity in American culture, heterosexual men and lesbians tended to wear baseball caps (see the shadow on their foreheads in Figure 1; this was also confirmed by a manual inspection of individual images).

Maybe the ANN was mostly guided by facial features, and maybe by other cues like smiling or wearing baseball caps. The training set was taken from a dating website. I expect gay men and lesbians to choose photos that would be attractive to other gay men and lesbians. The ANN could be picking up on those differences. In response to these questions the authors argue that:

First, we tested our classifier on an external sample of Facebook photos. It achieved comparable accuracy as on the dating website sample, suggesting that the images from the dating website were not more revealing than Facebook profile pictures. ...

Finally, the deep neural network used here was specifically trained to focus on fixed facial features that cannot be easily altered, such as the shape of facial elements. This helped in reducing the risk of the classifier discovering some superficial and not face-related differences between facial images of gay and straight people used in this study.

Note: Although they try to eliminate any dependence on things other than fixed facial features, I am not sure if they succeeded. I am not deeply familiar with facial recognition algorithms or neural networks.

The ANN in the paper used a training set with half gay and half straight faces. The ANN would be presented with a single gay face and a single straight face, and asked which was which. When the faces were made using 5 photographs, it was able to correctly choose which one was gay 91% of the time. This is quite different from any real life application, where roughly 10% of the population is gay.

Speculation: The highest accuracy prediction, 91% accuracy, was achieved with 5 photos of the same person. However, there were only 219 gay men and 223 lesbians with 5 photos in their data set, compared to ~3500 each for 1 photo. This could be a weakness in their methods, but 200 subjects is still quite a few. It is possible that their methods overstate their accuracy. The predictions based on 4 photos had significantly more subjects and only slightly lower accuracy. If I arbitrarily decided to throw out the multiple photo accuracy, we are still left with an 81% accuracy for gay men.

If you are reading this in the future, this google scholar link shows all of the articles that cite the article in question. Hopefully in a couple of years, there will be quite a list of articles at that link. The articles that will be published will hopefully be a stronger form of review than peer review. I would expect that many of them repeat the conclusions of this article briefly and uncritically. The authors of these papers probably did not read this paper deeply enough to give a real critique of its methods. If one of them discusses it at length, those authors probably understood the methods and conclusions well enough to be properly skeptical. If that paper also is on a very similar topic, or builds on this work, I would be very interested in that authors opinion of the paper in the question.