Summary: The claimed paper is a real scientific study that follows the norms and standards of science. In a direct response to this article, a peer scientist criticizes the methods and conclusions of the claimed paper. I am not qualified to sort out which of them is correct, but the criticisms seem valid to me.
This is definitely not a settled scientific question.
Background on the claim
The paper in the claim, which I will refer to as the claim paper, finds that homosexuals score higher on "Eysenck’s orthogonal personality scales Neuroticism and Psychoticism." The claiming paper cites scientific articles that find that these measures are correlated with depression, anxiety, schizophrenia and other psychotic disorders, antisocial behavior and conduct problems, and suicidal behavior.
The paper goes on to conclude that there may be a genetic basis that causes both neuroticism and psycopathy as well as homosexuality.
The claiming paper does not look at mental illness directly, nor does it sequence anyone's genes or look at genetic markers.
This followup paper by the same authors, makes a much more clear claim about a genetic cause.
Do the results of this study match the results of other studies in the scientific literature? To answer this, I began by reading the articles reviewed in this article's introduction.
A. This meta-analysis read 25 different studies on homosexuality and mental health. The 25 different studies collectively found that homosexuals were more at risk for depression, anxiety, and suicide attempts as well as drug and alcohol abuse.
B. This study finds that homosexuals reported "more childhood psychological and physical abuse by parents or caretakers, more childhood sexual abuse, more partner psychological and physical victimization in adulthood, and more sexual assault experiences in adulthood. Sexual orientation differences in sexual victimization were greater among men than among women." These things have been linked to a greater severity of mental illness, and are not genetic as the claim paper believes.
C. This section from the intro to the claim paper summarizes findings from other studies have found similar results to the claim paper.
Bozkurt et al. (2006) found in a sample of Turkish men that homosexuals scored significantly higher on Psychoticism (as well as nonsignificantly higher on Neuroticism) than a matched heterosexual control group and Eisinger et al.
(1972) found that homosexual women scored significantly higher than Neuroticism norms. In a small study, however, Wilson (1982) found that homosexual women scored lower than heterosexual women.
Literature Summary: Many other papers find a link between homosexuality and mental illness, although most of these attribute this to environmental factors, like abuse or victimization. A few other papers find a link between homosexuality and neuroticism/psycopathy, but I did not see a claim that there is a genetic causal claim. The claim paper seems to be the first to say that.
I skimmed a number of the articles that cite the claim paper articles and found that most of them cited the claim paper indifferently. They repeated the results but do not editorialize about this study's quality. Then I found this open letter that criticizes the methods and conclusions of this paper. They list three concrete objections. The easiest to understand, and most directly related to the claimed study is quoted below. (Emphasis mine)
Zeitsch et al. (2011) defined participants as non-heterosexuals if they described their sexual feelings as anything other than “exclusive attraction to the opposite sex.” The majority (59%) of non-heterosexuals in this sample scored a 1 on the Kinsey Scale, describing themselves as “predominantly heterosexual, only incidentally homosexual.” The study found that people who reported exclusive heterosexuality were less likely to be neurotic than those who reported other values (although the relationship was markedly non-linear: the lowest average neuroticism of all groups was among people who reported a Kinsey score of 4). The study then reported an r of 0.25 between the inferred (via a bivariate Cholesky decomposition) genetic predictors of sexual orientation and genetic determinants of neuroticism. In their second study,Zeitsch et al. (2012) found evidence of shared genetic causes of depression and non-heterosexuality. However, this study also found that depression and non-heterosexuality shared environmental causes, namely, sexual abuse in childhood and risky childhood family environments. These findings would appear to support, rather than refute, our own conclusions.
This is an accurate description of the methods of the claim paper.
We operationally defined those with any degree of sexual attraction to the same sex (Kinsey scores 1–6) as nonheterosexuals, and the associated trait as nonheterosexuality.
I personally have a hard time with any scientific study that considers completely gay men to be the same as men who have occasional gay thoughts, but are other wise straight. The fact that this represents most of the men in the study seems pretty damning to me.
