There is a study by Ryan et al about this topic. They also discuss the results of the study in this post at Rochester as well as this NYT article:
Using this methodology we identified a subgroup of participants who, despite self-identifying as highly straight, indicated some level of same-sex attraction (that is, they associated “me” with gay-related words and pictures faster than they associated “me” with straight-related words and pictures). Over 20 percent of self-described highly straight individuals showed this discrepancy.
Notably, these “discrepant” individuals were also significantly more likely than other participants to favor anti-gay policies; to be willing to assign significantly harsher punishments to perpetrators of petty crimes if they were presumed to be homosexual; and to express greater implicit hostility toward gay subjects (also measured with the help of subliminal priming). Thus our research suggests that some who oppose homosexuality do tacitly harbor same-sex attraction.
It’s important to stress the obvious: Not all those who campaign against gay men and lesbians secretly feel same-sex attractions. But at least some who oppose homosexuality are likely to be individuals struggling against parts of themselves, having themselves been victims of oppression and lack of acceptance. Ryan et al, 2012. Parental autonomy support and discrepancies between implicit and explicit sexual identities: Dynamics of self-acceptance and defense.
Specifically, the study found:
Implicit sexual orientation and explicit sexual orientation interacted in predicting participants’ self-reported homophobia, accounting for 7% of its variance
This Slate article suggests that the used IAT may not be reliable, and that the conclusion that is drawn is not the only possible one. The "me" and "gay" pairings could result in anxiety in homophobic men - not because they have homosexual feelings, but because of a fear of being labeled as gay; anxiety has been shown to reduce reaction time.
This study by Adams et al concludes:
Both groups exhibited increases in penile circumference to the heterosexual and female homosexual videos. Only the homophobic men showed an increase in penile erection to male homosexual stimuli. Adams et al, 1996. Is homophobia associated with homosexual arousal?
The exact finding by the numbers:
In the homophobic group, 20% showed no significant tumescence [in response to a male homosexual video], 26% showed moderate tumescence, and 54% showed definite tumescence to the homosexual video; the corresponding percentages in the nonhomophobic group were 66%, 10%, and 24%, respectively.
The paper also notes that this does not necessarily mean that these men are homosexual or bisexual. The reaction could result from negative feelings such as anxiety, which has in the past been shown to cause arousal on its own.
Cheval et al found:
These findings confirm the importance of considering the variability in impulsive processes to understand why some (but not all) men high in homophobia have homosexual interest. These findings reinforce the theoretical basis for elaborating a dual-process model for behaviors in the sexual context.Cheval et al, 2016. Homophobia: An Impulsive Attraction to the Same Sex? Evidence From Eye-Tracking Data in a Picture-Viewing Task.
Meier et al found:
Contrary to the idea that defensive homophobics (i.e., homophobics high in self-deception) harbor an implicit attraction for gay sex, we found that such individuals displayed evidence of a phobic-like aversion. Meier et al, 2006. A secret attraction or defensive loathing? Homophobia, defense, and implicit cognition
MacInnis & Hodson found:
Some theorists propose that homophobia stems from underlying same-sex attraction. A few studies have tested this hypothesis, yet without a clear measure of implicit sexual attraction, producing mixed results. For the first time, we test this attraction-based account of homophobia among both men and women using an implicit measure of sexual attraction. No evidence of an attraction-based account of homophobia emerged. Instead, implicit same-sex attraction was related to positive evaluations of gay men and lesbians among female participants.MacInnis & Hodson, 2013. Is Homophobia Associated with an Implicit Same-Sex Attraction?
The study is behind a paywall, but some details about the methodology and results are available in a Psychology Today article:
...the authors used an implicit association task (IAT) adapted to sexual attraction: a task in which participants have to categorize pictures as male/female and words as sexually attractive/unattractive, and the speed at which they do so should tell you something about the cognitive association between the two. I'm wary of the interpretations of IATs for a number of reasons, but I'll assume for the time being that such a test does indeed kind of measure what they hope. Participants were also asked about their explicit sexual attractions to men and women, and their attitudes towards gay/lesbian and heterosexual populations. In total, their sample represented 237 Canadian undergraduates (85 men).
...the IAT results only correlated modestly with explicit measures of sexual attraction (r = .37 for men, r = .15 for women). The correlations between those IAT measures and negative, explicit evaluations of homosexuals for men was r = -.06, and for women, r = -.24. In other words, not only were such correlations quite small, but they nominally went in the opposite direction of the repression account: as people showed more implicit attraction to the same sex, they also showed less explicit negativity.