Several sources claim that eating soy has an effect on estrogen levels (for example, Livestrong, American Cancer Society and sfgate).
Some sources go further and say eating soy affects men's sex-drives:
Soy foods lower your sex drive (Japanese housewives would feed more soy to their husbands to reduce their virility when they feared infidelity or pregnancy) Soy’s has also been proven to lead to infertility in males.
Source: HealthLevelUp
Does soy affect sex drive (via estrogen levels or otherwise)?