There have been a few studies conducted on this. The summary is that meat protein contributes more to Total Testosterone than a vegetarian diet.
This link quotes from a book by Lou Schuler - "The Testosterone Advantage Plan"
Rodale: USA, 2002.
1985 study found when it looked at a large cross-section of omnivores
and vegetarians ... The meat-eaters actually had 36 percent more T
than the guys who stuck to rabbit food
A 1989 study found pretty much the same thing: The meat-eaters ate
more fat, more cholesterol, more saturated fat, and less fiber than
the vegetarians and had 31 percent more testosterone (Schuler, p.86).
A Dutch study published in 1992 looked at changes in T levels on these
two diets. A group of young male endurance athletes ate and trained on
each diet for 6 weeks. (Half started on the meat-rich diet, half on
the vegetarian diet; then they switched.) Total testosterone declined
35 percent when the athletes used the vegetarian diet (Schuler, p.76).
From the same link
Men's Health magazine declared, "Perhaps the ideal diet to drop your
testosterone is high fiber and vegetarian-and the ideal way to raise
it is the red-meat approach" (Jim Thorton, p.154). " Maximum
Testosterone." Men's Health. April, 2005, pp. 146-155,182
An abstract of the 1992 Dutch study is available here
Sample size of 8 male endurance athletes, in summary the Total Testosterone is less on a Vegetarian diet compared to Meat diet, but Free Testosterone does not differ significantly.
Serum sex hormones and endurance performance after a lacto-ovo
vegetarian and a mixed diet.
Raben A, Kiens B, Richter EA, Rasmussen LB, Svenstrup B, Micic S,
Bennett P. SourceAugust Krogh Institute, University of Copenhagen,
The V diet resulted in a lower total T level (13.7, 9.8-32.4 nmol.l-1)
(median and range) compared with the M diet (17.4, 11.8-33.5
nmol.l-1). During exercise after 6 wk on the diets total T was also
significantly lower on the V than on the M diet
Given that most of the advice on this concentrates on eating red meat, your mileage may vary with other meats and fish protein.