This remains a controversial issue but the evidence doesn't appear to be strong enough to conclude there is a significant decline
NB I'm adding this answer because of two contradictory papers I stumbled across in recent months that illustrate just how divided the actual evidence base is and also how the stories that hit the headlines are not an unbiased selection of the ones that get published in the scientific literature.
The first recent paper was widely reported in the media. The BBC report summarises thus:
The sperm count of French men fell by a third between 1989 and 2005, a study suggests.
and quotes one of the paper's authors:
To our knowledge, this is the first study concluding a severe and general decrease in sperm concentration and morphology at the scale of a whole country over a substantial period.
This constitutes a serious public health warning.
The paper itself studied reported sperm counts in male partners of couples attending fertility clinics. The selection was refined to the subset where the woman was shown to be infertile due to blocked or missing fallopian tubes and was therefore assumed to be a fair sample of the male population (there is no reason to expect confounding by male fertility status).
The second paper was published in BMJ open and has received no obvious publicity. It concludes (my emphasises):
This large prospective study of semen quality among young men of the general population showed an increasing trend in sperm concentration and total sperm count. However, only one in four men had optimal semen quality. In addition, one in four will most likely face a prolonged waiting time to pregnancy if they in the future want to father a child and another 15% are at risk of the need of fertility treatment. Thus, reduced semen quality seems so frequent that it may impair the fertility rates and further increase the demand for assisted reproduction.
To put it another way, sperm counts are increasing slowly but there are a lot of people with poor quality semen.
One of the possible reason why the studies differ is that the French study has not accounted for possible confounding factors. If the nature of the sample has changed over time (eg because those seeking fertility treatment represent a different mix of people by socioeconomic class over the long time period of the study) then this will not be a good random sample from the same population. The Danish study is more likely to be a random sample so may be better.
The Danish study also showed that there are a lot of people with poor sperm counts or quality. If people are now more concerned with fertility than they used to be, that would explain many other (less careful) studies showing declines: the results reflect not actual declines in the same population but increasing concern among those with poor fertility. This is speculative, but possible.
The general lesson is that in statistical studies where the underlying variable is known to show wide variability among individuals, you have to be really careful to avoid confounding factors. And, not enough studies achieve this.
It is also worth noting that the studies you hear about are far more likely to be the worrying ones, not the ones proclaiming that there isn't a problem (they don't make such good headlines).