This remains a controversial issue but the evidence doesn't appear to be strong enough to conclude there is a significant decline
NB I'm adding this answer because of two contradictory papers I stumbled across in recent months that illustrate just how divided the actual evidence base is and also how the stories that hit the headlines are not an unbiased selection of the ones that get published in the scientific literature.
The first recent paper was widely reported in the media. The BBC report summarises thus:
The sperm count of French men fell by a third between 1989 and 2005, a study suggests.
and quotes one of the paper's authors:
To our knowledge, this is the first study concluding a severe and general decrease in sperm concentration and morphology at the scale of a whole country over a substantial period.
This constitutes a serious public health warning.
The paper itself studied reported sperm counts in male partners of couples attending fertility clinics. The selection was refined to the subset where the woman was shown to be infertile due to blocked or missing fallopian tubes and was therefore assumed to be a fair sample of the male population (there is no reason to expect confounding by male fertility status).
The second paper was published in BMJ open and has received no obvious publicity. It concludes (my emphasises):
This large prospective study of semen quality among young men of the general population showed an increasing trend in sperm concentration and total sperm count. However, only one in four men had optimal semen quality. In addition, one in four will most likely face a prolonged waiting time to pregnancy if they in the future want to father a child and another 15% are at risk of the need of fertility treatment. Thus, reduced semen quality seems so frequent that it may impair the fertility rates and further increase the demand for assisted reproduction.
To put it another way, sperm counts are increasing slowly but there are a lot of people with poor quality semen.
One of the possible reason why the studies differ is that the French study has not accounted for possible confounding factors. If the nature of the sample has changed over time (eg because those seeking fertility treatment represent a different mix of people by socioeconomic class over the long time period of the study) then this will not be a good random sample from the same population. The Danish study is more likely to be a random sample so may be better.
The Danish study also showed that there are a lot of people with poor sperm counts or quality. If people are now more concerned with fertility than they used to be, that would explain many other (less careful) studies showing declines: the results reflect not actual declines in the same population but increasing concern among those with poor fertility. This is speculative, but possible.
The general lesson is that in statistical studies where the underlying variable is known to show wide variability among individuals, you have to be really careful to avoid confounding factors. And, not enough studies achieve this.
It is also worth noting that the studies you hear about are far more likely to be the worrying ones, not the ones proclaiming that there isn't a problem (they don't make such good headlines).
Since this answer was originally written, the topic has remained controversial. Apocalyptic headlines have continued to appear, many driven by an influential meta-analysis by Swann et. al..
That meta-analysis led to headlines such as:
An Alarming Decline in Sperm Quality Could Threaten the Future of the Human Race, and the Chemicals Likely Responsible Are Everywhere
But the conclusions of that study have recently been challenged. The reanalysis appears in Human Fertility here.
The New York Times summarises the results of the reanalysis (my highlighting):
Now a group of interdisciplinary researchers from Harvard and the Massachusetts Institute of Technology contend that fears of an impending Spermageddon have been vastly overstated. In a study published in May in the journal Human Fertility, they re-evaluated the 2017 review and found that it relied on flawed assumptions and failed to consider alternate explanations for the apparent decline of sperm.
In an interview, Sarah Richardson, a Harvard scholar on gender and science and the senior author on the new study, called the conclusion of the 2017 review “an astonishing and terrifying claim that, were it to be true, would justify the apocalyptic tenor of some of the writing.” Fortunately, she and her co-authors argue, there is little evidence that this is the case.
The NYT reports some alternative explanations offered in the new analysis of the original observations:
Perhaps sperm levels naturally rise and fall over time and within populations. The question has not been explored by reproductive researchers and cannot be answered easily, as global sperm counts before 1970 are largely unknown...
Sperm-counting is a tricky business and notoriously prone to human error, Dr. Pacey said. ... In a 2013 review article, he noted that as methodologies for counting had improved and been standardized since the 1980s, sperm counts had appeared to fall. In other words, it may simply be that earlier scientists were overcounting sperm.
The reanalysis notes two alternative hypotheses that are consistent with the original data but fundamentally contradict the earlier conclusions:
The relationship between endocrine disrupting chemicals, geography, and sperm count has not been established by prospective studies across populations that control for confounders and use repeat individual measures in combination with a wide variety of social and biological measures.
Sperm counts rise and fall and vary across populations and time, and within individuals, in a dynamic, non-linear and non-continuous fashion.
In short both the apocalyptic conclusions (sperm counts are in sharp decline and so is fertility) and the supposed major cause (endocrine disrupting chemicals) are very doubtful. Should we panic: probably not (certainly unless someone generates much better quality data than the 2017 study).