Despite the consensus, there are good quality studies that don't find a strong link
I think the scientific consensus is clear: almost everyone thinks secondhand smoke is bad. But skeptics should carefully consider, especially in areas where emotions run as high as this, the alternative views.
Here is a large scale study of the long term effects of secondhand smoke that finds little impact on all cause mortality. The BMJ paper is here. The conclusion:
Conclusions. The results do not support a causal
relation between environmental tobacco smoke and
tobacco related mortality, although they do not rule
out a small effect. The association between exposure
to environmental tobacco smoke and coronary heart
disease and lung cancer may be considerably weaker
than generally believed.
For the sake of balance it is worth reading the follow-up debate. My reading is that there are few direct criticisms of the science but many attacks on the scientists (who have some relationship with the "evil" tobacco industry). The intemperate nature of many responses suggest that it is becoming hard to conduct and publish any research on the topic that does not agree with the consensus. I'd recommend any skeptic to read the paper and then the responses and make their own mind up while trying to consider only criticism of the results and not their funding.
Both poor quality studies agreeing with the consensus and entirely unreferenced claims are widely repeated to justify legislation. One example is the (entirely unreferenced) claim that secondhand smoke kills 600,000 people a year. The claim comes from a WHO factsheet which does not reference any sources. Yet this is repeated as an accurate assessment even by people who should know better such as David Nutt (it is in his, mostly sensible, book *Drugs-without the hot air"). This is not a small number of deaths and, if the claim were true, we would expect strong epidemiological evidence. This is not obviously there in the literature.
Update
There has been some new debate on the subject since the original answer was written.
One author at reasons.com argues that it is becoming easier to challenge the evidence as the political significance of the position you take is now less significant since many countries now have widely accepted indoor smoking bans. The debate is summarised thus (my emphasis):
Several years ago I was talking to an epidemiologist who is skeptical of the idea that smokers pose a mortal threat to people in their vicinity. Although he supported workplace smoking bans, he was frustrated by the willingness of so many anti-tobacco activists and public health officials to overlook or minimize the weakness of the scientific case that secondhand smoke causes fatal illnesses such as lung cancer and heart disease. He wondered when it would be possible to have a calm, rational discussion of the issue, one in which skeptics would not be automatically dismissed as tools of the tobacco industry.
The opinion piece references a new recent study that doesn't suffer from many of the biases that affect many other studies. The bias is explained here:
...the study has advantages over most of the research commonly cited as evidence that secondhand smoke causes lung cancer. "To our knowledge," the authors say, "this is the first study to examine both active and passive smoking in relation to lung cancer incidence in a complete prospective cohort of US women." The prospective design avoids a weakness of studies that start with lung cancer cases and "match" them to controls. "Many studies that showed the strongest links between secondhand smoke and lung cancer were case-control studies, which can suffer from recall bias," notes the JNCI article, since "people who develop a disease that might be related to passive smoking are more likely to recall being exposed to passive smoking."
The referenced study is from the Journal of the National Cancer Institute. Its key conclusions are (my emphasis):
A large prospective cohort study of more than 76,000 women confirmed a strong association between cigarette smoking and lung cancer but found no link between the disease and secondhand smoke.