From http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0002050/
(my emphasis)
Bacteria are normally present in the mouth. The bacteria convert all foods -- especially sugar and starch -- into acids
[...]
The acids in plaque dissolve the enamel surface of the tooth and create holes in the tooth (cavities). Cavities are usually painless until they grow very large and affect nerves or cause a tooth fracture. If left untreated, a tooth abscess can develop. Untreated tooth decay also destroys the internal structures of the tooth (pulp) and ultimately causes the loss of the tooth.
Carbohydrates (sugars and starches) increase the risk of tooth decay. Sticky foods are more harmful than nonsticky foods because they remain on the surface of the teeth. Frequent snacking increases the time that acids are in contact with the surface of the tooth.
Virtually all food contains carbohydrates, so every type of food will contribute to tooth decay if dental hygiene is not applied properly. However, the more sugar the food contains, the more the bacteria flourish and damage your teeth, as explained on that website.
If you apply proper dental hygiene, then eating sweets should not be a problem. The trouble is that sweets are usually consumed as snacks between meals and thus a long time passes until teeth are brushed, allowing more time for tooth decay.