Try putting hot water in a glass, empty it and then pour ice cold water in it. It is very likely it will break. The rapid change in temperature causes non-even contractions of the material and makes it break.
This happens to the enamel in your teeth. Although it does not need to break, since the temperature difference is not so extreme, it may weaken the enamel and make it easier for tooth cavities to form.
Here is a scientific paper which explains it more rigorously
Abstract as follows:
This paper presents the results of experiments conducted with extracted human and bovine teeth. The teeth were subjected to thermal cycling at temperatures between 140 and 90 F. The results offer conclusive evidence that thermal fracture may be induced by the thermal stresses caused by the temperature cycling. Less than 3,000 thermal cycles cause severe cracking or the propagation of cracks previously existing in the teeth, or both.