Your dad has the right of it. Unfortunately, the literature on human cold adaptation is not very lay-person friendly. US military publications seem to provide the most accessible information on the subject:
Habituation, the most common pattern observed in both acclimatization and acclimation studies, is characterized by a blunted shivering and vasoconstrictor response to cold exposure. Habituation appears to require only brief, intermittent cold exposures to he induced, and can develop when only small body regions are exposed unprotected to cold. It allows extremity skin temperatures to be maintained higher during cold exposure. The higher skin temperatures coupled with the absence of shivering are advantageous in that manual dexterity and comfort are enhanced. In one acclimation study in which subjects were exposed to moderate cold conditions for a prolonged period, a metabolic form of cold acclimation appeared to develop. This adaptation was characterized by an enhanced shivering thermogenesis during cold exposure. When individuals acclimatize or acclimate to cold conditions severe enough to repeatedly cause a significantly body temperature fall, an insulative pattern of adaptation develops, characterized by enhanced mechanisms for body heat conservation. The mechanisms determining the pattern of adaptation to chronic cold exposure appear related to type of cold exposure conditions, the amount of body heat lost and the degree to which shivering thermogenesis compensates for heat loss and defends body temperature.
Above is from this abstract
Humans get significantly better at handling heat or cold by repeated exposure to the extreme, and working in those extreme conditions.