If you are cold, your body will be trying to produce energy to generate heat. Calories are a measurement of energy. We consume food (calories), the body turns it into energy or stores for later use as fat. In order to generate the heat the body needs when it is cold, it needs to either burn this energy or use fat stores.
This is a natural process that our ancestors and many animals need in order to survive the winter time. Before the cold months when farming and hunting is virtually stopped, our bodies learned to stock up when it starts getting cold. This will help to create more insulation (fat) and provide better survivability rates so that our bodies can afford to shed 10-20 lbs due to a lack of abundant food. NPR posted an article in 2011 discussing this topic with a quick quote:
"We are driven by things implanted in our brain a long, long time ago," says Ira Ockene, a cardiologist at the University of Massachusetts Medical School who has long been interested in how seasonal variations influence our health.
When we are warm, our bodies do not want to intake any energy(food) or generate energy as the hot weather(heat) is already causing our bodies to overload. We dissipate energy from our bodies via heat to help cool down the body along with the sweat that the body uses to naturally cool and regulate our temperature. Intaking more energy (food) would cause our body temperatures to rise slightly.
How Eating Affects Body Temperature
Eating generally leads to a slight increase in body temperature, as your metabolic rate increases in order to allow the digestion of food. Your temperature may increase by as many as 2 degrees F as the chemical reactions of the digestive process take place within your body. These chemical reactions are what produces the heat that causes a slight augmentation in body temperature.
This is a big reason why many diet pills often claim to reduce hunger and burn fat via a thermic reaction.
So in conclusion, much of this has to do with our seasonal reactions that have been embedded since before the modern comforts and luxuries. There is evidence that temperatures do directly affect our hunger via how we regulate heat/energy and as a learned survival instinct.