Many devices when turned off still use a bit of power (e.g. to detect when the on button is pushed when there is no real physical power cutoff) leading them to become dead sooner than when you had removed them when you don't need the device.
As vartec mentioned battery leaking can be damaging to the electronics, depending on the design of the battery compartment
Here's an Knol article about it.
When carbon or alkaline batteries have become discharged, the
chemistry of the cells will degrade and some hydrogen gas will be
generated. This out-gassing will result in increasing pressure within
the battery. Eventually, the excess pressure will either rup- ture
the insulating seals at the end of the battery, or bulge and rupture
the outer metal canister, or both. When this happens, an acidic (for
carbon cells) or caustic (for alka- line cells) electrolyte gel will
ooze from the battery.
The leaking electrolyte can corrode the metal housing and battery
contacts of a simple flashlight, and it can damage or destroy the
delicate circuitry of an expensive electronic flashlight.
Furthermore, swelling of the battery canisters can render one or more
of the batteries hopelessly jammed within the flashlight body. One
leaking battery can cause a chain failure, when its leaking goo
corrodes the adjacent ones so that they then leak. A five dollar set
of dead batteries can and will destroy a five hundred dollar
instrument if you let them.