I read a book in which someone hacked into a computer and used the speakers (not a microphone) to listen to what was happening in the room. Is this even remotely possible without gaining physical access to the computer, assuming you could completely take over the computer electronically?
Most speakers really can work as microphones; the signal is weak, but it is there. See, for example, All speakers are microphones:
What happens is that in a speaker there's a magnet and coil, and a pair of wires connecting it up to something. What is supposed to happen when the device is used for outputting sound that electricity is converted to sound. Electrical output from an amplifier is put into the speaker and turned into sound. However, the process is reversible, so it's possible for the magnet and coil in a speaker to pick up sound in the air and turn it into electrical signals in the wires.
Some audio devices have configurable connections to the connectors (jacks) so that you can plug a speaker or microphone into any jack; this means that you can reconfigure them to record the signal coming from the speakers. See the manual for a common onboard audio chipset RealTek, Chapter 3 Audio I/O, page 11:
Realtek HD Audio Manager frees you from default speaker settings. Different from before, for each jack, they are not limited to perform certain functions. Instead, now each jack is able to be chosen to perform either output (i.e. playback) function or input (i.e. recording) function, we call this “Retasking”.
There is one more step: between the jack and the speaker may be an amplifier. Amplifiers don't pass any signal from their outputs to their inputs, so if a speaker signal is to be read, the speaker must be connected directly, without an amplifier. This would require the speaker to be a passive speaker plugged directly into the jack:
...a passive speaker (or unpowered speaker) is a speaker which does not have its own power source and draws power from an amplifier located outside of the loudspeaker enclosure.
So, if you hacked the computer into a state where you can install and run your own device drivers, and if there are passive speakers connected in a "retaskable" jack, then it is possible.