The goal of defragmentation is to reduce amount of random reads/writes vs sequential read/writes. It organizes files to be on the consecutive sectors of the disk.

I case of spinning HDDs, random access means there are two latencies involved. First the head has to move to the right track (seek time), then it must wait till the sector to be read spins into the position (rotational latency). In case of sequential read there is no need to change track and the sector you're going to read is just the next one, thus both latencies ar eliminated.

source: MS TechNet
Now how big is the performance hit on sequential vs random? Let take recent Seagate Barracuda as an example:

source: Storage Review
So the answer is obvious. More fragmented the file system, more random seeks you have to do.
BTW. most modern file systems try to reduce fragmentation when creating new files. So for typical use fragmentation might not be real issue, unless you got your disk to be almost 100% full.