2 of 4 added 233 characters in body

While I like DVK's answer, let's add another one that is I hope simpler.

An official has to decide which contractor to use to build a bridge. In the absence of bribery they will choose the bid that gets the best value, to the best of their ability. All well and good. With bribery in the picture the contract will go to a contractor with a higher price, or lower quality, costing the bridge builder (usually a government) more money. The actual amount of the bribe is usually trivial compared with the cost of the bridge, and doesn't figure in the calculation - it's the increased cost that causes the problem.

With other kinds of bribes there are bigger problems. Let's say the bridge builder uses substandard concrete and bribes the quality inspector not to notice. Now you have a faulty bridge and the cost of replacing it when it collapses. Or your police official is bribed to not arrest a crime lord. Crime usually costs someone money, so society is again worse off by quite a lot.