This strongly depends on definition of what smoothness is.
The Discover Magazine blog addressed this in 2008
OK, first, how smooth is a billiard ball? According to the World Pool-Billiard Association, a pool ball is 2.25 inches in diameter, and has a tolerance of +/- 0.005 inches. In other words, it must have no pits or bumps more than 0.005 inches in height. That’s pretty smooth. The ratio of the size of an allowable bump to the size of the ball is 0.005/2.25 = about 0.002.
The Earth has a diameter of about 12,735 kilometers (on average, see below for more on this). Using the smoothness ratio from above, the Earth would be an acceptable pool ball if it had no bumps (mountains) or pits (trenches) more than 12,735 km x 0.00222 = about 28 km in size.
The highest point on Earth is the top of Mt. Everest, at 8.85 km. The deepest point on Earth is the Marianas Trench, at about 11 km deep.
Hey, those are within the tolerances! So for once, an urban legend is correct. If you shrank the Earth down to the size of a billiard ball, it would be smoother.
I disagree with definition of smoothness used by Discovery Magazine. By that definition, medium sandpaper (grit particle size of 0.005 in) is also smooth, which doesn't quite go with my definition of smoothness. In fact I find claim that sandpaper is smooth to be ridiculous.
With mountains reaching in excess of 8,000m, scaled down that would be 0.0015 in. which means, that scaled down Earth's "smoothness" is equivalent to that of 320 grit sandpaper.
How does it compare with actual billiard ball, woliveirajr's answer is helpful:
What does ball surface look like:
Note, that variation is about 0.55μm, while 0.005 inches official tolerance for shape is 127μm. 0.55μm scaled up to Earth size would be less than 125 meters.
As for shape, which is really what the ±0.005 inches regulation is about, Earth is non-spherical, it's oblate spheroid with:
- Equatorial radius: 6,378.1370 km
- Polar radius: 6,356.7523 km
- Mean radius: 6,371.009 km
(sources: WGS-84 and IUGG )
Just the non-spherical shape already disqualifies scaled down Earth as official billiard ball, allowable tolerance in diameter would be 28,326 m while difference between Earth's polar diameter and mean diameter is 28,513 m. Although it is quite close call.