The Straight Dope reports that while the full environmental impact is hard to measure, there is not much evidence to support the "Hummers are better for the environment than the Prius" claim:
The cause of the controversy seems to be a report called "Dust to
Dust" by Oregon-based CNW Marketing Research. The report claims a
Prius has a higher lifetime energy cost than a Hummer, an assertion
cited by George Will in 2007 in his syndicated newspaper column. But
the report is ludicrous. It evidently was self-published, lists no
authors, quotes no technical literature, never explains its
methodology, and contains numerous unsupported and often bizarre
assertions.
Slate's Green Lantern also has a go at answering this question, and comes to much the same conclusion.