Yes, there is a net environmental benefit from photovoltaic panels.
It would be possible to contrive a corner case where there was not a net benefit (for example, having a PV system illuminated only by a light bulb, rather than by the sun); but such cases are extremely rare.
Let's break down the big, somewhat cloudy question of "net environmental benefit" into some component questions that are commonly raised. Does PV have net energy production? How much greenhouse gases from fossil-fuel production does it displace, when the full product lifecycle is accounted for? How do aggregate damage costs compare? I'll deal with each of those in turn.
Net energy production
Fthenakis and Alsema (1) looked at the net energy production of photovoltaics, based on 2004 technology (things have improved a lot since then). They found energy payback times of between 1.0 and 2.2 years, depending on the specific panel technology, and where it was manufactured. So all energy generated after that time is net energy production. Systems typically last 15-30 years.
The 2009 review of life cycle assessments by Sherwani, Usmani and Varun (3) found energy payback times of 1.5-5.7 years, for systems with lifetimes of 20-30 years.
In 2012, Fthenakis (4) looked at historic trends of energy payback times [EPBT], and found that they'd been improving for 40 years (NB the vertical axis has a log scale):

That gives values for energy returned on energy invested (EROEI) in the range 10-40.
So PV has a high net energy production
Greenhouse gases displaced
First, let's note that the power into a grid and the power consumed must match at all times. So if PV generation increases, some other source of power must decrease, in order to keep the system balanced. So some other generator will get turned down. The type of generation being displaced does make a difference. In most of the world, in 2012, it will usually be a fossil-fuel plant - coal or gas (see the World Energy Outlook, IEA).
Fthenakis and Alsema (1) cite the ExternE and Australian Coal Association research on full-lifecycle greenhouse-gas emissions, and found that monocrystalline silicon PV came out at 100-180gCO2e/kWh, whereas coal was 900g, and gas CCGT was 400-439g. Their own calculations for different PV technologies gave even lower results: 37g for polycrystalline silicon; 21g for EU Cadmium Telluride; 25g for US Cadmium Telluride.
Similarly, Sherwani et al (3) found full-lifecyle greenhouse gas emissions from PV were in the range 9.4-104gCO2e/kWh
On a full-lifecycle analysis, there are substantial greenhouse-gas savings with PV.
Total Damage costs
Flimzy poster, in comments on the question, notes that:
you're often comparing apples and oranges. Coal-generated electricity emits carbon into the atmosphere, whereas manufacturing a PV cell may result in some liquids being emitted into a river ... how do you measure which is "overall worse?"
That's true. And there is a standard method for aggregating externalities, which is to monetise each aspect of damage, and then sum those costs.
Fthenakis and Alsema (1) found that the unit damage cost for PV systems was 0.126-0.177 eurocent per Kwh, using a remarkably low value of CO2 damage (€19/tCO2e). Even at that level, the CO2-damage alone from fossil plant exceeds the total damage costs from PV (€19/tCO2e x 400gCO2e/kWh = 0.76 eurocent per kWh).
Total environmental damage costs from PV, are less than those of the fossil-fuel generation that they displace.
Metal specifics
For anyone interested in the detail of metal extraction for PV, Ftheankis, Wang and Kim looked at the specifics for zinc, copper, aluminum, cadmium, indium, germanium, gallium, selenium, tellurium, and molybdenum (2).
Sources
(1) Fthenakis VM, Alsema E. Photovoltaics energy payback times, greenhouse gas emissions and external costs: 2004-early 2005 status. Progress in Photovoltaics: Research and Applications. 2006;14(3):275–280. DOI:10.1002/pip.706
(2) Fthenakis VM, Wang W, Kim H. Life cycle inventory analysis of the production of metals used in photovoltaics. Renewable and Sustainable Energy Reviews. 2009;13(3):493–517. DOI:10.1016/j.rser.2007.11.012
(3) Sherwani AF, Usmani JA, Varun. Life cycle assessment of solar PV based electricity generation systems: A review. Renewable and Sustainable Energy Reviews. 2010;14(1):540–544. DOI:10.1016/j.rser.2009.08.003
(4) Fthenakis V. How long does it take for photovoltaics to produce the energy used?, in PE Magazine, from the National Society of Professional Engineers, Jan/Feb 2012. PDF available here.