In general, elderly drivers account for fewer accidents, as you correctly pointed out in your question. However, as age increases, fatality rate increases drastically.
Between April 2001 and March 2002, the
rate of fatal crash involvement per
100 miles driven for older drivers was
higher than any other age group
besides teenagers (See graph 4). The
over 85 age group had the highest [rate].
On closer inspection, it turns out the higher fatality rate of crashes involving the elderly is due to their own fragility.
Per licensed driver, drivers 75 and older kill fewer pedestrians, bicyclists, motorcyclists, and occupants in other vehicles compared with drivers aged 30-59. The majority of the harm caused by elderly drivers is inflicted on themselves and on other elderly passengers, as 75% of people who die in crashes involving elderly drivers are either an elderly passenger or driver themselves.
So elderly drivers do not cause the most accidents. And regarding the accidents they do cause, they are a bigger threat to themselves than to pedestrians or passengers of other cars. Check out the source for some interesting graphs regarding these statistics.