There has been a database of football/American Football/gridiron injuries maintained since 1931.
The 2011 Annual Survey of Football Injury Research contains a table (Table 1) which lists "Fatalities: Directly Due to Football".
The 2011 figure for High-School students is 678, and if you subtract the 13 deaths since 2008, you get the 665 figure quoted by Penn Jilette.
The tragedies don't stop there. Table 2 lists another 484 high-school fatalities (1931-2011) "indirectly" related to football. Also, these figures don't include "sandlot", pro, semi-pro and college deaths.
On the other hand, extending back to 1931 is distorting the picture, as the sport is now safer than it was 80 years ago.
The report explains:
What is important is that the National
Center for Catastrophic Sports Injury Research has been collecting
catastrophic football injury data (fatalities, disability, serious
injuries) at the University of North Carolina at Chapel Hill for
the past 48 years and has been directly responsible for the
reduction of football fatalities from 36 in 1968 to zero in 1990,
and the reduction of cervical cord injuries from 30 a year in the
late 1960s to single digits in most years since 1991. We will
continue to collect this data and to make safety recommendations
to the NFHS, the NCAA, and youth football programs.