Well, after a quick search it seems that Stephen Colbert is probably correct.
First, it's good to mention that according to the Department of Justice most Voter Fraud will fall under state jurisdiction and not Federal jurisdiction unless there are threats or discrimination based on race, religion, or national origin.
There is a popular figure oft-cited by The Daily Show and other sources who make arguments against certain Voter ID laws, which is 10 cases of "in-person" Voter ID fraud over the last decade or so. This seems technically correct, according to this article by ABC:
Over the past decade Texas has convicted 51 people of voter fraud, according the state's Attorney General Greg Abbott. Only four of those cases were for voter impersonation, the only type of voter fraud that voter ID laws prevent.
No doubt you noticed, though, that Stewart and other critics of Voter ID laws specifically mention cases that Voter ID laws would have prevented - although the total amount of Voter Fraud is higher.
Is it enough to matter? Well according to Google's statistics the difference in votes between the candidates was 3,378,662. A much closer race is the infamous 2000 Elections, where the FEC's statistics show a difference of 350,428 between the Democrat and Republican candidates.
It would have taken 7,009 cases of voter fraud per state in 2000 to make that gap, and over 67,500 cases of voter fraud per state to close the gap in 2012. I could not find a source for the total cases of voter fraud, but one of PolitiFact's articles attempts to address this:
To get the data, News21 reporters sent records requests to elections officers in all 50 states seeking every case of fraudulent elections activity, including registration fraud, absentee ballot fraud, voter impersonation fraud and casting an ineligible vote. News21 said it received no useful responses from several states. With some states, including Massachusetts, Oklahoma, South Carolina and South Dakota, the cases included in the database came from a survey of alleged election fraud conducted by the Republican National Lawyers Association. And in some states, some but not all local jurisdictions responded, and some responses were missing important details about each case. Despite those issues, News21 defends its work as "substantially complete" as the largest collection of election fraud cases gathered by anyone in the country.
They also go on to say that News21 found 307 cases of Voter Fraud (en totale, including Impersonation) in Georgia - which is 6x higher than Texas and over 10x higher than Pennsylvania (who News21 said had 29 cases). The Republican National Lawyer's Association challenged the analysis, saying Georgia had 375 cases.
Given that I've no baseline or average to work from, but Georgia appears to be one of the more rampant offenders - every state in the nation would have to have eighteen times the amount of voter fraud that Georgia had in a single year to make up the gap in the 2000 elections. It would take over 9,000-fold Georgia's cases per state in 2012 alone to have made up the gap in the 2012 elections.
Basically, even if all states had as much voter fraud as the RNLA said Georgia had per year (which they don't), it would still only represent .0155% of the total vote.
I'd agree with Stephen Colbert and other pundits; for all intents and purposes, Voter Fraud is practically a non-issue.