TL;DR:
The slipping on a banana skin gag could have been the sanitized equivalent of horse manure but there is absolutely no evidence that supports the QI anecdote.
It's often banana skin in British English.
In the 19th century, the problem of horse manure in the streets was no laughing matter. It wasn't so much humans who slipped on the horse droppings but the horses themselves–often with fatal consequences for the equine machine and its passengers. In 1894, The Times estimated that by 1950 “every street in London would be buried nine feet deep in horse manure” Luckily, the advent of asphalt, the bicycle and the trolley car drastically reduced the number of urban horses at the turn of the 20th century and with it the mass production of manure.
There are several sources that talk about the street hazard of horse dung in wet weather.
In a letter addressed to the editor of Mechanics' Magazine, 1845, a subscriber argues that the principal disadvantage of wood pavements, compared to stone, outweigh any of their benefits.
N.B pavement referred to the road surface not sidewalks.
[all emphasis in bold mine]
The porosity of the wood causes the pulverized dung, when slightly wet, and thereby in a greasy state, to adhere with such tenacity, that the horse slips on the greasy surface;…
Is Mr. H. aware that there are about three cartloads of horse dung deposited per day in Regent-street? If so, there must be in all London some hundreds of loads per day deposited on the surface of the streets. Would not that cause greasy mud, and consequently slipperiness? ...ridiculously clean, yet the wood is frequently so sticky and greasy that the brooms cannot act, and it is consequently very often unsafe to ride over.
From The Practical Magazine, Volume 2, 1873
From a series of most carefully conducted observations, it results that the slipping of horses only takes place upon dirty causeways. On every occasion when a horse passes from species of pavement to another it evinces a sort of hesitation. This is the more evident on a passage from a hard uneven pavement to the smooth and elastic surface of asphalt. But an accident is most to be dreaded when the horse passes from ordinary pavement to asphalt during a period of rainy weather. In itself the surface of the asphalt is never muddy but during foggy weather or mist rains the mass of dust and horse dung unites and is mixed with the wet, and from accurate observations made by one of the most trustworthy engineers of Paris, it increases in volume five-fold. Thus a soapy dough is produced which makes the entirely even surface of the asphalt into a slide.

[source]
The Country Gentleman's Magazine, 1876
Not only are the road surfaces too slippery to afford a proper foot-hold for horses struggling with heavy loads through the almost impassable roadways, but the horse-dung also accumulates, and mixes with the mass of snow and mud, until the atmosphere is disgustingly defiled.
In Paris, street cleaners not only removed horse excrement from the pavements, they also added sand to the road surface.
However by 1906, a journalist for Car Illustrated: A Journal of Travel by Land, Sea, & Air, argued that side-slipping occurred when the roads were wet, regardless of whether horse droppings were present or not.
It is suggested that side-slipping is caused by the horse manure, but such a suggestion is absurd. There would be no side-slipping in dry weather even if the manure covered the street from side to side, but in London it is swept up almost as soon as it falls on to the road surface. Skidding occurs on country roads equally as on town thoroughfares. In both it happens only when the weather is wet.
All of the above appears to support the QI claim; however, the shiny yellow banana peel (or ‘skin’) was already infamous for being slippery in the 19th century.
Comedians on stage or on the silver screen had no need to use it as a euphemism for stepping on horse manure.
Banana skins were hazardous
Bellevue Gazette, March 4, 1858 “How He came to be Married”
We were just entering the parlor door, when down I went slap on the oilcloth, pulling Sal after me. Some cussed fellow had dropped a banana skin on the floor, and it floored me. It split an awful hole in my rassimeres right under my dress coat tail.
1884 Monthly Magazine
“Wasn't it a little singular, old man, that you should have slipped on the very banana skin that you threw on the sidewalk so carelessly?"
Over The Teacups, 1891, by Oliver Wendell Holmes
I would not for the world insinuate that he wishes she would slip and twist her foot a little, – just a little, you know, but so that it would have to be ... There was a banana-skin which she might naturally have trodden on, in her way to the tea-table.
In what appears to be a humorous article in the Monthly Magazine, it shows that people slipping on banana peels was a common occurrence. The piece was replicated in many newspapers across the United States with various titles, one of which was “Four very different ways of describing a common accident” by The Morning Call, San Francisco, 1891. I'll limit myself to the first two:
The Vulgate. — A lady slipped on a banana skin on School Street yesterday
afternoon and broke a leg.
The Staccato Style.— It is a job for the surgeon.
A banana skin did it.
There was a slide, a scream, and a dull thud.
The atmosphere was full of lingerie, frou-frou, hair-pins, and bric-a-brac.
It was laughable.
But only to a few.
A dozen men gallantly rushed to the rescue.
It was a woman who had fallen.
Horror!
A nether limb was broken.
She was in agony.
And all because of somebody’s carelessness.
Saul has slain his thousands.
The banana skin has slain its ten thousands.
The following snippet might raise a wry smile or two among readers.
On May 16, 1894, claim on the Boston and Maine Railroad of having been injured by slipping on a banana peel when stepping out of a car at the Prospect Hill Station. The banana peel was produced in evidence.
Banana: The Fate of the Fruit That Changed the World (2008)
Bananas were cheap and plentiful in the United States. They became hugely popular for being healthy, tasty, and surprisingly sweet and by 1910, Americans were buying over 40 million bunches of bananas. In addition, the fruit was a real bargain. In 1913, a dozen bananas in the US cost the same as two homegrown apples.
In chapter 11 the author goes on to explain how the elongated yellow fruit earned its reputation for being amusing, and says
The downside to the banana's portability, according to historian Virginia Scott Jenkins, was that there was no place, other than the gutter, to discard what banana marketers had taken to describing as the “sanitary wrapper.” People actually did slip, fall, and sometimes injure themselves when they stepped on them. What we know as a movie gag was real enough that in 1909 the St. Louis city council passed an ordinance prohibiting persons from “throwing or casting” a “banana rind” on public streets or sidewalks
There is a “funny” postcard, dated 1906, which verifies this municipal legislation was not isolated to St. Louis.

Conclusion
I found no evidence–absolutely none–that supported Jimmy Carr's anecdote of early 20th century filmmakers replacing the icky ‘slipping on horse manure’ gag with the more sanitary, and visually appealing, banana skin. The QI episode responsible for this claim was first broadcast on 10th December, 2010. In the YouTube link posted above, the British comedian–not the show's writers– says
- “Someone told me a fact about early cinema […] they felt like that horse manure was a bit–kind of–cheapened the cinema because this was ‘Art’ […] but everyone understood it at this time […] that was in place of horse manure.”[sic]
One of the earliest references that I found replicating Jimmy Carr's fact is from All the Tropes, which appeared on November 1, 2013, but three years after the British Quiz show aired.
It has been claimed that the trope actually originated in the time before cars with combustion engines and electric trams, when almost all public transport was powered by horses. And the huge amount of horse manure that piled up in the streets became a serious slipping hazard. When film was invented, this was considered inappropriate to show, so the banana peel became a substitute.
That paragraph was more likely prompted by the sub-Reddit Today I Learned (TIL). “TIL The early comedy cinema joke of slipping on a banana peel was a reference to slipping in horse manure.” (September 10, 2013)
Laura Turner Garrison's thoroughly researched article, “How Did Slipping on a Banana Peel Become a Comedy Staple?” (2012) never once mentions the banana peel as an elegant substitute for horse poop in slapstick comedy.
And last but not least, the 2009 Banana Slip episode by MythBusters fails to mention that slipping on dung was or is more perilous than banana peel despite the analysis of the banana peel routine.
- “According the laws of comedy the myth is that one banana peel plus one person equals one person on their butt.”
In the words of a mythbuster, the QI story is “…busted but only just.”
Further reading: