From Wikipedia's article on Eidetic memory's prevalence
According to Herman Goldstine, the mathematician John von Neumann was able to recall from memory every book he had ever read. [15]
Refering to Page 167 of Goldstine's The Computer from Pascal to von Neumann
As far as I could tell, von Neumann was able on once reading a book or article to quote it back verbatim; moreover he could do it years later without hesitation.
He then goes on with a story about him challenging von Neumann to tell how a particular book's first chapter started with and JVN being able to recite it flawlessly.
The same section of the Wikipedia article starts with the following sentence
Eidetic memory is typically found only in young children, as it is virtually nonexistent in adults
Now , extraordinary does little favor in describing von Neumann's mental capabilities and scientific contributions but I think Goldstine's claims are bold even for someone like him. There are many more anecdotes about his abilities and, indeed, there are people who have savant skills, however even the ones who have had those skills "over-developed" performed significantly below average in other cognitive tasks. That makes the story about Von Neumann even less plausible, considering he was a polymath.
Finding a source disputing Goldstine's claims about von Neumann is obviously not the answer one should be looking for however it would be interesting to discuss, from a neuroscientific point of view, if it's really probable that the human brain can store such large amounts of information.