The 6000 year old figure comes from the Genealogies in the bible -Ussher - He added up the lifetimes from Jesus (1 AD) back to Adam. The Genealogies are surprisingly complete. If you believe in the non-miraculous parts of the bible, it's actually a decent way of figuring out the date of "The Fall of Man." Let us assume, that there was a man called Adam, who was the first follower of Jehovah, and that he lived some time on the order of 6000 years ago.
If we accept that, there is still no evidence that the "Adam" who was kicked out of Eden is the same Adam who was "created" on the sixth day.
What we cannot assume, unless we are biblical literalists, is that the garden of Eden and creation stories are anything but metaphor. The "days" in creation could very well be arbitrary periods of time on the order of millions or billions of years. Jesus himself Peter alluded to this possibility when he said "A"... With the Lord a day is as tolike a thousand years to the Father, and a thousand years are like a day." [NIV]"2 Peter 3:8
So, even taking the Judeo-Christian genealogies, we still have very weak evidence of a young earth. The only people who believe this are the aforementioned literalists, who believe in the young earth because they are literalists. (That is, they are not literalists because they believe in a young earth.)
TLDR: The only evidence for a young earth is biblical. Take it as you will.