It is within the range of 35,000–75,000 depending on your level of learning, so a better number would be 50,000.
Underestimates
Part of it is going to depend on what you consider to be your vocabulary. Do you only count words you use on a regular basis and know thoroughly, or does a word you might only scarcely recognize once you see it count? It is typically established 90% of English you might use or encounter on a daily basis comprises of just 3,000 words, and these 3,000 words are of special interest to E.S.L. studies. However, some very common words that people would consider a regular part of their vocabulary do not exist within these 3,000 words. As a native speaker of the English Language, I would suppose that almost any other native speaker above 5 years of age should recognize the word frown, but it is not listed in the Longman Communication 3000, which is a list prefaced by the claim that these 3,000 words are the 3,000 most commonly occuring words based on analysis of a 390 million word corpus and constitute 86% of the words used. The Oxford 3,000 does not have it either.
Part of it is going to depend on what you determine to be a single word and the Test Your Vocab website is using a very conservative definition and that skews their numbers towards the lower bound. They only count headwords, and not derivations. They would count quick as a word, but not quickly because it is just a derivation of the word quick with a predictable change in signification. Maybe some scholars would find this a reasonable restriction, but I for one do not.
Test your vocabulary also chose to omit technical specialist terms from the test simply because they lack general applicability. I for one do not consider these to be sound criteria, nor would Samuel Johnson as he explained in A Plan for An English Dictionary. Notably he states:
The academicians of France, indeed, rejected terms of science in their first essay, but found afterwards a necessity of relaxing the rigour of their determination; and, though they would not naturalize them at once by a single act, permitted them by degrees to settle themselves among the natives, with little opposition; and it would surely be no proof of judgment to imitate them in an errour which they have now retracted, and deprive the book of its chief use, by scrupulous distinctions.
Now the Test Your Vocabulary test certainly includes many challenging words in the commonly shared vocabulary that would test the limits of anybody's knowledge, but they might gloss over many perfectly legitimate words due to simple obscurity, and cut off the results at an estimated 45,000 words.
So again, the test your vocabulary estimate is probably too low and their summary of results on May 10th 2013 suggest that the average speaker knows 20,000–35,000 words. Therefore, the actual number should be greater than that.
The Upper Bound and Typical Estimate
A far more usual estimate is 35,000–75,000, depending on the level of the speaker's learning. This is the range given by Professor David Crystal as cited in the B.B.C. News Magazine article Words in the Mental Cupboard which was written by Caroline Gall and published on their website on April 28th 2009. Professor Crystal supposes that the size of a typical vocabulary is around 50,000. Another source which makes this claim is Improve Your Speed Reading by Phil Chambers, which claims there was a joint google/harvard study on the matter of vocabulary size.
I believe these numbers to be accurate because we have concrete evidence that the upper bound is achievable in the form of historical dictionaries.
According to Peter Sokolowski, editor at large for Merriam-Webster, the 1864 edition of their Dictionary was "the first dictionary in English created by a large staff", so before then we can presume that they were principally the work of one man.
According to the Noah Webster House & West Hartford Historical Society:
Webster’s greatest achievement was the dictionary. In 1800 he published his intentions of writing a dictionary. He published a shortened, concise but comprehensive, version in 1806. The final version was finished in 1825 and published in 1828. It contained 70,000 words.
This information is somewhat misleading. The A Compendious Dictionary of the English Language (1806) may have lead to the creation of An American Dictionary of the English Language (1828), but they were distinct works.
However, the word count is probably correct, or if not a close enough estimate. I know this because in the past, I had run a wildcard search on A.R.T.F.L. project, which returned somewhere around 65,000 words, and like most of the online transcriptions it had some glaring omissions that are included in a hardcopy, such as the usual form of rank. You will not be able to verify this with them directly, since A.R.T.F.L. Project seems to have taken down the resource, but there is a hardcopy fascimile of the original (albeit bound in one volume) published by The Foundation of American Christian Education, with the I.S.B.N. of 978-0-912498-03-04 if the especially skeptical among you would like to verify my claim with a hand-count.
Granted, An American Dictionary of the English Language has some questionable inclusions which may inflate the number, such as compounding loving-kindness as a single world. Also, Webster was not entirely without aid, as he had help from Emerson Worcester and his Son in Law. However, on the other hand, this was during the early 19th century so he had fewer learning aids and an aacessibility handicap, and An American Dictionary of the English Language was the largest at the time of its initial publication, containing 12,000 words that were not published in prior dictionaries, which is over half of the 20,000 count mark alone.
These will not all be all of the words you use on a daily basis daily, and since a dictionary acts as a record it may not even be representative of all of the words you might be able to recognize off of the top of your head. However, what this does prove is that learning roughly 75,000 words is an accomplishable feat with dedicated study, lending credit to the notion that a people have the capacity to learn 20,000 or more words.
This lends credit to the notion that the numbers given by Professor Crystal and the joint Google/Harvard study are indeed accurate, at least under the provision that their definition of what constitutes a "word" is reasonable, since their upper bound roughly matches that which has been actually been documented by a famously learned man.
However it also gives us reason to cast doubt that any estimate of the average above or even approaching 75,000 words is reasonable, given that Webster dedicated 25 years to creating his dictionary in addition to whatever pre-existing knowledge he had. It is true we have better learning tools and greater accessibility to information than we did in the 19th century, but not very many individual people spare 25 years of their life to the sole task of compiling words, and there is less motivation to do so since the practice of independent lexicography has effectively stopped.