ScienceDaily has a nice article on this, as well as many related articles. In the cited article, they state:
But the process of building pyramids, while complicated, was not as colossal an undertaking as many of us believe, Redford says. Estimates suggest that between 20,000 and 30,000 laborers were needed to build the Great Pyramid at Giza in less than 23 years. By comparison, Notre Dame Cathedral in Paris took almost 200 years to complete.
I think what gets people so confused is they mistake old cultures for being unintelligent. Humans have had the same level of intelligence for nearly 200,000 years, just not the full benefit of technology. (Citation: Hominid Brain Evolution Testing Climatic, Ecological, and Social Competition Models, Drew H. Bailey & David C. Geary, Hum Nat (2009) 20:67-79, DOI 10.1007/s12110-008-9054-0)
Furthermore, in the article, it states "laborers". A common misconception is that slaves built the pyramids, which is not the case. Archaeological evidence shows the builders were skilled and paid for their efforts. From the same article:
the image most people have of slaves being forced to build the pyramids against their will is incorrect.
An additional collection of articles can be found at this Discover Magazine Blog post by Andrew Moseman. It starts out saying:
Forget the myths about massive numbers of slaves or Jews building the great pyramids, Egypt‘s chief archaeologist argues this week. He says Egyptian researchers have found the tombs of more pyramid builders, and in those tombs more evidence that free men erected these monumental tributes to the ancient pharaohs.
And continues with numerous links to even more articles.
As to the assertion that it was impossible to do many of the things that the builders of the pyramids did, that is a common misconception people seem to have. Most people don't consider ancient humans to have been as intelligent as we are, when in fact they possessed the exact same intellect as we do today, just not the technology. And since we rely so much on advanced technology, many people make an argument of incredulity because we just don't do things the old fashioned way. Some people have started to collect reconstructions of those methods on the web.
The same argument regarding the mathematical precision could be made. Also, in ancient times, without our calendars and clocks, astronomical observations played a much more important role than today (i.e. when to plant, when to expect rains, etc.). Again, ancient humans were not stupid.