You would not need to "infer" that people with high IQ generally have less offspring; this concept has long been known to be true, statistically. Just note that "fertility" as defined in this manner is simply a measure of the total number of offspring, not how capable one is of conceiving.
I found these sources in 15m of searching. I'm sure you can find many more with concentrated effort. Open up any article and refer to their references for a start.
- Campbell, C., & Preston, S. (1993). Differential fertility and the distribution of traits: the case of IQ. The American Journal of Sociology, 98(5), 997+.
- Gelade, G. A. (2008). The geography of IQ. Intelligence, 36, 495-501.
- Harvey, J., & Lynn, R. (2008). The decline of the world's IQ. Intelligence, 36(2), 112+.
- Hondroyiannis, G., & Papapetrou, E. (2005). Fertility and output in Europe: new evidence from panel cointegration analysis. Journal of Policy Modeling, 27, 143-156.
- Loehlin, J. C. (1997). Dysgenesis and IQ: What Evidence Is Relevant?. American Psychologist, 52(11), 1236-1239.
- Lynn, R., & Jarvey, J. (2008). The decline of the world's IQ. Intelligence, 36, 112-120.
- Lynn, R., & Van Court, M. (2004). New evidence of dysgenic fertility for intelligence in the United States. Intelligence, 32, 193-201.
- Meisenberg, G. (in press). National IQ and economic outcomes. Personality and Individual Differences.
- Meisenberg, G. (2010). The reproduction of intelligence. Intelligence, 38(2), 220+.
- Preston, S. H., & Campbell, C. (1993). Differential fertility and the distribution of traits: the case of IQ. American Journal of Sociology, 98(5), 997-1019.
- Redwood, A. L. (1983). An economic-demographic approach to forecasting national and subnational birth rates. Socio-Economic Planning Sciences, 17(5-6), 355-363.
- Reeve, C. L. (2009). Expanding the g-nexus: Further evidence regarding the relations among national IQ, religiosity and national health outcomes. Intelligence, 37(5), 495-505.
- Retherford, R. D., & Sewell, W. H. (1989). How intelligence affects fertility. Intelligence, 13(2), 169-185.
- Retherford, R. D., & Sewell, W. H. (1988). Intelligence and family size reconsidered. Biodemography and Social Biology, 35(1-2), 1-40.
- Shatz, S. M. (2008). IQ and fertility: A cross-national study. Intelligence, 36(2), 109.
- Van Court, M., & Bean, F. D. (1985). Intelligence and fertility in the United States: 1912–1982. Intelligence, 9(1), 23-32.
- Vining, D. R., Bygren, L., Hattori, K., Nystrom, S., & Tamura, S. (1988). IQ/fertility relationships in Japan and Sweden. Personality and Individual Differences, 9(5), 931-932.
- Vining, D. R. (1982). On the possibility of the reemergence of a dysgenic trend with respect to intelligence in American fertility differentials. Intelligence, 6(3), 241-264.