Bricker and Ibbitson claimed (in a Guardian article): "But a growing body of opinion believes the UN is wrong. We will not reach 11 billion by 2100. Instead, the human population will top out at somewhere between 8 and 9 billion around the middle of the century, and then begin to decline." I assume that is the claim you are talking about.
We don't know what will happen, there is a variety of projections or predictions - even at the UN alone. The projection referred to by Bricker and Ibbitson is, I think, from the UN's 'medium variant' from its 'Standard Projections' or the 'median' from its 'Probabilistic Projections', which both put the peak population at just under 11bn in 2100 (no numbers for post-2100 but if we continued the trend then the numbers would seem to decline). But in the UN's 'low variant' from its standard model and 'lower 95' and 'lower 80' (of the respective confidence intervals) from its probabilistic models the population peaks at lower numbers earlier than 2100 after which the populations decline (here is a nice chart representing the probablistic models).
So Bricker and Ibbitson may disagree with the UN's medium or median projections but perhaps they don't disagree with some of the UN's other projections. Indeed the low variant of the Standard Projection puts the peak population at 8,923,228 in 2054 after which the population declines - and recall that Bricker and Ibbitson said "the human population will top out at somewhere between 8 and 9 billion around the middle of the century, and then begin to decline."
Out of interest, let's look at the fundamentals of what we're talking about because is it conceivable that the global population could decline? I think it is.
A commonly used measure of births per woman in the population is the Total Fertility Rate (TFR) or simply the fertility rate. This is "the average number of live births a hypothetical cohort of women would have at the end of their reproductive period if they were subject during their whole lives to the fertility rates of a given period and if they were not subject to mortality. It is expressed as live births per woman." (UN). Here is a chart representing the world's TFR since the 1950s. You can see that the TFR has halved since then - from 5 to 2.5 live births per woman.
The Replacement Level Fertility (RLF) or simply the replacement rate is the TFR at which women give birth to enough babies to sustain the population at or around the current numbers - in other words the rate at which the current generation will replace itself with the next, and so on. Conceivably there can be TFRs such that the population would grow and TFRs such that the population would decline. The UN Population Division says the global RLF is about 2.1. Each country has its own RLF, depending on female mortality and gender imbalance at birth. In the UK, for example, it is 2.075 but in less developed countries (with higher mortality) it might exceed 2.3.
The Fertility Factors, or predictors of Total Fertility Rate, in other words positive or negative correlations with fertility, include contraception take-up, participation of women in the workforce, women's education, women's incomes, changing attitudes, the proportions of households that live in the city or the country and of course deliberate population control - some of which are non-causative, difficult to disentangle and about which there are disagreements and competing theories among demographic experts. And of course there have been changes in life expectancy.
Now, there are some countries where their Total Fertility Rate is lower than their Replacement Level Fertility. Without immigration, if such rates persisted the populations of such countries would indeed decline (assuming life expectancy would not sufficiently increase to counter that). For example, the UK has TFR of 1.7 and RLF of 2.075. If similar circumstances occurred globally (and with no immigration from extra-terrestrials!), the world's population would decline over time. But there is disagreement among experts about whether and when that will happen globally or the likelihood of any one scenario compared to the other scenarios.