You ask whether it's a reasonable interpretation of the text.
Book IV of Augustine's Confessions (in Latin), which was cited in this answer, includes:
Itaque illos planos, quos mathematicos vocant, plane consulere non desistebam, quod quasi nullum eis esset sacrificium, et nullae preces ad aliquem spiritum ob divinationem dirigerentur.
So the text includes the word mathematicus (-cos).
Looking at the definition of this word according to Lewis and Short it means,
A. Măthēmătĭcus , i, m.
1. A mathematician, Cic. de Or 1, 3, 10; id. Ac. 2, 36, 116; id. Tusc. 1, 2, 5; Sen. Ep. 88, 26.—
2. An astrologer (post-Aug.): “mathematici, genus hominum potentibus infidum, sperantibus fallax, quod in civitate nostra et vetabitur semper et retinebitur,” Tac. H. 1, 22: “nota mathematicis genesis tua,” Juv. 14, 248; Tert. Apol. 43: “qui de salute principis ... mathematicos consulit, cum eo qui responderit, capite punitur,” Paul. Sent. 5, 21, 3.—
So I think that dictionary is saying that in "post-Aug." times the word is used for "an astrologer".
The same sentence (of Augustine) includes the word divinationem which Lewis and Short defines as,
dīvīnātĭo , ōnis, f. divino.
I. The faculty of foreseeing, predicting, divination, μαντική (cf.: “augurium, auspicium, vaticinium, praesagium, praedictio),” Cic. Div. 1, 1; 2, 5, 13; 2, 63, 130; id. N. D. 1, 22, 55; id. Leg. 2, 13, 32; id. Rosc. Am. 34, 96; Nep. Att. 9, 1; 16 fin.: “animi,” Cic. Fam. 3, 13: “mendax,” Vulg. Ezech. 13, 7.—
So that (context, including also the context of the previous sentences) supports the "astrology" sense of the term.
So instead of "beware the mathematician and all those who make empty prophecies", a more faithful translation might be, "beware the astrologer" (and presumably other people who practice augury).
My personal guess is that the practice of astrology might have looked like a branch of what we know as mathematics: including e.g. geometry to divide the zodiac into houses or whatever.
Looking at the (ancient) Greek the word derives from a root meaning "learning", so it means like "fond of learning", and can be used to mean "scientific" (especially "mathematical"); "astronomical"; and/or "astrological".
Can you find an expert confirmation that your interpretation is correct? Do historians or philosopher agree with your (somewhat arbitrary) choice of meaning?
Yes, for example this translation with footnotes talks about astrology w.r.t. that passage, e.g.:
Astrology recurs pastorally throughout Augustine's career. The same vocabulary, and the same scriptural quotations, appear over and over. The place of astrology in African life etc.