Ironically, the answer itself lies in anti-semitism.
"Semite" itself does indeed come from "Shem"; according to the Encyclopedia of Christian Civilization, "August Ludwig von Schlözer is generally credited with the origin of the name Semite in 1781. Von Schlözer derived the term from Shem, the eldest son of Noah (Gen. 5:32; 6:10; 9:18; 10:1; 1 Chron. 1:4), to describe a family of related languages." So it's not literally intended to suggest that everyone "semitic" is descended from Shem; the use of the name is just a cutesy (by academic standards) way to group the language family together.
Interestingly, the term itself became more anti-semitic the more connected to Judaism it became.
"Semitic" itself initially referred to the linguistic family, rather than to any one group of people (other than perhaps the peoples who spoke semitic languages). But soon enough it would make the jump to specifically referring to Judaism.
In the 1840s, Ernest Renan, an avid French philosopher and linguist, began studying Semitic languages. In 1847, he obtained the Volney prize for the manuscript of his "General History of Semitic Languages." His writings still grouped people together in linguistic groups, but had a strongly racist focus: he would blur the lines between language and race, referring frequently to the supposed superiority of the "Indo-European race" over the "Semitic race".
(To give you some idea of his thoughts: he would later combine his studies of theology and philology in his book "The Life of Jesus," published in 1863, in which he "claimed Jesus was able to purify himself of Jewish traits and that Jesus became an Aryan." He claimed, "I am therefore the first to recognize that the Semitic race, compared to the Indo-European race, truly represents an inferior combination of human nature."
He was certainly not the first to claim these anti-semitic views. But you can see how smoothly the transition was made from "semitic languages" to "Jews are bad, mmkay?" in these very anti-semitic times.)
The term at the time for Renan's feelings, in German, was not "anti-semitic", it was the extremely straightforward "Jew-hatred" ("Judenhass"). But it was actually his own racist views that would lead to the invention of the term we now use.
"Anti-semitic" was first used in 1860 by German Jewish scholar Moritz Steinschneider, and didn't solely mean "opposition to Jewishness" - it was "a criticism of Ernest Renan's contrast of Semites and Indo-Europeans." In other words, it still referred to the entire family of people using Semitic languages, although there was certainly no pretense that that didn't include the Jewish community.
The term comes from the German. Steinschneider coined it as antisemitisch, but nineteen years later, proto-Nazi Wilhelm Marr would coin it as antisemitismus -- essentially the same word. Marr needed a word that was more divorced from its visceral racism than "Jew-hatred". Wiktionary's page on anti-Semitism notes that he used the term "to make hatred of the Jews seem rational and sanctioned by scientific knowledge."
TL;DR: These terms have nothing to do with thinking people are descended from Shem. And everything to do with assembling faux-scientific and faux-historical "evidence" for anti-semitism.