According to the United States Holocaust Memorial Museum, itself a major scholarly archive of primary materials:
Calculating the numbers of individuals who were killed as the result of Nazi polices is a difficult task. There is no single wartime document created by Nazi officials that spells out how many people were killed in the Holocaust or World War II.
Complicating matters further is the question of definitions. In some sense, the very definition of the Holocaust relates to the fate of the European Jews. According to the Oxford English Dictionary, Holocaust means "the mass murder of the Jews by the Nazis in the war of 1939–1945." However, it can also be used as a transferred attribute, "of the similar fate of other groups," according to Oxford English.
The United States Holocaust Memorial Museum is therefore careful in discussing numbers related to "the victims of the Holocaust and Nazi persecution." The site lists the following figures:
Jews: up to 6 million
Soviet civilians: around 7 million (including 1.3 Soviet Jewish civilians, who are included in the 6 million figure for Jews)
Soviet prisoners of war: around 3 million (including about 50,000 Jewish soldiers)
Non-Jewish Polish civilians: around 1.8 million (including between 50,000 and 100,000 members of the Polish elites)
Serb civilians (on the territory of Croatia, Bosnia and Herzegovina): 312,000
People with disabilities living in institutions: up to 250,000
Roma (Gypsies): 196,000–220,000
Jehovah's Witnesses: Around 1,900
Repeat criminal offenders and so-called asocials: at least 70,000
German political opponents and resistance activists in Axis-occupied territory: undetermined
Homosexuals: hundreds, possibly thousands (possibly also counted in part under the 70,000 repeat criminal offenders and so-called asocials noted above)
We can conclude the argument is not about the exact numbers, which are widely available (and amount to more than five million total), but about definitions. A greater care to address the specific categories of victims involved can resolve the ambiguity in the original quote.
Consult the Holocaust Encyclopedia for more information.