54

According to Unherd's summary of a CMU study

[Title:] The most vaccine-hesitant group of all? PhDs

[...] more surprising is the breakdown in vaccine hesitancy by level of education. It finds that the association between hesitancy and education level follows a U-shaped curve with the highest hesitancy among those least and most educated. People with a master’s degree had the least hesitancy, and the highest hesitancy was among those holding a Ph.D.

enter image description here

The National Review also reproduced the graph above, with even fewer details.

I suspect the study only concerned Covid-19 vaccines, but that's not too clear in Unherd's take. So is this a true relationship in general (between PhDs and vaccines in the US), or particular to one specific period and vaccine?

As a "sanity check" I looked for surveys inside universities, and found one, which doesn't quite match those findings above that supposedly was using a nation-wide representative sample. In this Wayne State survey, graduate students and post-docs had less hesitancy than undergraduates, and faculty had even less than both:

enter image description here

Granted industry-working PhDs would not be capture in the latter. There's also the issue that a university faculty is substantially older than students.

Improve this question
7
  • And what about university staff vs ordinary working folk, as a dividing factor.
    – Daniel R Hicks
    Sep 27, 2021 at 15:41
  • 2
    Link to the CMU study for convenience medrxiv.org/content/10.1101/2021.07.20.21260795v1.full.pdf
    – Rob Watts
    Sep 27, 2021 at 15:45
  • 31
    PhDs are 2% of the sample, and the education level is self-reported, so this is probably one of the least reliable data points in that study. The total number is still high at almost 11000, but I really don't know how well this kind of Facebook survey works.
    – Mad Scientist ♦
    Sep 27, 2021 at 19:28
  • 1
    There are many types of PhDs. I would be more interested in the results for (1) verified possessors of PhDs (2) educated in fields related to biology or epidemiology, than I would be in the results for the History of English Literature or Music Theory.
    – Technophile
    Sep 30, 2021 at 17:25
  • 1
    @MadScientist who with a PhD is on Facebook AND has time to answer questionaires?^^ Okay okay, they exist and I'm a mean prejudiced weirdo... still... I could not imagine any of my academic friends to be there and spend the time. ...buuut I'm also not US-based...^^
    – Frank Hopkins
    Sep 30, 2021 at 23:30

4 Answers 4

Reset to default
102

The graph accurately represents the survey result but the survey cannot be taken as an accurate representation of the true position.

If you follow the link to the paper that Unherd has you'll see the following data on page 17:

Chart Headers

Hesitancy by education level

The fourth column is "COVID-19 vaccine hesitant % (95% CI)".

The data itself came from an online survey via Facebook:

This analysis used the COVID Trends and Impact Survey (CTIS)9, created by the Delphi Group at Carnegie Mellon University (CMU) and conducted in collaboration with Facebook Data for Good.

It appears they've used a subset of the data for their paper (as a side note the January 6th start date does coincide with an update made to the survey):

The analysis sample includes 5,121,436 survey responses from participants who completed the survey at least once January 6 to May 31, 2021

The survey itself does have some published limitations. Looking through those, there are two caveats I find noteworthy. The first is the second sample limitation:

Non-response bias. Only a small fraction of invited users choose to take the survey when they are invited. If their decision on whether to take the survey is random, this is not a problem. However, their decision to take the survey may be correlated with other factors—such as their level of concern about COVID-19 or their trust of academic researchers. If that is the case, the sample will disproportionately contain people with certain attitudes and beliefs. This implies the potential for the sample to be unrepresentative of the US population as all self-selected surveys inevitable are.

The second caveat is from the section about response behavior:

While less than 1% of respondents opt to self-describe their own gender, a large percentage of respondents who do choose that option provide a description that is actually a protest against the question or the survey; for example, making trans-phobic comments or reporting their gender identification as “Attack Helicopter”. Additionally, these respondents disproportionately select specific demographic groups, such as having a PhD, being over age 75, and being Hispanic, all at rates far exceeding their overall presence in the US population, suggesting that people who want to disrupt the survey also pick on specific groups to troll.

The first caveat indicates an unknown level of reliability about the views of the population, while the second indicates that the responses are clearly not representative. Because PhDs make up around 2% of the sample size, having even "less than 1% of respondents" attempt to disrupt the data for PhDs means we cannot be confident about the accuracy of such data as representing the population. As such we need to take the data for PhDs with a major grain of salt. It could still be useful to look at the data for PhDs over time, but any comparison with other groups cannot be trusted without further studies being done that reach the same conclusions.

A truly trustworthy analysis would need to randomly sample the population as self-selecting surveys can be very, very unrepresentative.

Improve this answer
22
  • 50
    Seems irresponsible that the troll responses weren't thrown out entirely.
    – R.. GitHub STOP HELPING ICE
    Sep 28, 2021 at 2:17
  • 63
    A clearer claim is "there is some correlation between self-reported education level and vaccine hesitancy". The confounder of "vaccine hesitancy is correlated to false self-reporting of PhDs" is explicitly stated by the data.
    – obscurans
    Sep 28, 2021 at 2:24
  • 38
    If you get replies from "Attack Helicopters" I suspect the entire survey should be thrown out, because there's no way of knowing which replies are serious
    – MrSparkly
    Sep 28, 2021 at 2:51
  • 48
    So they may have been p-hacking (why would anyone choose a dataset start date of the 6th of a month?), some respondents were known to specifically input fake data at a very high rate and the respondents are biased towards the conclusion they got? Any one of those, especially the first two, would be enough reason to disregard the results completely. Rather than saying there's an "unknown level of inaccuracy" or "we cannot be confident", I'd probably say the data is fraught with issues and a proper study is needed to even think that that conclusion has a reasonable chance of being valid.
    – NotThatGuy
    Sep 28, 2021 at 3:08
  • 41
    It strikes me as quite careless to not put a large asterisk on the PhD column, with a note saying that that particular estimate is unreliable due to deliberate trolling. Alternately, it would be possible to not include the PhD numbers in the figure (while still mentioning it in the paper, obviously): an effect size being smaller than "random variation" is sufficient reason to not include it in a summary, and even though this variation is clearly non-random, a similar principle still holds. Particularly with such a sensitive issue, one that is vulnerable to context-free dissemination.
    – Obie 2.0
    Sep 28, 2021 at 7:07
40

The survey has more than 98% non-response. What we're seeing here is selection bias: people who (claim to) have a PhD and are FaceBook users and responded to this survey are not representative of PhDs in general.

Taking the survey is voluntary, and only 1-2% of those users who are invited actually take the survey.

(This fact is mentioned in the middle of Rob Watts's answer. But I think it should be the headline. You shouldn't take a survey seriously with such a low quality sample.)

For easy reference, I'll repeat the links given in Rob Watts's answer: limitations of the survey and response behaviour.

Improve this answer
10
  • 5
    @Fizz doesn't matter, it's still a sample size that's self selected AND very small. You might as well ask 100 people working at vaccination sites and show that 100% of medical staff are vaccinated.
    – jwenting
    Sep 29, 2021 at 7:35
  • 4
    @jwenting The sample size is plenty large enough - a proportion derived from a population of 10,000 is highly likely to be accurate within 1%. If 100 of 100 sampled individuals are vaccinated, that's sufficient evidence to show that at least 95% of that population is vaccinated. The problem here isn't sample size, it's that the sample is not representative of the population. It doesn't matter how big of a population they sampled, since the sampling methodology doesn't actually sample the population they're drawing conclusions about. The study would be no better with 100M respondents.
    – Nuclear Hoagie
    Sep 29, 2021 at 13:37
  • 2
    @Fizz The covariates included in the RR adjustment included "demographics, geographic factors, political/COVID-19 environment, health status, beliefs and behaviors" - unclear how any of that relates to my previous comment about sample size or representativeness, though. Covariate adjustment is pretty normal and isn't necessarily indicative of methodological issues, those can be identified from the methodology alone. You don't need to look at any data at all to be suspicious of the representativeness of a self-reported Facebook survey, no matter how large it is.
    – Nuclear Hoagie
    Sep 29, 2021 at 15:59
  • 2
    @jwenting That's exactly my point - it's an issue of representativeness, not sample size. If you actually drew 100 individuals randomly from the population of all healthcare workers everywhere, it would sufficient to estimate the true vaccination rate within a few percent. But since "people working at vaccination sites" is not a random sample of "all healthcare workers", it doesn't matter how many of them you sample - you'll never properly estimate the vaccination rate among all healthcare workers by sampling vaccination site workers. Adding more people to the sample won't help.
    – Nuclear Hoagie
    Oct 1, 2021 at 12:52
  • 2
    @jwenting The problem is with the sampling methodology, not the sample size. Had they collected a true random sample from the population they're making claims about, a sample size of >10,000 would be plenty to make very precise estimates of the true vaccination rate to within 1%. The sample is not "very small". It's a very large sample, but of an unknown population that is not actually made up of people who truly hold PhDs.
    – Nuclear Hoagie
    Oct 1, 2021 at 13:02
22

I'll add here that while the data is in the paper, the bar graph from UnHerd is not. The paper has this line graph instead:

enter image description here

Indeed in May 2021 their data points to PhDs having the highest hesitancy, subject to the limitations of their study. (From Jan to April however, the "high school or less" topped the chart or at least tied the PhD line.)

Also, one of the paper's authors did talk about that to the press a bit later:

But some of their work appears to be misrepresented online, missing the overall point that hesitancy dropped.

“There are people that can kind of take a data point and twist it around to mean something that it doesn’t mean, and that’s unfortunate,” King said.

A sensitivity analysis found some people answered in the extreme ends of some demographic categories to throw off some of the numbers. King said it appeared to be a “concerted effort” that “did make the hesitancy prevalence in the Ph.D. group look higher than it really is.”

For example, they observed higher hesitancy rates than expected in the oldest age group — 75 and over — as well as the top end in terms of education level.

“We found that people basically used it to write in political … statements,” King said. “So they weren’t genuine responses. They didn’t really complete the survey in good faith.” [...]

People taking the survey were on the honor system, with no way to make sure people who claimed to have Ph.D. degrees actually have them.

And the Ph.D. group does not include medical doctors or nurses.

“So it’s not representative of the medical profession,” King said.

Regarding the age issue mentioned in the quote, there is indeed this odd data point in the paper's charts where 75+ y.o. Hispanics have much, much higher hesitancy than either Whites or Blacks of the same age...

enter image description here

Improve this answer
13
  • 38
    The authors seem absurdly attached to their data despite its dubious connection to any reality.
    – jeffronicus
    Sep 28, 2021 at 14:50
  • 3
    It may be useful to note that tracking changes over time is the first recommended use of the data in its survey limitations info. Given the known inaccuracies in the data, it makes sense that they'd focus on the overall trend rather than on comparing the groups.
    – Rob Watts
    Sep 28, 2021 at 17:21
  • 23
    Wow - as few as negative 5% of Pacific Islanders age 18-24 may be vaccine hesitant
    – Tim
    Sep 28, 2021 at 20:23
  • 16
    ... that Hispanic + 75+ age. This alone is enough to basically throw out the entire survey as worthless. 10+stdev away from the adjacent age group? You've literally see "attack helicopters"? Reporting the numbers at all is irresponsible.
    – obscurans
    Sep 29, 2021 at 7:22
  • 9
    @obscurans - The number of people self-reporting their gender at all (let alone putting in anti-transgender troll responses) was less than 1% of respondents. I don't think a responsible researcher would throw out an entire survey based on that alone. It is slightly more problematic for the study's validity that Hispanic people over 75 are such an outlier, since they should represent about 5% of a random selection. But the most troublesome factor is an incredibly high non-response rate, over 98%. Without careful analysis, there is no reason to assume that participants are representative.
    – Obie 2.0
    Sep 29, 2021 at 14:43
-4

The question as given in the titled is broader in its inferences than the evidence presented in the question body. Indeed, it gets quite fuzzy when the broad statement in the title rests on a self-selection bias prone survey, and it doesn't help either when most people do not stop inferring and take the character combination "PhD" to halo-effect mean "intelligent people".

The gist of what's transported in the question title:

Q: Are people with a PhD least likely to be vaccinated in the US?

however is corroborated in other studies:

COVID-19 Vaccine Hesitancy Among Medical Students

Results: A total of 58.2% of medical students reported vaccine hesitancy. The most common reasons for this were worrying about the side effects of vaccines (44.4%), uncertainty about vaccine safety (40.4%), and underestimating the risk of exposure to COVID-19 (27.9%). The main factors associated with COVID-19 vaccine hesitancy among participants were their knowledge about COVID-19 vaccine, training related to COVID-19 vaccines, family address, and education level (P < 0.05).

— Gao X, Li H, He W, Zeng W.: "COVID-19 vaccine hesitancy among medical students: The next COVID-19 challenge in Wuhan, China", Disaster Medicine and Public Health Preparedness, Published online 2021 Sep 9. pubmed, doi

To summarise that adequately, we see that those who say "no, thanks for nothing" — among those trained in medicine — look out for 'vaccine' safety, take the all too common side effects seriously, and are those who have knowledge about COVID-19 vaccines, training related to COVID-19 vaccines, and again: education level. And that's a match.

Improve this answer
3
  • 2
    There's a few issues here: (a) "hesitant" ≠ "unvaccinated"; (b) China ≠ USA (note the reason "I do not need vaccines because the COVID-19 is no longer common here"); (c) med students ≠ PhDs; (d) the paper is behind a paywall so I can't be sure, but I expect the participants are reporting in regards to their third dose; (e) is 58.2% hesitancy high or low (vs. general population)?
    – Rebecca J. Stones
    Nov 20, 2021 at 0:14
  • A non-pay walled version is available: ncbi.nlm.nih.gov/pmc/articles/PMC8564029 Regarding d: “medical students completed our questionnaire from February to March 2021. The government of Wuhan has been providing COVID-19 vaccines for college students since April 2021“
    – FifthArrow
    Nov 20, 2021 at 0:18
  • 3
    I believe you have misinterpreted this study (Table 3 does seem rather confusing.) Table 2 shows LOWER vaccine hesitancy in postgraduates than undergraduates, LOWER vaccine hesitancy in students with training in COVID-19 vaccines. It shows that students that score better in a test about COVID-19 had lower vaccine hesitancy. It doesn't show that medical students are more hesitant than the general population.
    – Oddthinking ♦
    Nov 20, 2021 at 12:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .