43

In their 2020 study termed "Relative Incidence of Office Visits and Cumulative Rates of Billed Diagnoses Along the Axis of Vaccination" [full text], appearing in the International Journal of Environmental Research and Public Health, Lyons-Weiler et al. analyze a dataset spanning ten years of pediatric practice.

Study Synopsis

In a nutshell, this dataset contains a total of 21.777 actual patients and includes vaccination history, office visits, and billed diagnosis.

First, the authors pool vaccinated and non-vaccinated patients. Second, the authors perform statistical analysis on the distribution of (a) office visits and (b) billed diagnosis among both groups.

Findings seem to indicate a drastic enrichment (>95% CI) of various diagnosed diseases within the vaccinated group, compared to the non-vaccinated one (see image below).

Based on those results, the study concludes:

We can conclude that the unvaccinated children in this practice are not, overall, less healthy than the vaccinated and that indeed the vaccinated children appear to be significantly less healthy than the unvaccinated

Thoughts

On the one hand, the authors include at least some statistical evaluations on the robustness of their results.

On the other hand, the article is lacking any hypothesis why similar effects can't be found with the same methods on other datasets.

Furthermore, many of their "disease" criteria are not actual diseases but a loose group of symptoms generally associated with a vague disease-like term that lacks proper definition, such as "behavioral issues".

In summary, those results seem credible at first sight, but also highly counter-intuitive.

Therefore, I am highly skeptical.

Questions

  1. Are vaccinated people significantly more susceptible to a wide range of diseases?
  2. Is the approach of this study following good scientific practice?
  3. Which follow-up analysis are adequate to verify/falsify their hypothesis?
  4. Are there any studies that directly support/contradict those findings on a larger dataset?

enter image description here

12
  • 9
    This page healthfeedback.org/claimreview/… has some good reasons to doubt. Including their usage of the (made up) RIOV metric, the authors (one of whom is now suspend). The references there also include many larger studies which directly contradict these “results”.
    – Tim
    May 9 at 9:20
  • 25
    This is essentially a duplicate of this one except replayed 6 months later on a different paper. It has the same holes that @Fizz describes - that there is a glaring confounding factor of the degree to which parents of the children seek paediatric advice. (Both also ignore the benefits that unvaccinated children get from herd immunity. Very few unvaccinated children get measles now, but that doesn't mean that vaccines are dangerous or ineffective.)
    – Oddthinking
    May 9 at 11:19
  • 1
    @Oddthinking You're arguing bravely against straw men. Herd immunity being such. The papers in this direction do not shoot at measles vaccine is dangerous or ineffective, but at 'long term interactions from countless shots towards seemingly completely unrelated effects/illnesses'. Different game. It supports the argument: jabbed kids may be safer from measles, but in addition to the side effects, the overall health outcome did not improve, it deteriorated. Whether one dies of measles, the shots or added to the latter from any other causes, 'those jabbed are in all worse off' is the essence. May 9 at 12:26
  • 5
    @LangLаngС: Thanks for calling me brave, but it isn't a strawman; "the unvaccinated children in this practice are not, overall, less healthy than the vaccinated". One possibility to explain this surprising result is that the large number of vaccinated children (and adults) are protecting the small number of unvaccinated children from expose to disease. The paper doesn't (to my admittedly brief reading) address this possibility.
    – Oddthinking
    May 9 at 12:32
  • 3
    @LangLangC: I would replace "common wisdom" with "the consilience of scientific evidence over decades", but yes. The longer form of the sentence is better handled by the other part of my comment - which Fizz's answer explores.
    – Oddthinking
    May 9 at 12:41
92

Almost all studies in this arena are correlative-based and not causation-based, so that I would take them all with a grain of salt. I would all suggest that Weiler and Thomas (2020) omit some important data, such as those that die from the disease from which they could have been vaccinated, and these are important data points. You raise some good points as well. I am concerned about using office visits as a measure; as those who never took their children to their PCP to get vaccinated (what many consider very important treatment) may be much less likely to take their child to the PCP for minor future ailments.

I am not going to spend too much time on if or not MDPI journals are reputable journals. Others have done this (search for MDPI at https://beallslist.net/). I will tell you they were on Beall’s predatory list for some time and are not well-regarded among scientists. Let us move on from the type of study (correlation) and quality of a journal (debatable) and look at what the accumulation of evidence suggests.

This BBC article (https://www.bbc.com/future/article/20200915-the-mystery-of-why-some-vaccines-are-doubly-beneficial) explains the current paradigm well. Scientists are unsure why some vaccines have such positive outcomes on health beyond what can be explained beyond the resistance to the disease they are protecting you from. Read this BBC piece and follow the links.

It was around a year after the vaccinations began that they made an extraordinary discovery: those who had been vaccinated against measles were 50% less likely to die than those who had not. "It was stunning," says Aaby – but not for the reasons you might at first think. *Today Aaby to the BBC, 2020)

Please review these three articles in Science, Frontiers, and Lancet (no question about the validity of these three journals), and you will get a very different picture. All three are far more robust articles than Weiler and Thomas (2020) and represent the currently accepted thinking.

Weiler and Thomas (2020) are one article in one very low-ranked journal, their research is worthy of more follow-up, but you will see the current scientific evidence does not suggest they are correct. Indeed the current paradigm based on solid evidence opposes this idea.

It is good to challenge existing ideas with new data, and Weiler and Thomas (2020) should certainly not be discarded, but that paper will not move the scientific evidence on this topic one millimeter. The current evidence is that vaccines not only protect against the disease for which they are targeted but they appear to have positive health-outcomes far beyond the disease targeted as well.

12
  • 89
    "those who never took their children to their PCP to get vaccinated (what many consider very important treatment) may be much less likely to take their child to the PCP for minor future ailments" I feel like this is a more likely cause. At the risk of generalization, anti-vax parents are more likely to not trust "establishment" doctors and are more likely to not bring kids to standard PCPs and instead to either holistic healers, homeopaths, etc
    – DenisS
    May 9 at 14:18
  • 29
    Another point in favor of this is ADHD visits being identically zero for unvaccinated patients. Unless vaccines are the sole cause of ADHD, this suggests parents of unvaccinated children are not seeking diagnosis for that disorder. May 9 at 15:11
  • 10
    I challenge calling the Lancet a trustworthy journal with respect to vaccines. They were the ones publishing the "vaccine causes autism" study and it took them 12 years to retract that: ncbi.nlm.nih.gov/pmc/articles/PMC2831678 So maybe its safer to wait until 2032 before accepting a vaccine related paper published in the Lancet in 2020.
    – laolux
    May 10 at 1:16
  • 13
    @DenisS And note that many pediatricians won't see unvaccinated children in the first place, cutting down on the number of visits. May 10 at 2:16
  • 8
    @EarlGrey "Validity of journal" is not an argument from authority fallacy, it is an assessment of the prior on source quality.
    – jakebeal
    May 11 at 8:57
43

David Gorski has some (updated) comments about the methodology of the study such as:

The authors claim that RIOV “reflects the total number of billed office visits per condition per group, reflecting the total disease burden on the group and the population that it represents,” but no good analysis or references are provided to show that RIOV does, in fact, correlate with disease burden, particularly when using billing data.

[...] because it is billing records being analyzed, it is impossible to know how many emergency room visits there were in each group or how many unbilled telephone calls were received from parents of children in each group. At least, these measures cannot be determined without a chart review, which appears not to have been done.

And because the data comes from the 2nd author's own clinic...

there’s likely to be ascertainment bias, which is the systematic distortion in measuring the true frequency of a phenomenon due to the way in which the data are collected. How could this happen? Think about it. Dr. Thomas believes that vaccines cause autism. That right there introduces unconscious bias that could affect how likely he and his staff are to investigate subtle signs of autism and refer out to for evaluation based on vaccination status and how likely he is to ascribe various diagnoses to “unvaxxed” children compared to “vaxxed” children. One can easily imagine this bias leading to unvaccinated children to be less likely to be given an autism diagnosis than vaccinated children or to be—dare I say?—brought in to the office as often for various conditions that Dr. Thomas attributes to vaccines.

Gorski also has some comments about the official Oregon Medical Board allegations against study co-author Thomas, regarding the practice from which the data apparently came:

It turns out though, that if the following allegations recounted in the Board’s order are true, Dr. Thomas is not so respectful of parents who are pro-vaccine and want the standard vaccine schedule:

3.2 Licensee is insistent and direct in his communication with parents and guardians that they should accept his alternative vaccine schedule.

What does the Board mean by that? It provides two examples:

3.2.1 A patient’s mother sought subsequent treatment by Provider X after having been “reduced to tears” by Licensee’s “bullying” her into his personal vaccine schedule against her express wishes for full vaccination for her child.
3.2.2 Patient A’s mother requested polio and rotavirus vaccinations for Patient A according to CDC Recommendations, but Licensee did not have those vaccines in the clinic, and Patient A would therefore not be able to get them. Patient A’s mother reported that the Licensee questioned why she wanted Patient A to get the polio vaccine and asked whether they were traveling to Africa. During the appointment, Licensee continually connected vaccines (not specific) with autism. Licensee asked her how awful she would feel if Patient A got autism and she could have prevented it.

One cannot really have much confidence about the data gathered in such a setting...

And one point I've noticed in the paper itself is that it never seems to mention the word "hospital". In other words, it seems that all the billing hours are for the clinic itself, the rest are basically a zero-cost externality from the point of view of this study. Which brings us for instance to how this case might have been handled in terms of cost:

3.3.3 Patient D, a now 9-year-old male, was completely non-immunized. Patient D sustained a large, deep scalp laceration at home in a farm setting on August 8, 2017, and was treated with colloidal silver and with his parents suturing the wound independently. Patient D subsequently developed acute tetanus requiring intubation, tracheotomy, feeding tube placement and an almost two- month ICU stay at Doernbecher Children’s Hospital. Patient D was then transferred to Legacy Rehabilitation. Licensee saw Patient D for follow-up in clinic on November 17, 2017. Licensee’s notes documented a referral to a homeopath, recommendation of fish oil supplements, and “phosphatidyl seine.”

I mean, sure, there weren't many hours billed at the clinic itself, despite the two-month stay in the ICU at a hospital. So low RIOV in this case, no doubt.

The first author of this study, Lyons-Weiler, is also (more) mainstream-media famous for making various unsound claims, the most recent ones about Covid vaccines exaggerating the risks: "Lyons-Weiler’s number is more than 40 times too high" (according to Politifact.) Since we have some evidence how he presented/interpreted known data (on Covid-19 vaccines), one can only wonder how he might have processed data that is in itself hard to verify, such as that coming from Thomas' practice. (Earlier in the present pandemic, Lyons-Weiler also made unsound claims about the genetics of the Covid-19 virus itself.) His qualifications are discussed in vaccine court proceedings in which he was introduced as an expert witness on PANDAS, but the court largely rejected his credentials as irrelevant; there is some news coverage of that as well.

2
  • 4
    While the majority of the answer is completely valid and correct (+1), I'm not wholly convinced by the "ascertainment bias" argument. One can say the same about any doctor who undertakes studies based on results from their own medical practice, which doesn't in itself render the study inaccurate. Unconscious bias exists, but its effect is likely negligible in light of the other glaring flaws in the study.
    – March Ho
    May 10 at 1:25
  • 2
    @MarchHo: I'm not sure it's that far of the mark in this case. Look at the study's ADHD graph. It's a completely flat line for the unvaccinated. That's precisely the kind of disorder that takes (1) the parents [or school] to consider it and report some symptoms to the healthcare provider and (2) the latter to take it them into account. Yeah, it's probably more likely that the main source of bias is in (1) rather than (2)... but only if the parents are the only source of reports.
    – Fizz
    May 10 at 12:19
26

It is worth noting that almost all of the conditions listed in the chart, unless they are unusually severe, do not need professional medical intervention. Skin rashes, coughs, minor to moderate asthma, many eye and ear infections etc. either go away on their own, can be treated with over-the-counter medicine or, if they become chronic, can be manged with simple adaptation. Additionally, chronic conditions like asthma or skin issues can often not be healed but are rather managed and treated symptomatically, so a doctor visit is less valuable there, too.

It is likely that parents who for whatever reason — in the U.S., the first reason coming to mind is being uninsured — don't get their child vaccinated also don't visit the doctor for anything that doesn't really need treatment.

One would wish to compare the number of emergency room visits and, even more significant, the mortality of the two groups. The global effect in particular of Tetanus and Measles vaccination on mortality is undoubtedly beneficial (https://academic.oup.com/aje/article/182/9/791/96333), but the average global living conditions are very different from the ones in the U.S so that the result is not directly applicable.

3
  • 2
    I think you nailed it Peter - in summary, the study shows how likely someone is to seek/receive/afford medical treatment given that they have previously had a vaccination. May 11 at 11:03
  • 3
    It would be interesting to see prior and post vaccination data. If your hypothesis is correct we would expect to see increased hospital visit rates prior and post vaccination in vaccinated individuals, compared to non-vaccinated individuals. May 11 at 11:10
  • 1
    @LittleBobbyTables That is a very good idea. It would be nice to compare it to the non-vaccinated control group (to protect against unrelated effects like a bad flu season that would effect everybody equally) but one would have to invent the individual "dates of non-vaccination" for the unvaccinated with a distribution matching the distribution in the vaccinated group. May 11 at 11:22
2

A quick glimpse on the article reveals that their study didn't have a control group, that is, children who received a saline shot and were told to be vaccinated. A difference in disease rate between such children and the ones who have truly been vaccinated would indicate an actual effect from the vaccine. That's how COVID vaccines are tested for instance.

Without a control group, you will inevitably find that sugar pills which are labelled as drugs are effective against every disease because of the placebo effect. This works for vaccines too, but in the opposite way.

In a similar way, researchers should not know whether the patient is in the control group or not until the end of the study, otherwise their expectations could affect the result in a way similar to placebo effect, as in this homeopathy study. It's not clear for me whether blinding was done or not in the paper in question.

PS. The claim that a double blind study of vaccines against dangerous deceases could not be performed because such studies would be unethical is not true. Such studies were and are done. I already mentioned COVID vaccines. Another example is the classical 1986 Finnish MMR trial on twins. For the reference, the MMR vaccine was licensed 15 years before the study.

The Finnish study found that the true rate of side effects attributed to the vaccine was statistically insignificant, and some of the side effects attributed to the vaccine were actually more often present in the placebo-injected group. Perhaps that's the actual reason that anti-vaccination articles don't mention such high quality papers (large RCT, level I in USPSTF hierarchy of evidence) and instead rely on a methodologically weak study (historical cohort study, ISPSTF level II-3) that "prove" their point.

15
  • 4
    Technically true but I need to give the authors the benefit of the doubt here. I assume that fake-vaccinations in order to have a negative control are ethically somewhat challenging :)
    – sudonym
    May 11 at 9:23
  • 4
    @sudonym they would be blatantly unethical, you cannot perform this kind of study for proven vaccines against dangerous diseases. There are many things to criticize in the study, this is not one of them. There are simply cases where you cannot perform randomized double-blind studies with placebos.
    – Mad Scientist
    May 11 at 13:37
  • 2
    @DmitryGrigoryev if treatments are available, any new drug is tested compared to the current standard of care. So for a cancer with established treatments, it would certainly be unethical to withhold that treatment from the control group. The first vaccine for a disease will be compared to a placebo, but if there is an established vaccine available you would always compare a new vaccine with the old one, not the new one with a placebo.
    – Mad Scientist
    May 11 at 16:07
  • 2
    @DmitryGrigoryev Other study types are acceptable, and often the only way to study certain issues. Those studies of course have more confounding factors and are harder to get right, that doesn't make them invalid. I'm not defending this particular study, it is deeply flawed, but not in the way you answer here.
    – Mad Scientist
    May 11 at 16:21
  • 2
    Your description of the placebo effect is unreferenced and oversimplified to the point of being misleading.
    – Oddthinking
    May 12 at 0:20
0

TL;DR: This study doesn't provide any information worth using to answer the question so it should be just ignored. Considering other available information, it seems plausible to expect vaccinated children to be more healthy overall.

Long answer:

As with every study that tries to extrapolate generic results from statistics, the important questions are

  1. Is the sample size big enough?
  2. Is the sample being studied really a random selection from the whole population?

For this study, it seems clear that 1 is true. For the 2, I think the selection is far from random. Instead, we are actually studying if there's correlation with following groups:

  1. People that vaccinate their children.
  2. People that visit office for even with minor evidence of any disease.

It seems highly probable that there's correlation between these groups.

The correct way to implement study like this is to select required sample size, randomize the test subjects from the whole population and then investigating if they have been vaccinated and if they suffer from symptoms of the tested diseases. This is because people that avoid visiting the office, will not have their diseases diagnosed as often, even if they suffer from the symptoms. Implementing such study is obviously much harder than the hack job done for this study, which is probably the reason why such proper study wasn't done.

As such, you should just ignore this "research" and look for additional information if you want scientifical results.

1
  • 1
    Whilst you are correct, this is more of an unreferenced comment on the methodology than a specific answer to the question. Please take our tour and refer to the help center for guidance as to our ways. Welcome to skeptics. May 12 at 11:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .