Did the Cass Review ignore studies that were not double-blinded?

Question

The Cass Review examined gender services for children and young people in the UK, and made recommendations to the country's health services. It has been controversial.

The Feminist Gender Equality Network (which works to fight transphobia) wrote a letter/petition that claims:

It does not include a proper systematic literature review since it disregards most research evidence because it fails to reach the impossibly high bar of a double-blind trial. In fact, trans healthcare is inappropriate for a double-blind trial since you cannot give hormones to one group of people and not to another. This practice would be unethical. Indeed, much medical practice is not based on this kind of evidence.

The Quackometer blog published some articles defending the report against claims like this.

Myth 3: Cass is demanding only Double Blind Randomised Controlled Trials be used as evidence in “Trans Healthcare”.

While it is acknowledged that conducting double-blind randomized controlled trials (DBRCT) for puberty blockers in children would present significant ethical and practical challenges, the Cass Review does not advocate solely for the use of DBRCT trials in making treatment recommendations, nor does it mandate that future trials adhere strictly to such protocols. Rather, the review extensively discusses the necessity for appropriate trial designs that are both ethical and practical, emphasizing the importance of maintaining high methodological quality.

We can see from the report and papers that Cass did not insist that only randomised controlled trials were used to assess the evidence. The York team that conducted the analyses chose a method to asses the quality of studies called the Newcastle Ottawa Scale. This is a method best suited for non RCT trials. Cass has selected an assessment method best suited for the nature of the available evidence rather than taken a dogmatic approach on the need for DBRCTs. [...]

Did the Cass Review ignore all trans health-care studies that were not double-blinded?

Answer 1

Did the Cass Review ignore studies that were not double-blinded [randomised controlled trials]?

No. In fact none of the reviewed studies were randomised controlled trials (RCTs). The researchers did not require RCTs, they looked for some means of following groups over time, comparing groups and controlling for differences.

The Cass Review commissioned the University of York to conduct a number of 'systematic reviews' (page 54, Explanatory Box 2).

• The highest form of evidence is that generated by a systematic review (Figure 5). A systematic review is different from a general review article. It is a summary of the literature on a particular question that uses explicitly defined and reproducible methods to systematically search, critically appraise, and synthesise primary research information (Cochrane, 2016; NIHRtv, 2010). It is designed to be reproducible, reliable and to eliminate bias.
• Standardised quality assessment tools or questionnaires are available for assessing different types of studies. This ensures that, as far as is possible, different people appraising a paper will come to similar conclusions.

They categorised each study as low, medium or high quality according to "a modified version of the Newcastle-Ottawa Scale" for the review topic. The Newcastle-Ottawa Scale "was developed to assess the quality of nonrandomised studies."

For the area of "Masculinising/feminising hormones", 53 studies "met the inclusion criteria" (Cass Review page 182). None were randomised controlled trials. They were cohort, pre-post or cross-sectional. Only one was categorised as high quality, 35 were considered low quality.

The studies had many methodological problems including the selective inclusion of patients, lack of representativeness of the population, and in many of the studies there were no comparison groups. Where there was a comparison group, most studies did not control for key differences between groups

For the area of "Understanding intended benefits and risks of puberty blockers", 50 studies were reviewed (Cass Review page 175) - again, none were randomised controlled trials. They were cohort, pre/post or cross-sectional. One was considered high quality and 24 as low quality.

The 'low quality' studies were not "ignored" - they were evaluated and found to be so lacking that they were not used to inform the 'synthesis' of the review. Cass Review page 56:

The purpose of synthesising the data is to combine multiple different studies to get an overall impression of the strength of the evidence; for example, in favour or against a particular intervention. To do this, the reviewers need to assess the quality of the studies in terms of recruitment, bias, design, analysis and all the other factors described above. They will only include studies that meet a quality standard. Provided that those studies have used similar measures and outcomes, their outcomes can be combined (synthesised) across a much larger sample of participants.

Page 50:

There are hardly any RCTs in children and young people receiving endocrine treatment for gender incongruence/dysphoria, but the following are the most commonly reported types of studies:

• Two types of studies that are sometimes used to look at the effects of puberty blockers and masculinising/feminising hormones are cohort studies and cross-sectional studies, which are different ways of looking at outcomes in groups who did or didn’t get a particular intervention. These are all called observational studies whereas RCTs are called experimental studies. This is because in cohort or cross-sectional studies the researcher did not allocate which patients receive an intervention. There may still be a comparison group, but participants will not have been randomly allocated to the two groups.
• The most common study for patients receiving puberty suppression is a pre-post study. This is where study participants are assessed before and after they receive an intervention. Because there is no comparison group of individuals who did not receive the treatment, and because one cannot rule out changes that would have occurred over time without treatment, it is not possible to draw strong conclusions from these studies.

Cohort studies make observations at different times during a time period (ideally a number of years, in this context). Cross-sectional studies make observations at a particular time. Pre-post aka before-and-after studies make observations before and after an 'intervention' e.g. treatment.

Myth 3: Cass is demanding only Double Blind Randomised Controlled Trials be used as evidence in “Trans Healthcare”.

Correct, it is a myth. Neither Hillary Cass nor the Cass Review have made such a demand or recommendation.

The Review says:

It is not always possible for people to be blind to a treatment; for example, in a trial of acupuncture versus physiotherapy, patients will know which treatment they are receiving.

Speaking to the BBC's More or Less podcast, Hillary Cass said that "obviously" young people cannot be blinded as to whether they are on puberty blockers or hormone blockers as opposed to placebo, because "it rapidly becomes obvious to them".

Cass also said:

"... the biggest weakness of the studies was not that they were not randomised-controlled trials, the biggest weakness was just that they did not follow young people through for long enough for us to really understand the longterm outcomes."