3

I recently came across this 2013 study: High school graduation rates among children of same-sex households which looks at a large sample of Canadian families, and concludes:

Children living with gay and lesbian families in 2006 were about 65 % as likely to graduate compared to children living in opposite sex marriage families. Daughters of same-sex parents do considerably worse than sons.

This, the abstract admits, is in contradiction to almost all previous studies of same-sex parenting.

I read claims that the difference found in this study was not significant.

Is this study reliable?

10
  • 1
    ... do the column headings (1), (2) and (3) show the number of parents the child has? Aug 7, 2023 at 22:32
  • @WeatherVane: "The different columns result from different types of controls. Column (1) includes controls for child characteristics, and these include: province, visible minority, disabled, mobility, urban, age, family size, family income, female, and same race. Column (2) adds the parental education controls: did the mother/father graduate from high school, and did the mother/father have a graduate degree. Column (3) adds the parental marital status variables found in Table 2." Aug 7, 2023 at 23:38
  • "I read in many publications" is not a notable source. The question should include explicit quotations and citations of this claim. Aug 8, 2023 at 0:07
  • 2
    @RayButterworth: I have included the core claim from the paper, so we do have a notable claim.
    – Oddthinking
    Aug 8, 2023 at 3:00
  • 1
    @Oddthinking, I think the table shows the sensitivity of the result to various other factors, and shows that those factors don't have a significant effect on the result. I suspect that any claims that the overall result isn't significant are based on a misinterpretation of this table. Aug 8, 2023 at 3:06

3 Answers 3

15

Is this study reliable?

We often get questions on Skeptics.SE about one particular study, asking if it "proves" some controversial claim.

The short answer, without even looking at the study, is "No. One single study is not reliable. Studies are often wrong. Unfortunately, sometimes it is all the evidence we have."

If we have the luxury of more than one study of a phenomena, and they disagree, we can use a systematic review (or even better, a meta-analysis) to try to resolve the dilemma. More on that in a moment.


This particular study has received some criticism.

For example, Philip N. Cohen, [now] Professor of Sociology, wrote a blog article in 2013 criticising several aspects.

He claims the sample size is actually about 85 gay-father kids and 194 lesbian-mother kids. That should be enough to get statistically significant results for large effect sizes, but it is considerably smaller than the headline figure of "20% of the Canadian Census" might suggest.

He claims the sampling has selection bias by including 17-22 year olds that still live with their parents, while excluding those who moved out.

[Note: This means even if the results are "statistically significant", they are still biased.]

He claims the odds ratio presented is misleading:

Anyway, the paper does provide the marginal effects, which show that the children living with gay parents have graduated from high school at an adjusted predicted percentage 6 points lower than those living with married different-sex parents


Meanwhile, Mark Regnerus, [now] Professor of Sociology, wrote a 2020 article, Understanding how the social scientific study of same-sex parenting works defending his own 2012 paper that found differences in children of same-sex relationships. That paper caused some controversy in the social science community, with open letters attacking the research and defending it being signed by dozens or hundreds of scholars.

Regnerus argues that population studies are hard, and subject to coding errors, that the differences in outcomes are hidden in the choice of factors to control for and sampling strategies, and accuses fellow scientists of not being neutral in their choice of papers to criticise:

As a result, it is difficult to conduct solid social science research on such topics, when the world of scholarship on sexuality has tacitly ruled some conclusions more worthy of publication than others. Scientific neutrality is out. Political expedience is in.

So, we can see that individual papers on this subject have triggered a lot of criticism and defence, reinforcing that we should treat each paper with care.


But back to systematic reviews...

In 2022, a team published Family outcome disparities between sexual minority and heterosexual families: a systematic review and meta-analysis.

They looked at 34 studies (16 for the meta-analysis) for a large number of outcomes.

In general:

Most of the family outcomes are similar between sexual minority and heterosexual families, and sexual minority families have even better outcomes in some domains. Relevant social risk factors of poor family outcomes included stigma and discrimination, poor social support and marital status, etc.

However, the question here is about academic achievement, so let's look at what they found there.

We conducted a narrative synthesis of six studies on children’s educational outcomes. Four studies indicated that children from same-sex couples appear to have the higher rate of grade retention, lower graduation rate or worse educational attainment than children from different-sex couples. On the contrary, two studies reported that children in sexual minority parent families outperform children in heterosexual parent families on standardised test scores, high school graduation rates, college enrolment, and school/academic competence.

[Note that the original study in the question is one of the four.]

They did not conduct a meta-analysis on this data, so it isn't clear how strong each study is, but as you can see: The result is unclear:

Four studies found children of same-sex parents do worse at school. Two studies found they do better.

11
  • 1
    I wonder if one difference between studies might be who they compare the students to. Consider this: as far as I know (and I could be wrong), most children of same-sex couples are adopted, and adopted children tend to fare worse academically. So, if most of the children of the same-sex couples are adopted, then do they have the same ratio in the control group of adopted children? Aug 8, 2023 at 13:05
  • 1
    The short answer, without even looking at the study, is "No. One single study is not reliable. Studies are often wrong. Unfortunately, sometimes it is all the evidence we have." This is incorrect and 'I don't even need to read something to know what it means' is incredibly intellectually lazy. It's the opinion of the most overconfident and incompetent first-year graduate student, and I hate it. It diminishes what is otherwise a fine answer to the question.
    – CJR
    Aug 8, 2023 at 15:05
  • 1
    "The journal is Polish, but the paper is in English. That is a very odd choice of publication location for a Texan professor." There's nothing odd about that. Academic journals are international, and authors select journals based on their prestige and suitability for the topic, not on where the journal is published. And English has become a popular choice for scientific publication all over the world. Aug 8, 2023 at 18:14
  • 1
    As an example, I am an American mathematician working in the US. I am currently collaborating with two other US-based researchers, one American and the other Russian-American, and a fourth collaborator who is an Italian working in Switzerland, on a paper written in English, which we plan to submit to a journal owned by an Israeli university and operated by a German publisher. All recent papers in this Israeli journal are in English, and I am not sure that they would even accept a paper in Hebrew. None of this is unusual in any way. Aug 8, 2023 at 18:20
  • 3
    @NateEldredge: In principle, you're right: international journals may be the right outlet even for researchers from the US as well. However, if you look at the Editorial Team and the Scientific Council, it's a bit difficult to see a broad international perspective here – and the fact that many of the academics involved seem to come from institutions that have at least a nominal affiliation to the Catholic church may be seen as a red flag when it comes to a study on LGBT issues.
    – Schmuddi
    Aug 8, 2023 at 19:51
3

Unknown

Usually, an analysis of this type would be independently verifiable. The authors would summarize how the data was generated, how it was analyzed, and a skilled scientist could then reproduce the analysis (or could perform a different analysis to test a different hypothesis).

This is a core concept for science - Reproducibility.

In this case, the author provides very little methodology and omits virtually the entire analysis:

This file is not a public use data set. To use the data, a proposal is screened by the Social Sciences Research Council of Canada, an RCMP criminal check is conducted, and the researcher becomes a deemed employee of Statistics Canada subject to the penalties of the Statistics Act. Empirical work was conducted at the SFU Research Data Center, and all results were screened by Statistics Canada before release. Statistics Canada does not allow any unweighted observations or descriptives to be released, nor any maximums or minimums of weighted estimates, nor sample sizes for the weighted regressions.

It is possible that this analysis is correct. It is possible that this analysis is partially correct, but uses inappropriate controlling (for example, by regressing out or controlling for a causal factor which then confounds the analysis). It is possible that this analysis is completely invalid, the result of a trivial coding or statistical mistake. It is possible that this analysis is completely fraudulent.

With the information provided, it is unknown and unknowable if this study accurately analyzes the underlying data or if the analysis is appropriate.

3
  • 2
    This disclaimer looks very strange. Statistics Canada is an official agency of the Canadian government but his claim that they forbid him from releasing any mathematicals details on the statistical analysis is almost guaranteed to be a misrepresentation. If such an agency does stats work for someone they want this to look like sound statistics whereas he claims full secrecy.
    – quarague
    Aug 9, 2023 at 8:29
  • 1
    @quarague I've worked with similar data (in a different country) and this kind of restriction is very plausible. It's about privacy: the agency has a legal and moral obligation to safeguard sensitive personal data, which includes ensuring that releases of this kind can't be reverse-engineered to recover anything at individual level. Every piece of information makes that harder to guarantee, and with small populations (only 85 children of male-male relationships) the "information budget" is quite small to begin with. Protecting sources trumps reproducibility. Aug 13, 2023 at 5:15
  • 1
    Which is not to say that the lack of reproducibility isn't an issue, mind - it would be entirely reasonable to take that into account when deciding how much weight to put on its findings. Just that there are bona fide reasons for it in this case. Aug 13, 2023 at 5:19
-2

What is there to be skeptical about with respect to statistical reliability?

The results are based on a 20% sample of the census of the entire country. That is by far a much greater sample size than is available for almost any other statistical study on almost any other topic.

Whatever the results are, they should be considered very statistically significant (i.e. the sample reliably represents the actual population).

What is there to be skeptical about with respect to 65% being a significant difference?

The results themselves show a 35% reduction in high school graduation. That children are more than 50% likelier to graduate if they aren't living with same-sex parents is a significant difference.

One could argue that say a ¼% reduction has statistical significance, but have little meaning when applied to reality. But this isn't ¼%, it is 65%, which is highly significant in all contexts.

Here are the relevant abstract and conclusion from the paper:

Abstract

Almost all studies of same-sex parenting have concluded there is “no difference” in a range of outcome measures for children who live in a household with same-sex parents compared to children living with married opposite-sex parents. Recently, some work based on the US census has suggested otherwise, but those studies have considerable drawbacks. Here, a 20 % sample of the 2006 Canada census is used to identify self-reported children living with same-sex parents, and to examine the association of household type with children’s high school graduation rates. This large random sample allows for control of parental marital status, distinguishes between gay and lesbian families, and is large enough to evaluate differences in gender between parents and children. Children living with gay and lesbian families in 2006 were about 65 % as likely to graduate compared to children living in opposite sex marriage families. Daughters of same-sex parents do considerably worse than sons.

5 Conclusion

A casual reading of the literature on child performance would suggest that no-difference in child outcomes exists between children in same-sex or opposite-sex households. Indeed, the unanimous opinion of so many studies would appear conclusive—as noted by Justice Walker. However, a closer inspection reveals that there are really fifty-plus “preliminary” studies, and no general conclusion about child performance differences is warranted based on the literature. The samples used in these studies are often biased in some way, and the sample sizes are often very small. The one study that did use a large random sample and address a reliable performance measure (Rosenfeld 2010), turned out to draw the wrong conclusion, did not compare gay versus lesbian homes, did not examine the gender mix of the household, and did not control for parental marital status. As a result, there is little hard evidence to support the general popular consensus of “no difference.”

I have argued that the 2006 Canada census—though not perfect—is able to address most of these issues, and the results on high school graduation rates suggest that children living in both gay and lesbian households struggle compared to children from opposite sex married households. In general, it appears that these children are only about 65 % as likely to graduate from high school compared to the control group—a difference that holds whether conditioned on controls or not. When the households are broken down by child gender it appears that daughters are struggling more than sons, and that daughters of gay parents have strikingly low graduation rates.

This paper confirms the findings of Allen et al. (2013), and taken together they cast doubt on the ubiquitous claim that no difference exists in child outcomes for children raised by same-sex parents compared to married opposite sex parents. That is, both the US census and Canada census show that children living with same-sex parents perform poorer in school when compared to children from married opposite sex families.

The question is: why? This study suggests further work is necessary to narrow down the source of this difference. This will require an exceptional data set that not only identifies sexual orientation of parents, but also has a retrospective or panel design to completely control for marital history. A better data set would also be able to test for the reasons behind any difference. An economist may be inclined to think that fathers and mothers are not perfect substitutes and that there must be some gains from a sexual division of labor in parenting. Others may suspect that children of same-sex parents are more likely to be harassed at school, and therefore, less likely to graduate. In any event, it is time to investigate the difference and reject the conventional wisdom of “no difference.”

4
  • 5
    I don't think this answers the question; "What is there to be skeptical about with respect to statistical reliability?" While the sample may be large, there are plenty of reasons why the data could be flawed. How was the 20% selected? Is it biased? Was the analysis fair and statistically valid? Were there mistakes in the code? Sample size is only one issue.
    – Oddthinking
    Aug 8, 2023 at 3:03
  • "What is there to be skeptical about with respect to 65% being a significant difference?" Be careful not to confound effect size (aka clinical significance) and statistical significance.
    – Oddthinking
    Aug 8, 2023 at 3:06
  • 1
    Yes, the total sample size is large, but that's not very relevant here. The point of this study is to compare between subsets of that population, and the significance of such comparisons is limited by the sizes of those subsets - in particular, by the smallest of them. The issue is further complicated by the need to control for other variables. Merely looking at the size of the total sample is not a valid way to gauge significance here. Aug 13, 2023 at 5:25
  • @GBsupportsthemodstrike, so one could at least reliably conclude that the incidence of same-sex couples with adult children was quite small in 2006, with children having to have been born in the 1980s or earlier. But given the way the world has changed though, I suspect it has considerably increased during the ensuing 17 years. Aug 13, 2023 at 15:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .