Forty Years of Confirmation Bias in Social Science: Two Case Studies of Selective Citations

Walter Schumm, Catherine R. Pakaluk, Duane W. Crawford


Objectives:  To provide a further test of possible hypothesis conformity or confirmation bias in science based on differential citations of two sets of articles published between 1979 and 2011 that concerned same-sex parenting, as part of a larger issue of similar biases in all sciences and medical research in general. 

Methods: Two sets of articles used here formed nearly perfect natural experiments because the articles were from the same university team of scholars, published about the same time, often concerned the same samples, with some being published in the same scholarly journal. Differences in citation rates might typically be related to article quality, but with these two sets of articles, research value was usually similar.  Based on Google Scholar citations, citation rates were compared using binomial tests, including normal approximation z scores, and one-sample chi-square tests. 

Results:  For both sets of articles, those that presented findings more favorable to same-sex parenting were cited far more often than those with significant but less favorable results.  Some authors disregarded their own significant research findings in later articles.    

Conclusions:  Confirmation bias or hypothesis conformity, considered to be a questionable research practice (QRP), have existed in at least one area of the sciences for as long as four decades.  Even when noticed and reported, such QRPs have continued, almost certainly leading to errors in reviews of the literature in such areas, errors – if accepted - that could lead to incorrect policy decisions.  However, scholars who dare to challenge incorrect scholarly “consensus”, report fraudulent research, or who dare to dispute widely accepted findings may have their professional careers destroyed.  Without better policing of QRPs, the value of social science research for society may decline even if it continues to benefit those who engage in QRPs.  However, our observations are only a narrow slice of a much larger problem of various kinds of bias in science and medical or public health research in general.


Citation rates, confirmation bias, hypothesis conformity, social science, research methodology, literature reviews, questionable research practices (QRP), lesbian mothers, LGBT, same-sex parenting

Full Text:



Reisig, MD, Holtfreter K, Berzofsky M. The perceived prevalence of research fraud among faculty at research-intensive universities in the USA. Accountability in Research. 2020, advance online.

Kostoff RN. Under-reporting of adverse events in the biomedical literature. J Data Information Sci. 2017; 1(4): 10-32.

Oreskes N, Conway EM. Merchants of doubt: How a handful of scientists obscured the truth on issues from tobacco smoke to global warming. New York, NY: Bloomsbury Press.

Yeo-The NSL, Tang BL. An alarming retraction rate for scientific publications on Coronavirus Diseas 2019 (COVID-19). Accountability in Research. 2020, advance online.

Kleck G. Targeting guns: firearms and their control. New York: Aldine de Gruyter, 1997.

Kates DB, Schaffer HE, Lattimer JK, Murray, GB, Cassem EH. Guns and public health: epidemic of violence or pandemic of propaganda? Tennessee Law Review. 1994; 61: 513-596.

Clark CJ, Winegard BM. Tribalism in war and peace: the nature and evolution of ideological epistemology and its significance for modern social science. Psychological Inquiry. 2020; 31(1): 1-22.

Haas, IJ. Ideological asymmetries in social psychological research: rethinking the impact of political context on ideological epistemology. Psychological Inquiry. 2020; 31(1): 29-34.

Honeycutt N, Jussim L. A model of political bias in social science research. Psychological Inquiry. 2020; 31(1): 73-85.

Azevedo F. Not so simple: science is in the details. Psychological Inquiry. 2020; 31(1): 61-65.

Pennycook G. Belief bias and its significance for modern social science. Psychological Inquiry. 2020; 31(1): 57-60.

Lai CK. Ordinary claims require ordinary evidence: a lack of direct support for equalitarian bias in the social sciences. Psychological Inquiry. 2020; 31(1): 42-47.

Inbar Y. Unjustified generalization: an overlooked consequence of ideological bias. Psychological Inquiry. 2020; 31(1): 90-93.

Tong KJ, von Hippel W. Sexual selection, history, and the evolution of tribalism. Psychological Inquiry. 2020; 31(1): 23-25.

Haidt J. Tribalism, forbidden baserates, and the telos of social science. Psychological Inquiry. 2020; 31(1): 53-56.

Winegard BM, Clark CJ. Without contraries is no progression. Psychological Inquiry. 2020; 31(1): 94-101.

Duyx B, Urlings MJE, Swaen GMH, Bouter LM, Zeegers MP. Scientific citations favor positive results: a systematic review and meta-analysis. J. Clin. Epidemiology. 2017; 88: 92-101.

Fanelli D. Positive results receive more citations, but only in some disciplines. Scientometrics.2013; 94: 701-709.

Urlings MJE, Duyx B, Swaen GMH, Bouter LM, Zeegers MPA. Citation bias in the literature on dietary trans fatty acids and serum cholesterol. J Clin. Epidemiology. 2019; 106: 88-97.

Schofield T. Knowing what we don’t know: A meta-analysis of children raised by gay or lesbian parents. The Winnower. 2016; 3: e147568.84110.

Harris MJ. Controversy and cumulation: Meta-analysis and research on interpersonal expectancy effects. Personality and Social Psychology Bulletin. 1991; 17: 316-322.

Martin M. The philosophical importance of the Rosenthal effect. Journal for the Theory of Social Behavior. 1977; 7: 81-97.

Rosenthal R. Covert communication in classrooms, clinics, courtrooms, and cubicles. American Psychologist. 2002; 57: 839-849.

Grant-Mejer K. Worldview and confirmation bias: the problem of social science. Filozofia Naukiu (Philosophy of Science). 2017; 25(2): 103-119.

Schumm WR. Navigating treacherous waters – one researcher’s 40 years of experience with controversial scientific research. Comprehensive Psych. 2015; 4(24): 1-40.

Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Quarterly. 2016; 94: 485-514.

Schumm WR, Crawford DW. Evaluating the quality of literature reviews in the social science: developing a measure of quality with an illustration. Psychology Research and Applications. 2019; 1(2): 47-69.

Schumm WR. Same-sex parenting research: a critical assessment. London, UK: Wilberforce Press, 2018.

Schumm WR. A review and critique of research on same-sex parenting and adoption. Psychological Reports. 2016; 119: 641-760.

Schumm WR, Crawford DW. Violations of fairness in social science research: The case of same-sex marriage and parenting. International Journal for the Jurisprudence of the Family. 2015; 6: 67-113.

Allen DW, Palakuk C, Price J. Nontraditional families and childhood progress through school: A comment on Rosenfeld. Demography. 2013; 50: 955-961.

Mucklow BM, Phelan GK. Lesbian and traditional mothers’ responses to Adult Response to Child Behavior and self-concept. Psychological Reports. 1979; 44(3): 880-882.

Miller JA, Jacobsen RB, Bigner JJ. The child’s home environment for lesbian vs. heterosexual mothers: a neglected area of research. J. Homosexuality. 1981; 7(1): 49-56.

Miller JA, Mucklow BM, Jacobsen RB, Bigner JJ. Comparison of family relationships: homosexual versus heterosexual women. Psychological Reports. 1980; 46(3 Suppl): 1127-1132.

Schumm WR. Evidence of pro-homosexual bias in social science: citation rates and research in lesbian parenting. Psychological Reports. 2010; 106(2): 374-380.

Goldberg NG, Bos HMW, Gartrell NK. Substance use by adolescents of the USA National Longitudinal Lesbian Family Study. Journal of Health Psychology. 2011; 16(8): 1231-1240.

Gartrell NK, Bos HMW, Goldberg NG. Adolescents of the U.S. National Longitudinal Lesbian Family Study: sexual orientation, sexual behavior, and sexual risk exposure. Archives of Sexual Behavior. 2011; 40: 1199-1209.

Gartrell NK, Bos HMW, Koh A. Sexual attraction, sexual identity, and same-sex sexual experiences of adult offspring in the U.S. National Longitudinal Lesbian Family Study. Archives of Sexual Behavior. 2019; 48(5): 1495-1503.

Schumm WR, Crawford DW. Scientific consensus on whether LGBTQ parents are more likely (or not) to have LGBTQ children: an analysis of 72 social science reviews of the literature published between 2001 and 2017. J Int Women’s Studies. 2019; 20(7): 1-12.

Schumm WR. Changes over the decades in selected LGBTQ research findings. JSM Sexual Medicine. 2020; 4(2): 1029.

Schumm WR. Avenues for future LGBT theory and research. JSM Sexual Medicine. 2020; 4(2), 1031.

Schumm WR, Crawford DW. Is research on transgender children what it seems? Comments on recent research on transgender children with high levels of parental support. Linacre Quarterly. 2020; 87(1): 9-24.

Schumm WR, Crawford DW, Lockett L. Using statistics from binary variables to detect data anomalies, even possibly fraudulent research. Psychology Research and Applications. 2019; 1(4): 112-118.

Schumm WR, Crawford DW, Lockett L. Patterns of means and standard deviations with binary variables: a key to detecting fraudulent research. J Sci & Tech Res. 2019; 23(1): 17151-17153.

Martin CC. How ideology has hindered sociological insight. American Sociologist. 2016; 47(1): 115-130.

Chamlee-Wright E. Self-censorship and associational life in the liberal academy. Society. 2019; 56: 538-549.



  • There are currently no refbacks.
Copyright 2016. All rights reserved.