ArticleProceedings of the National Academy of Sciences of the United States of America2017
Reviewer bias in single- versus double-blind peer review.
Article in Proceedings of the National Academy of Sciences of the United States of America, 2017. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 102 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
102 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Exploring Gender Bias in Six Key Domains of Academic Science: An Adversarial Collaboration.Psychological science in the public interest : a journal of the American Psychological Society · 2023Pooled it
- A randomized controlled trial on anonymizing reviewers to each other in peer review discussions.PloS one · 2024Trial
- Trial
- Trial
- Underrepresentation of women in computer systems research.PloS one · 2022Trial
- How gender determines the way we speak about professionals.Proceedings of the National Academy of Sciences of the United States of America · 2018Trial
- Pediatric Cardiology Editors: Insights, Challenges, and Future Growth Opportunities.Pediatric cardiology · 2026Article
- Editorial and peer review dynamics at elite general science journals.Science advances · 2026Article
- 10 Things I Wish I Knew Before My First Peer Review.Pediatric cardiology · 2026Review
- The paradox of peer review: protecting science or policing thought?Internal medicine journal · 2026Article
- Double anonymous peer review inJournal of vascular surgery cases and innovative techniques · 2026Article
- Perceptions of Arab researchers regarding publishing scientific research: A cross-sectional study.Accountability in research · 2026Article
- Scientific autonomy in the structural bubble: from institutional bias to AI-mediated consensus.Frontiers in research metrics and analytics · 2026Article
- Trust, transparency, and disciplinary alignment in research funding evaluations: an illustrative case from the Norwegian AI Center call (2024-2025).Frontiers in research metrics and analytics · 2026Article
- Diversity, Equity, and Inclusion in Biomedical Research and Publishing.Indian journal of psychological medicine · 2026Article
- Geographical diversity of peer reviewers shapes author success.Proceedings of the National Academy of Sciences of the United States of America · 2025Article
- Decentralized knowledge assessment.Innovation (Cambridge (Mass.)) · 2025Article
- Understanding author choices in the current conservation publishing landscape.Conservation biology : the journal of the Society for Conservation Biology · 2025Article
- The present and future of peer review: Ideas, interventions, and evidence.Proceedings of the National Academy of Sciences of the United States of America · 2025Article
- Evaluation of Large Language Model Performance in Assessing Health Economic Study Quality.Journal of health economics and outcomes research · 2025Article
42 more citing papers are in PubMed but not listed here.
Corrections and comments
- Commented on by
- Commented on by
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Peer review may be "single-blind," in which reviewers are aware of the names and affiliations of paper authors, or "double-blind," in which this information is hidden. Noting that computer science research often appears first or exclusively in peer-reviewed conferences rather than journals, we study these two reviewing models in the context of the 10th Association for Computing Machinery International Conference on Web Search and Data Mining, a highly selective venue (15.6% acceptance rate) in which expert committee members review full-length submissions for acceptance. We present a controlled experiment in which four committee members review each paper. Two of these four reviewers are drawn from a pool of committee members with access to author information; the other two are drawn from a disjoint pool without such access. This information asymmetry persists through the process of bidding for papers, reviewing papers, and entering scores. Reviewers in the single-blind condition typically bid for 22% fewer papers and preferentially bid for papers from top universities and companies. Once papers are allocated to reviewers, single-blind reviewers are significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors, top universities, and top companies. The estimated odds multipliers are tangible, at 1.63, 1.58, and 2.10, respectively.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.