ArticlePloS one2015
Menage a quoi? Optimal number of peer reviewers.
Article in PloS one, 2015. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 16 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
16 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Human researchers are superior to large language models in writing a medical systematic review in a comparative multitask assessment.Scientific reports · 2025Pooled it
- Pooled it
- Article
- Evaluation of Large Language Model Performance in Assessing Health Economic Study Quality.Journal of health economics and outcomes research · 2025Article
- Peer review: Risk and risk tolerance.PloS one · 2022Article
- Individual versus general structured feedback to improve agreement in grant peer review: a randomized controlled trial.Research integrity and peer review · 2021Article
- Article
- Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion.Research integrity and peer review · 2020Article
- Article
- Assessment of potential bias in research grant peer review in Canada.CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne · 2018Article
- Should Authors be Requested to Suggest Peer Reviewers?Science and engineering ethics · 2018Article
- An output evaluation of a health research foundation's enhanced grant review process for new investigators.Health research policy and systems · 2017Article
- Calibration with confidence: a principled method for panel assessment.Royal Society open science · 2017Article
- What do we know about grant peer review in the health sciences?F1000Research · 2017Review
- Article
- Does single blind peer review hinder newcomers?ScientometricsArticle
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
1 author.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Peer review represents the primary mechanism used by funding agencies to allocate financial support and by journals to select manuscripts for publication, yet recent Cochrane reviews determined literature on peer review best practice is sparse. Key to improving the process are reduction of inherent vulnerability to high degree of randomness and, from an economic perspective, limiting both the substantial indirect costs related to reviewer time invested and direct administrative costs to funding agencies, publishers and research institutions. Use of additional reviewers per application may increase reliability and decision consistency, but adds to overall cost and burden. The optimal number of reviewers per application, while not known, is thought to vary with accuracy of judges or evaluation methods. Here I use bootstrapping of replicated peer review data from a Post-doctoral Fellowships competition to show that five reviewers per application represents a practical optimum which avoids large random effects evident when fewer reviewers are used, a point where additional reviewers at increasing cost provides only diminishing incremental gains in chance-corrected consistency of decision outcomes. Random effects were most evident in the relative mid-range of competitiveness. Results support aggressive high- and low-end stratification or triaging of applications for subsequent stages of review, with the proportion and set of mid-range submissions to be retained for further consideration being dependent on overall success rate.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.