Trial reportPloS one2015
Grant Peer Review: Improving Inter-Rater Reliability with Training.
Trial report in PloS one, 2015. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 31 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
31 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Pooled it
- Applicants as reviewers: a mixed methods evaluation of the risks, benefits, and potential of distributed peer review for grant funding allocations.Research integrity and peer review · 2026Article
- Researcher profiles, proposal submission, and project approval: an exploratory case study at a Brazilian Federal Institute.Frontiers in research metrics and analytics · 2026Article
- Lottery before peer review is associated with increased female representation and reduced estimated economic cost in a German funding line.Nature communications · 2025Article
- Bayesian Nonparametric Models for Multiple Raters: A General Statistical Framework.Psychometrika · 2025Article
- Elements of successful NIH grant applications.Proceedings of the National Academy of Sciences of the United States of America · 2024Article
- Development and Validation of a Scoring Rubric for Editorial Evaluation of Peer-review Quality: A Pilot Study.The western journal of emergency medicine · 2024Article
- An experimental study of simulated grant peer review: Gender differences and psychometric characteristics of proposal scores.PloS one · 2024Article
- Reviewer training for improving grant and journal peer review.The Cochrane database of systematic reviews · 2023Article
- A new approach to grant review assessments: score, then rank.Research integrity and peer review · 2023Article
- Article
- Leadership Development Strategies in Interprofessional Healthcare Collaboration: A Rapid Review.Journal of healthcare leadership · 2023Review
- Bias Amplification in Gender, Gender Identity, and Geographical Affiliation.Journal of chemical information and modeling · 2022Article
- What works for peer review and decision-making in research funding: a realist synthesis.Research integrity and peer review · 2022Article
- 'Science by consensus' impedes scientific creativity and progress: A simple alternative to funding biomedical research.F1000Research · 2022Article
- The effect of the scale of grant scoring on ranking accuracy.F1000Research · 2022Article
- Individual versus general structured feedback to improve agreement in grant peer review: a randomized controlled trial.Research integrity and peer review · 2021Article
- A retrospective analysis of the peer review of more than 75,000 Marie Curie proposals between 2007 and 2018.eLife · 2021Article
- The leaky pipeline in research grant peer review and funding decisions: challenges and future directions.Higher education · 2021Review
- Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
This study developed and evaluated a brief training program for grant reviewers that aimed to increase inter-rater reliability, rating scale knowledge, and effort to read the grant review criteria. Enhancing reviewer training may improve the reliability and accuracy of research grant proposal scoring and funding recommendations. Seventy-five Public Health professors from U.S. research universities watched the training video we produced and assigned scores to the National Institutes of Health scoring criteria proposal summary descriptions. For both novice and experienced reviewers, the training video increased scoring accuracy (the percentage of scores that reflect the true rating scale values), inter-rater reliability, and the amount of time reading the review criteria compared to the no video condition. The increase in reliability for experienced reviewers is notable because it is commonly assumed that reviewers--especially those with experience--have good understanding of the grant review rating scale. The findings suggest that both experienced and novice reviewers who had not received the type of training developed in this study may not have appropriate understanding of the definitions and meaning for each value of the rating scale and that experienced reviewers may overestimate their knowledge of the rating scale. The results underscore the benefits of and need for specialized peer reviewer training.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.