ArticlePloS one2016
The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.
Article in PloS one, 2016. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 17 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
17 citing papers in PubMed.
- Applicants as reviewers: a mixed methods evaluation of the risks, benefits, and potential of distributed peer review for grant funding allocations.Research integrity and peer review · 2026Article
- Homophily in grant evaluations.Nature human behaviour · 2026Article
- A peer-review decision support tool for research funding decision-making using the analytic hierarchy process.PloS one · 2026Article
- Two changes that may help to improve NIH peer review.Proceedings of the National Academy of Sciences of the United States of America · 2022Article
- Conservatism Gets Funded? A Field Experiment on the Role of Negative Information in Novel Project Evaluation.Management science · 2022Article
- Peer reviewers' dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences.Humanities & social sciences communications · 2022Article
- Getting funded in a highly fluctuating environment: Shifting from excellence to luck and timing.PloS one · 2022Article
- Peer review: Risk and risk tolerance.PloS one · 2022Article
- Industry Collaborations of Research Teams: Are They Penalized or Rewarded in the Grant Evaluation Process?Frontiers in research metrics and analytics · 2021Article
- Article
- Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program.Research policy · 2020Article
- Article
- Recommendations for Reviewers of Biomedical Imaging Grant Applications.Molecular imaging and biology · 2019Article
- Why can't we make research grant allocation systems more consistent? A personal opinion.Ecology and evolution · 2019Article
- Low agreement among reviewers evaluating the same NIH grant applications.Proceedings of the National Academy of Sciences of the United States of America · 2018Article
- Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study.F1000Research · 2018Article
- What do we know about grant peer review in the health sciences?F1000Research · 2017Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Although the scientific peer review process is crucial to distributing research investments, little has been reported about the decision-making processes used by reviewers. One key attribute likely to be important for decision-making is reviewer expertise. Recent data from an experimental blinded review utilizing a direct measure of expertise has found that closer intellectual distances between applicant and reviewer lead to harsher evaluations, possibly suggesting that information is differentially sampled across subject-matter expertise levels and across information type (e.g. strengths or weaknesses). However, social and professional networks have been suggested to play a role in reviewer scoring. In an effort to test whether this result can be replicated in a real-world unblinded study utilizing self-assessed reviewer expertise, we conducted a retrospective multi-level regression analysis of 1,450 individual unblinded evaluations of 725 biomedical research funding applications by 1,044 reviewers. Despite the large variability in the scoring data, the results are largely confirmatory of work from blinded reviews, by which a linear relationship between reviewer expertise and their evaluations was observed-reviewers with higher levels of self-assessed expertise tended to be harsher in their evaluations. However, we also found that reviewer and applicant seniority could influence this relationship, suggesting social networks could have subtle influences on reviewer scoring. Overall, these results highlight the need to explore how reviewers utilize their expertise to gather and weight information from the application in making their evaluations.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.