ArticleRoyal Society open science2021
How best to quantify replication success? A simulation study on the comparison of replication success metrics.
Article in Royal Society open science, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 13 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
13 citing papers in PubMed.
- Article
- Article
- Evaluation of replication success: A statistical perspective and extended analysis of the SCORE data.PloS one · 2026Article
- A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics.Royal Society open science · 2025Review
- Examining the replicability of online experiments selected by a decision market.Nature human behaviour · 2025Article
- Replication marketplaces would help science to become more self-correcting.Royal Society open science · 2024Article
- The assessment of replicability using the sum ofRoyal Society open science · 2024Article
- Power priors for replication studies.Test (Madrid, Spain) · 2024Article
- Evaluating meta-analysis as a replication success measure.PloS one · 2024Article
- Research Into Evidence-Based Psychological Interventions Needs a Stronger Focus on Replicability.Clinical psychology in Europe · 2023Article
- The process of replication target selection in psychology: what to consider?Royal Society open science · 2023Article
- Effects of tDCS on the attentional blink revisited: A statistical evaluation of a replication attempt.PloS one · 2022Article
- How best to quantify replication success? A simulation study on the comparison of replication success metrics.Royal Society open science · 2021Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.