ArticleProceedings of the National Academy of Sciences of the United States of America2015
Using prediction markets to estimate the reproducibility of scientific research.
Article in Proceedings of the National Academy of Sciences of the United States of America, 2015. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 65 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
65 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- A meta-analysis assessing the effectiveness of demand-side interventions for sustainable food consumption and food waste reduction.Nature food · 2026Pooled it
- 3D printed patient-specific prostate cancer models to guide nerve-sparing robot-assisted radical prostatectomy: a systematic review.Journal of robotic surgery · 2023Pooled it
- There is no clear target for replication rates.Nature human behaviour · 2026Article
- Article
- fNIRS reproducibility varies with data quality, analysis pipelines, and researcher experience.Communications biology · 2025Article
- A metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments.Royal Society open science · 2025Article
- Subjective Confidence as a Monitor of the Replicability of the Response.Perspectives on psychological science : a journal of the Association for Psychological Science · 2025Article
- A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics.Royal Society open science · 2025Review
- The reliability of replications: a study in computational reproductions.Royal Society open science · 2025Article
- The Alternative Factors Leading to Replication Crisis: Prediction and Evaluation.Evaluation review · 2025Article
- Examining the replicability of online experiments selected by a decision market.Nature human behaviour · 2025Article
- Predicting the replicability of social and behavioural science claims in COVID-19 preprints.Nature human behaviour · 2025Article
- Can large language models help predict results from a complex behavioural science study?Royal Society open science · 2024Article
- Americans harbor much less favorable explicit sentiments toward young adults than toward older adults.Proceedings of the National Academy of Sciences of the United States of America · 2024Article
- Efficient alternatives for Bayesian hypothesis tests in psychology.Psychological methods · 2024Article
- Review
- Academics are more specific, and practitioners more sensitive, in forecasting interventions to strengthen democratic attitudes.Proceedings of the National Academy of Sciences of the United States of America · 2024Article
- Joint-outcome prediction markets for climate risks.PloS one · 2024Article
- Eleven years of student replication projects provide evidence on the correlates of replicability in psychology.Royal Society open science · 2023Article
- The replication crisis has led to positive structural, procedural, and community changes.Communications psychology · 2023Review
5 more citing papers are in PubMed but not listed here.
Corrections and comments
- Commented on by
- Commented on by
Authors and funding
8 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.