ArticleProceedings of the National Academy of Sciences of the United States of America2020
Estimating the deep replicability of scientific findings using human and artificial intelligence.
Article in Proceedings of the National Academy of Sciences of the United States of America, 2020. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 30 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
30 citing papers in PubMed.
- Saturation of moral language predicts lower content engagement on social media.Nature human behaviour · 2026Article
- A large-scale comparison of divergent creativity in humans and large language models.Nature human behaviour · 2026Article
- A framework for assessing the trustworthiness of scientific research findings.Proceedings of the National Academy of Sciences of the United States of America · 2026Article
- A metric of knowledge as information compression reflects reproducibility predictions for biomedical experiments.Royal Society open science · 2025Article
- A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics.Royal Society open science · 2025Review
- Public Disclosure of Results From Artificial Intelligence/Machine Learning Research in Health Care: Comprehensive Analysis of ClinicalTrials.gov, PubMed, and Scopus Data (2010-2023).Journal of medical Internet research · 2025Article
- Examining the replicability of online experiments selected by a decision market.Nature human behaviour · 2025Article
- Quantifying the use and potential benefits of artificial intelligence in scientific research.Nature human behaviour · 2024Article
- Can large language models help predict results from a complex behavioural science study?Royal Society open science · 2024Article
- Science communication with generative AI.Nature human behaviour · 2024Article
- Article
- Article
- Challenging the N-Heuristic: Effect size, not sample size, predicts the replicability of psychological science.PloS one · 2024Article
- Eleven years of student replication projects provide evidence on the correlates of replicability in psychology.Royal Society open science · 2023Article
- Reply to Crocket et al. and Mottelson and Kontogiorgos: Machine Learning's scientific significance and future impact on replicability research.Proceedings of the National Academy of Sciences of the United States of America · 2023Article
- The limitations of machine learning models for predicting scientific replicability.Proceedings of the National Academy of Sciences of the United States of America · 2023Article
- Data, measurement and empirical methods in the science of science.Nature human behaviour · 2023Review
- Biomedical data analyses facilitated by open cheminformatics workflows.Journal of cheminformatics · 2023Article
- Non-White scientists appear on fewer editorial boards, spend more time under review, and receive fewer citations.Proceedings of the National Academy of Sciences of the United States of America · 2023Article
- Gender inequality and self-publication are common among academic editors.Nature human behaviour · 2023Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Replicability tests of scientific papers show that the majority of papers fail replication. Moreover, failed papers circulate through the literature as quickly as replicating papers. This dynamic weakens the literature, raises research costs, and demonstrates the need for new approaches for estimating a study's replicability. Here, we trained an artificial intelligence model to estimate a paper's replicability using ground truth data on studies that had passed or failed manual replication tests, and then tested the model's generalizability on an extensive set of out-of-sample studies. The model predicts replicability better than the base rate of reviewers and comparably as well as prediction markets, the best present-day method for predicting replicability. In out-of-sample tests on manually replicated papers from diverse disciplines and methods, the model had strong accuracy levels of 0.65 to 0.78. Exploring the reasons behind the model's predictions, we found no evidence for bias based on topics, journals, disciplines, base rates of failure, persuasion words, or novelty words like "remarkable" or "unexpected." We did find that the model's accuracy is higher when trained on a paper's text rather than its reported statistics and that n-grams, higher order word combinations that humans have difficulty processing, correlate with replication. We discuss how combining human and machine intelligence can raise confidence in research, provide research self-assessment techniques, and create methods that are scalable and efficient enough to review the ever-growing numbers of publications-a task that entails extensive human resources to accomplish with prediction markets and manual replication alone.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.