Evidence map›Paper›PMID 37945809›Full record

ArticleNature human behaviour2024

High replicability of newly discovered social-behavioural findings is achievable.

John Protzko, Jon Krosnick, Leif Nelson, Brian A Nosek, Jordan Axt, Matt Berent, Nicholas Buttrick, Matthew DeBell, Charles R Ebersole, Sebastian Lundmark and 7 more

RetractedAbstract readRetracted Publication
PubMed Full text at PMC Publisher
In one paragraph

Article in Nature human behaviour, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It has been retracted, and should not be counted. Cited by 16 papers.

0numbers the graph read from it
0cells of the map it votes in
16citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

16 citing papers in PubMed.

  1. Trial
  2. Article
  3. Article
  4. Article
  5. Article
  6. Article
  7. The assessment of replicability using the sum ofRoyal Society open science · 2024
    Article
  8. Article
  9. Article
  10. Article
  11. Article
  12. Article
  13. Article
  14. Article
  15. Article
  16. Article
4 · The record

Corrections and comments

5 · Who and what money

Authors and funding

17 authors.

John ProtzkoDepartment of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA. protzko@gmail.com.ORCID 0000-0001-5710-8635
Jon KrosnickInstitute for Research in the Social Sciences, Stanford University, Stanford, CA, USA.
Leif NelsonHaas School of Business, University of California, Berkeley, Berkeley, CA, USA.
Brian A NosekCenter for Open Science, Charlottesville, VA, USA.
Jordan AxtDepartment of Psychology, McGill University, Montreal, Quebec, Canada.ORCID 0000-0003-3944-8331
Matt BerentMatt Berent Consulting, Sharon, PA, USA.
Nicholas ButtrickDepartment of Psychology, University of Wisconsin-Madison, Madison, WI, USA.
Matthew DeBellInstitute for Research in the Social Sciences, Stanford University, Stanford, CA, USA.ORCID 0000-0002-1126-1586
Charles R EbersoleDepartment of Psychology, University of Virginia, Charlottesville, VA, USA.
Sebastian LundmarkSOM Institute, University of Gothenburg, Gothenburg, Sweden.ORCID 0000-0003-4096-988X
Bo MacInnisInstitute for Research in the Social Sciences, Stanford University, Stanford, CA, USA.
Michael O'DonnellMcDonough School of Business, Georgetown University, Washington, DC, USA.ORCID 0000-0002-8953-3609
Hannah PerfectoOlin School of Business, Washington University in St. Louis, St. Louis, MO, USA.ORCID 0000-0002-1121-3437
James E PustejovskyEducational Psychology Department, University of Wisconsin-Madison, Madison, WI, USA.ORCID 0000-0003-0591-9465
Scott S RoederDarla Moore School of Business, University of South Carolina, Columbia, SC, USA.
Jan WalleczekPhenoscience Laboratories, Berlin, Germany.
Jonathan W SchoolerDepartment of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, CA, USA.

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.

Indexed as

Research DesignSocial BehaviorHumansProspective StudiesReproducibility of ResultsSample Size

Identifiers

PMID37945809
PMCPMC10896719

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.