ArticleProceedings of the National Academy of Sciences of the United States of America2016
Contextual sensitivity in scientific reproducibility.
Article in Proceedings of the National Academy of Sciences of the United States of America, 2016. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 87 papers, 5 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
87 citing papers in PubMed, 5 syntheses or guidelines pooled it.
- A research agenda for understanding how social inequality is linked to brain structure and function.Nature human behaviour · 2024Pooled it
- The Effect of Concurrent Depression on PTSD Outcomes in Trauma-Focused Psychotherapy: A Meta-Analysis of Randomized Controlled Trials.Behavior therapy · 2021Pooled it
- Systematic reviews and meta-analyses in the health sciences: Best practice methods for research syntheses.Social science & medicine (1982) · 2019Pooled it
- Replication and reproducibility issues in the relationship between C-reactive protein and depression: A systematic review and focused meta-analysis.Brain, behavior, and immunity · 2018Pooled it
- Meta-assessment of bias in science.Proceedings of the National Academy of Sciences of the United States of America · 2017Pooled it
- Recruitment Strategies Bias Sampling and Shape Replicability.Personality & social psychology bulletin · 2026Article
- Evaluation of the replicability of systematic reviews with meta-analyses of the effects of health interventions.Research synthesis methods · 2026Article
- Spicy Personality: On the Relationship Between Personality Traits and the Preference for Spicy Foods.Foods (Basel, Switzerland) · 2026Review
- Estimating the effect size of moral contagion in online networks: A pre-registered replication and meta-analysis.PNAS nexus · 2025Article
- Making Multimethod Latent State-Trait Models for Random and Fixed Situations Accessible: A Tutorial.Journal of personality · 2025Article
- A transparency statement improves trust in community-police interactions.Nature communications · 2025Article
- The Daily Lives of Crowdsourced U.S. Respondents: A Time Use Comparison of MTurk, Prolific, and ATUS.Sociological methodology · 2025Article
- Concerns About Replicability Across Two Crises in Social Psychology.International review of social psychology · 2025Article
- Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects.PloS one · 2025Article
- Methodological concerns underlying a lack of evidence for cultural heterogeneity in the replication of psychological effects.Communications psychology · 2024Article
- High replicability of newly discovered social-behavioural findings is achievable.Nature human behaviour · 2024Article
- Article
- Evaluating meta-analysis as a replication success measure.PloS one · 2024Article
- Article
- Licensing via Credentials: Replication Registered Report of Monin and Miller (2001) with Extensions Investigating the Domain-Specificity of Moral Credentials and the Association Between the Credential Effect and Trait Reputational Concern.International review of social psychology · 2024Article
27 more citing papers are in PubMed but not listed here.
Corrections and comments
- Commented on by
Authors and funding
4 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39% could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher's degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed "hidden moderators") between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should be mindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.