ArticlePsychonomic bulletin & review2012
Too good to be true: publication bias in two prominent studies from experimental psychology.
Article in Psychonomic bulletin & review, 2012. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 55 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
55 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Robust Bayesian meta-analysis: Model-averaging across complementary publication bias adjustment methods.Research synthesis methods · 2023Pooled it
- Underpowered samples, false negatives, and unconscious learning.Psychonomic bulletin & review · 2016Pooled it
- A retrospective analysis of dissemination biases in the brief alcohol intervention literature.Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors · 2015Pooled it
- Training spatial cognition enhances mathematical learning in a randomized study of 17,000 children.Nature human behaviour · 2021Trial
- Attribute Conditioning is insensitive to cue competition and is not predicted by the Big Five Personality Traits.Personality & social psychology bulletin · 2026Article
- Article
- Contrast analysis for competing hypotheses: A tutorial using the R package cofad.Behavior research methods · 2025Article
- Article
- Evidence that publication bias contaminated studies on visual event boundaries and anchoring effects.Proceedings of the National Academy of Sciences of the United States of America · 2024Article
- Bilingual Language Experience and Its Effect on Conflict Adaptation in Reactive Inhibitory Control Tasks.Psychological science · 2023Article
- Reversing Bonferroni.Psychonomic bulletin & review · 2021Review
- Reporting in Experimental Philosophy: Current Standards and Recommendations for Future Practice.Review of philosophy and psychology · 2021Article
- Article
- The volatile nature of positive affect effects: opposite effects of positive affect and time on task on proactive control.Psychological research · 2020Article
- Replications in Comparative Cognition: What Should We Expect and How Can We Improve?Animal behavior and cognition · 2020Article
- Low replicability can support robust and efficient science.Nature communications · 2020Article
- A preregistered multi-lab replication of Maier et al. (2014, Exp. 4) testing retroactive avoidance.PloS one · 2020Article
- Moderation of classic social psychological effects by demographics in the U.S. adult population: New opportunities for theoretical advancement.Journal of personality and social psychology · 2019Article
- Improving Psychological Science through Transparency and Openness: An Overview.Perspectives on behavior science · 2019Article
- The quest for an optimal alpha.PloS one · 2019Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
1 author.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Empirical replication has long been considered the final arbiter of phenomena in science, but replication is undermined when there is evidence for publication bias. Evidence for publication bias in a set of experiments can be found when the observed number of rejections of the null hypothesis exceeds the expected number of rejections. Application of this test reveals evidence of publication bias in two prominent investigations from experimental psychology that have purported to reveal evidence of extrasensory perception and to indicate severe limitations of the scientific method. The presence of publication bias suggests that those investigations cannot be taken as proper scientific studies of such phenomena, because critical data are not available to the field. Publication bias could partly be avoided if experimental psychologists started using Bayesian data analysis techniques.
Indexed as
Identifiers
22351589What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.