SynthesisPloS one2016
Replication Validity of Initial Association Studies: A Comparison between Psychiatry, Neurology and Four Somatic Diseases.
Synthesis in PloS one, 2016. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 7 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
7 citing papers in PubMed.
- Editorial Perspective: Misaligned incentives in mental health research - the case for Registered Reports.Journal of child psychology and psychiatry, and allied disciplines · 2024Article
- Effect size, sample size and power of forced swim test assays in mice: Guidelines for investigators to optimize reproducibility.PloS one · 2021Article
- Understanding and Mitigating the Replication Crisis, for Environmental Epidemiologists.Current environmental health reports · 2019Review
- No significant enrichment of rare functionally defective CPA1 variants in a large Chinese idiopathic chronic pancreatitis cohort.Human mutation · 2017Article
- Article
- Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions.PLoS biology · 2016Article
- Messaging in Biological Psychiatry: Misrepresentations, Their Causes, and Potential Consequences.Harvard review of psychiatryReview
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
contextThere are growing concerns about effect size inflation and replication validity of association studies, but few observational investigations have explored the extent of these problems.
objectiveUsing meta-analyses to measure the reliability of initial studies and explore whether this varies across biomedical domains and study types (cognitive/behavioral, brain imaging, genetic and "others").
methodsWe analyzed 663 meta-analyses describing associations between markers or risk factors and 12 pathologies within three biomedical domains (psychiatry, neurology and four somatic diseases). We collected the effect size, sample size, publication year and Impact Factor of initial studies, largest studies (i.e., with the largest sample size) and the corresponding meta-analyses. Initial studies were considered as replicated if they were in nominal agreement with meta-analyses and if their effect size inflation was below 100%.
resultsNominal agreement between initial studies and meta-analyses regarding the presence of a significant effect was not better than chance in psychiatry, whereas it was somewhat better in neurology and somatic diseases. Whereas effect sizes reported by largest studies and meta-analyses were similar, most of those reported by initial studies were inflated. Among the 256 initial studies reporting a significant effect (p<0.05) and paired with significant meta-analyses, 97 effect sizes were inflated by more than 100%. Nominal agreement and effect size inflation varied with the biomedical domain and study type. Indeed, the replication rate of initial studies reporting a significant effect ranged from 6.3% for genetic studies in psychiatry to 86.4% for cognitive/behavioral studies. Comparison between eight subgroups shows that replication rate decreases with sample size and "true" effect size. We observed no evidence of association between replication rate and publication year or Impact Factor.
conclusionThe differences in reliability between biological psychiatry, neurology and somatic diseases suggest that there is room for improvement, at least in some subdomains.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.