ArticleProceedings of the National Academy of Sciences of the United States of America2024
Heterogeneity in effect size estimates.
Article in Proceedings of the National Academy of Sciences of the United States of America, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 22 papers, 10 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
22 citing papers in PubMed, 10 syntheses or guidelines pooled it.
- The Role of Tumor Debulking Surgery in Improving Survival of Patients with Head and Neck Cancer: A Systematic Review.Current oncology (Toronto, Ont.) · 2026Pooled it
- Work-family conflict and mental health: A systematic review and meta-analysis.PLoS medicine · 2026Pooled it
- The anti-inflammatory effects of Citrus hystrix DC.: a systematic review and meta-analysis of in vitro and in vivo studies.Scientific reports · 2026Pooled it
- Pooled it
- Evolution of clinical benefit in cancer trials: comparing pivotal approval data with updated efficacy.Frontiers in public health · 2026Pooled it
- Heterogeneity in gender differences in self-reported political preferences, trust, and well-being across 39 European countries.Scientific reports · 2025Pooled it
- A systematic review and meta-analysis of the effectiveness of social norms messaging approaches for improving health behaviours in developed countries.Nature human behaviour · 2025Pooled it
- A Critical Review of the Health Benefits Associated with Chia Seeds (Salvia Hispanica L.).Plant foods for human nutrition (Dordrecht, Netherlands) · 2025Pooled it
- Systematic Analysis of the Differential Effects of Red Meat on Colorectal Cancer Risks: A Meta-Analytic Approach.Journal of gastrointestinal cancer · 2025Pooled it
- Health-related quality of life and influencing factors in parents of children with congenital heart disease: a systematic review and meta-analysis.Frontiers in public health · 2025Pooled it
- Review and Meta-Analyses of the Effects of MLC601/MLC901 (NeuroAiD) on Post-Stroke Functional and Motor Recovery.Neurology international · 2026Review
- Article
- Change ofFrontiers in psychology · 2026Article
- Review
- Global insights into healthcare providers' knowledge, attitudes, practices, and concerns on pre-exposure prophylaxis: A systematic review and meta-analysis.Preventive medicine reports · 2025Review
- Beyond reweighting: On the predictive role of covariate shift in effect generalization.Proceedings of the National Academy of Sciences of the United States of America · 2025Article
- Effects of concurrent continuous aerobic and short rest resistance exercise training on metabolic biomarkers in type 2 diabetes patients: a systematic review and meta-analysis.Diabetology & metabolic syndrome · 2025Review
- Measuring population heterogeneity requires heterogeneous populations.Proceedings of the National Academy of Sciences of the United States of America · 2025Article
- Reply to Krefeld-Schwalb et al.: Measuring population heterogeneity requires upholding good scientific practice.Proceedings of the National Academy of Sciences of the United States of America · 2025Article
- Examining the replicability of online experiments selected by a decision market.Nature human behaviour · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors.
Funding
Abstract
A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.