SynthesisBMC medical research methodology2026
Agreement between ChatGPT and human-derived multilevel meta-analyses: a reproducibility study across clinical evidence syntheses.
Synthesis in BMC medical research methodology, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Not yet cited in PubMed.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
0 citing papers in PubMed.
No citing paper in PubMed yet.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundLarge language models are increasingly integrated into biomedical research workflows, yet their ability to reliably reproduce complex statistical analyses remains insufficiently evaluated. While prior studies suggest that ChatGPT can replicate conventional meta-analyses, its performance in recalculating statistically demanding multilevel meta-analyses is unknown.
methodsWe conducted a systematic review to identify published multilevel meta-analyses (i.e., models accounting for dependency among multiple effect sizes within studies) of hip-related clinical studies that provided extraction tables enabling independent recalculation. For each outcome, pooled estimates and heterogeneity parameters were recomputed using ChatGPT under predefined analytical assumptions and compared with statistician-derived results. Agreement was assessed using absolute and relative differences and Bland-Altman analyses for pooled effects, and qualitative comparison of heterogeneity measures (I², τ²) based on absolute differences.
resultsSeven studies, each reporting a multilevel meta-analysis, comprising more than 40 pooled effect estimates were included. ChatGPT-derived pooled estimates showed close numerical agreement with human-derived results, with small median absolute differences for continuous (2.25) and binary outcomes (0.005) and minimal overall bias. Heterogeneity estimates demonstrated similar concordance, with predominantly small differences that did not alter qualitative interpretation. Larger deviations were limited to a small number of outcomes with extreme heterogeneity or inconsistent input structures.
conclusionChatGPT can reproducibly approximate the results of multilevel meta-analyses under predefined analytical conditions. These findings support the potential role of large language models as adjunct tools for reproducibility assessment and methodological validation in evidence synthesis, while emphasizing the continued need for expert oversight.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.