ArticleBMJ (Clinical research ed.)2022
Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study.
Article in BMJ (Clinical research ed.), 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 40 papers, 5 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
40 citing papers in PubMed, 5 syntheses or guidelines pooled it.
- Probiotic mediated modulation of neonatal health: a meta-analysis of prematurity-related morbidity, jaundice, and respiratory distress.Frontiers in cellular and infection microbiology · 2026Pooled it
- Can integrated care interventions strengthen primary care and improve outcomes for patients with chronic diseases? A systematic review and meta-analysis.Health research policy and systems · 2025Pooled it
- The effect of down-titration and discontinuation of heart failure pharmacotherapy in older people: A systematic review and meta-analysis.British journal of clinical pharmacology · 2025Pooled it
- The sound and surprise: overlapping meta-analyses on the topic of safety and efficacy of PD-1 and PD-L1 inhibitors in the treatment of non-small cell lung cancer.European journal of clinical pharmacology · 2023Pooled it
- Synthesizing evidence from the earliest studies to support decision-making: To what extent could the evidence be reliable?Research synthesis methods · 2022Pooled it
- How Often Do Large Language Models Agree with Each Other-And with the Truth? A Consensus- and Complexity-Stratified Analysis of Data Extraction for Neuroimaging AI.Journal of clinical medicine · 2026Article
- Automated data extraction for systematic reviews using GPT-5.2 and Google Gemini Pro 3: A dual-large language model approach in orthopaedic research.Knee surgery, sports traumatology, arthroscopy : official journal of the ESSKA · 2026Article
- An Open-Source Systematic Reviews Integrated System (OSSYRIS) - Streamlining Processes and Standardising Data Structures.Cochrane evidence synthesis and methods · 2026Article
- Toward Evidence Synthesis of Adverse Events in Imbalanced Time-to-Event Data.Journal of evidence-based medicine · 2026Article
- Evaluating Large Language Models for Automated Evidence Synthesis in Neuroimaging AI: A Multi-Model Benchmark.Journal of clinical medicine · 2026Article
- Automating data extraction in meta-research: A multi-model benchmark in network psychometrics papers.Behavior research methods · 2026Article
- Evaluation of the replicability of systematic reviews with meta-analyses of the effects of health interventions.Research synthesis methods · 2026Article
- Article
- An alternative method for assessing the fragility of survival analysis results: a proof-of-concept study based on the log-rank test.American journal of epidemiology · 2026Article
- Organic Pollutants and Risk of Type 2 Diabetes: A Systematic Review and Meta-analysis.Mayo Clinic proceedings. Innovations, quality & outcomes · 2026Article
- Assessing data extraction in randomized clinical trials with large language models.BMC medical research methodology · 2026Article
- Artificial Intelligence-Assisted Data Extraction With a Large Language Model: A Study Within Reviews.Annals of internal medicine · 2025Article
- Article
- Data extraction error and its implications on systematic reviews in urology: a protocol.International journal of surgery protocols · 2025Article
- Artificial intelligence for the science of evidence synthesis: how good are AI-powered tools for automatic literature screening?BMC medical research methodology · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
Abstract
objectivesTo investigate the validity of data extraction in systematic reviews of adverse events, the effect of data extraction errors on the results, and to develop a classification framework for data extraction errors to support further methodological research.
designReproducibility study. DATA SOURCES: PubMed was searched for eligible systematic reviews published between 1 January 2015 and 1 January 2020. Metadata from the randomised controlled trials were extracted from the systematic reviews by four authors. The original data sources (eg, full text and ClinicalTrials.gov) were then referred to by the same authors to reproduce the data used in these meta-analyses. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Systematic reviews were included when based on randomised controlled trials for healthcare interventions that reported safety as the exclusive outcome, with at least one pair meta-analysis that included five or more randomised controlled trials and with a 2×2 table of data for event counts and sample sizes in intervention and control arms available for each trial in the meta-analysis.
main outcome measuresThe primary outcome was data extraction errors summarised at three levels: study level, meta-analysis level, and systematic review level. The potential effect of such errors on the results was further investigated.
results201 systematic reviews and 829 pairwise meta-analyses involving 10 386 randomised controlled trials were included. Data extraction could not be reproduced in 1762 (17.0%) of 10 386 trials. In 554 (66.8%) of 829 meta-analyses, at least one randomised controlled trial had data extraction errors; 171 (85.1%) of 201 systematic reviews had at least one meta-analysis with data extraction errors. The most common types of data extraction errors were numerical errors (49.2%, 867/1762) and ambiguous errors (29.9%, 526/1762), mainly caused by ambiguous definitions of the outcomes. These categories were followed by three others: zero assumption errors, misidentification, and mismatching errors. The impact of these errors were analysed on 288 meta-analyses. Data extraction errors led to 10 (3.5%) of 288 meta-analyses changing the direction of the effect and 19 (6.6%) of 288 meta-analyses changing the significance of the P value. Meta-analyses that had two or more different types of errors were more susceptible to these changes than those with only one type of error (for moderate changes, 11 (28.2%) of 39
conclusionSystematic reviews of adverse events potentially have serious issues in terms of the reproducibility of the data extraction, and these errors can mislead the conclusions. Implementation guidelines are urgently required to help authors of future systematic reviews improve the validity of data extraction.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.