ArticleBMC medical research methodology2021
Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool.
Article in BMC medical research methodology, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 13 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
13 citing papers in PubMed, 1 synthesis or guideline pooled it, 20 citations in OpenAlex.
- How much can we save by applying artificial intelligence in evidence synthesis? Results from a pragmatic review to quantify workload efficiencies and cost savings.Frontiers in pharmacology · 2025Pooled it
- Automated tools for evidence quality assessment: a scoping review.BMC medical research methodology · 2026Article
- The AuTOMATIC trial: a multicentre digitally-automated, Bayesian, adaptive, parallel, factorial randomised controlled trial of SMS reminders for childhood vaccination.The Lancet regional health. Western Pacific · 2026Article
- Review
- Trial characteristics, geographic distribution, and selected methodological issues of 1425 infertility trials published from 2012 to 2023: a systematic review.Human reproduction open · 2025Review
- An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review.BMC medical research methodology · 2024Article
- Applying Artificial Intelligence in Pediatric Clinical Trials: Potential Impacts and Obstacles.The journal of pediatric pharmacology and therapeutics : JPPT : the official journal of PPAG · 2024Article
- Patterns and Pitfalls of Short-cuts Used in Environmental Management Rapid Reviews.Environmental management · 2024Article
- How does air pollution threaten mental health? Protocol for a machine-learning enhanced systematic map.BMJ open · 2024Article
- Editorial: Evidence-based approaches in aging and public health.Frontiers in public health · 2024Article
- Rapid reviews methods series: Guidance on team considerations, study selection, data extraction and risk of bias assessment.BMJ evidence-based medicine · 2023Article
- Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour-Change Project.Wellcome open research · 2023Article
- Artificial Intelligence Applied to clinical trials: opportunities and challenges.Health and technology · 2023Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors at 1 institution in 1 country.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundMachine learning tools that semi-automate data extraction may create efficiencies in systematic review production. We evaluated a machine learning and text mining tool's ability to (a) automatically extract data elements from randomized trials, and (b) save time compared with manual extraction and verification.
methodsFor 75 randomized trials, we manually extracted and verified data for 21 data elements. We uploaded the randomized trials to an online machine learning and text mining tool, and quantified performance by evaluating its ability to identify the reporting of data elements (reported or not reported), and the relevance of the extracted sentences, fragments, and overall solutions. For each randomized trial, we measured the time to complete manual extraction and verification, and to review and amend the data extracted by the tool. We calculated the median (interquartile range [IQR]) time for manual and semi-automated data extraction, and overall time savings.
resultsThe tool identified the reporting (reported or not reported) of data elements with median (IQR) 91% (75% to 99%) accuracy. Among the top five sentences for each data element at least one sentence was relevant in a median (IQR) 88% (83% to 99%) of cases. Among a median (IQR) 90% (86% to 97%) of relevant sentences, pertinent fragments had been highlighted by the tool; exact matches were unreliable (median (IQR) 52% [33% to 73%]). A median 48% of solutions were fully correct, but performance varied greatly across data elements (IQR 21% to 71%). Using ExaCT to assist the first reviewer resulted in a modest time savings compared with manual extraction by a single reviewer (17.9 vs. 21.6 h total extraction time across 75 randomized trials).
conclusionsUsing ExaCT to assist with data extraction resulted in modest gains in efficiency compared with manual extraction. The tool was reliable for identifying the reporting of most data elements. The tool's ability to identify at least one relevant sentence and highlight pertinent fragments was generally good, but changes to sentence selection and/or highlighting were often required. PROTOCOL: https://doi.org/10.7939/DVN/RQPJKS.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.