ArticleJMIR medical education2021
The Impact of Systematic Review Automation Tools on Methodological Quality and Time Taken to Complete Systematic Review Tasks: Case Study.
Article in JMIR medical education, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 21 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
21 citing papers in PubMed, 2 syntheses or guidelines pooled it, 61 citations in OpenAlex.
- How much can we save by applying artificial intelligence in evidence synthesis? Results from a pragmatic review to quantify workload efficiencies and cost savings.Frontiers in pharmacology · 2025Pooled it
- Experimental procedures for flow cytometry of wild-type mouse brain: a systematic review.Frontiers in immunology · 2023Pooled it
- Artificial Intelligence Tools for Automating Evidence Synthesis: Scoping Review.Journal of medical Internet research · 2026Article
- Automated tools in systematic reviews: Current trends.Indian journal of anaesthesia · 2025Article
- Artificial intelligence as team member versus manual screening to conduct systematic reviews in medical sciences.iScience · 2025Article
- Artificial Intelligence and Machine Learning to Improve Evidence Synthesis Production Efficiency: An Observational Study of Resource Use and Time-to-Completion.Cochrane evidence synthesis and methods · 2025Article
- Accelerating Disease Model Parameter Extraction: An LLM-Based Ranking Approach to Select Initial Studies For Literature Review Automation.Machine learning and knowledge extraction · 2025Article
- Machine learning for identifying randomised controlled trials when conducting systematic reviews: Development and evaluation of its impact on practice.Research synthesis methods · 2025Article
- Validation of automated paper screening for esophagectomy systematic review using large language models.PeerJ. Computer science · 2025Article
- Analysis of article screening and data extraction performance by an AI systematic literature review platform.Frontiers in artificial intelligence · 2025Article
- Artificial intelligence in food and nutrition evidence: The challenges and opportunities.PNAS nexus · 2024Article
- An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review.BMC medical research methodology · 2024Article
- Rapid review methods series: Guidance on the use of supportive software.BMJ evidence-based medicine · 2024Article
- Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed.Systematic reviews · 2024Article
- Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain.Systematic reviews · 2024Article
- Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study.Journal of medical Internet research · 2024Article
- Machine Learning Methods for Systematic Reviews:: A Rapid Scoping Review.Delaware journal of public health · 2023Article
- An Automated Literature Review Tool (LiteRev) for Streamlining and Accelerating Research Using Natural Language Processing and Machine Learning: Descriptive Performance Evaluation Study.Journal of medical Internet research · 2023Article
- How to make a systematic review live up to its name: perspectives from journal editors.Annals of translational medicine · 2023Article
- The effect of machine learning tools for evidence synthesis on resource use and time-to-completion: protocol for a retrospective pilot study.Systematic reviews · 2023Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors at 1 institution in 2 countries.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundSystematic reviews (SRs) are considered the highest level of evidence to answer research questions; however, they are time and resource intensive.
objectiveWhen comparing SR tasks done manually, using standard methods, versus those same SR tasks done using automated tools, (1) what is the difference in time to complete the SR task and (2) what is the impact on the error rate of the SR task?
methodsA case study compared specific tasks done during the conduct of an SR on prebiotic, probiotic, and synbiotic supplementation in chronic kidney disease. Two participants (manual team) conducted the SR using current methods, comprising a total of 16 tasks. Another two participants (automation team) conducted the tasks where a systematic review automation (SRA) tool was available, comprising of a total of six tasks. The time taken and error rate of the six tasks that were completed by both teams were compared.
resultsThe approximate time for the manual team to produce a draft of the background, methods, and results sections of the SR was 126 hours. For the six tasks in which times were compared, the manual team spent 2493 minutes (42 hours) on the tasks, compared to 708 minutes (12 hours) spent by the automation team. The manual team had a higher error rate in two of the six tasks-regarding Task 5: Run the systematic search, the manual team made eight errors versus three errors made by the automation team; regarding Task 12: Assess the risk of bias, 25 assessments differed from a reference standard for the manual team compared to 20 differences for the automation team. The manual team had a lower error rate in one of the six tasks-regarding Task 6: Deduplicate search results, the manual team removed one unique study and missed zero duplicates versus the automation team who removed two unique studies and missed seven duplicates. Error rates were similar for the two remaining compared tasks-regarding Task 7: Screen the titles and abstracts and Task 9: Screen the full text, zero relevant studies were excluded by both teams. One task could not be compared between groups-Task 8: Find the full text.
conclusionsFor the majority of SR tasks where an SRA tool was used, the time required to complete that task was reduced for novice researchers while methodological quality was maintained.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.