Evaluating Elicit's systematic reviews workflow in an umbrella review on air pollution and acute lower respiratory infections: a methodological study for quality appraisal.

backgroundThe exponential growth of scientific publications has increased the complexity of evidence synthesis. Systematic reviews remain essential but highly resource-intensive. Large language models (LLMs) offer new opportunities to support or partially automate key steps of this process. This study evaluates the performance of Elicit's Systematic Reviews workflow in comparison to the traditional methodology, using as reference a published umbrella review on the association between air pollution and acute lower respiratory infections (ALRI).

methodsA parallel workflow was developed to reproduce each phase of the traditional review. Considering article retrieval, for the traditional workflow, articles were retrieved through a Boolean search, while for the AI-assisted workflow a natural-language query submitted to Elicit. Screening was conducted in two steps, emulating the traditional PECOS-based criteria. Full-text evaluation and quality appraisal were performed through Elicit's "data extraction" functionalities. For quality appraisal the validated AMSTAR-2 EH questionnaire was applied.

resultsThe traditional Boolean search identified 324 unique articles. When compared with the 500 records retrieved by Elicit, an overlap of 8% was observed, which prevented a direct, recall-oriented comparison of search performance. To enable a controlled comparison of downstream steps, we applied Elicit's screening and data-extraction functions to the 324 records identified through the Boolean search, using empirically defined screening-score threshold in Elicit to select studies for further evaluation. In the screening on title and abstract, 33 articles were identified through the traditional workflow and 70 through Elicit, 30 articles overlapping (recall 90.9%, precision 42.9%). The full-text screening selected 15 articles with the traditional methodology and 24 with Elicit, all 15 articles from the traditional methodology being included in Elicit selection (recall 100%, precision 62.5%). In the quality assessment, Elicit showed 24.4% disagreement on general items and 30.4% on additional items of the AMSTAR-2 EH. Errors clustered around multi-component questions, items requiring expert interpretation, and information located in supplementary materials.

conclusionsElicit can support several phases of systematic reviews and reduce manual workload, but it cannot independently reproduce the methodological rigor required for high-quality evidence synthesis. At present, LLM-based tools are best positioned as complementary systems within human-supervised workflows.