ArticleNPJ digital medicine2024
Closing the gap between open source and commercial large language models for medical evidence summarization.
Article in NPJ digital medicine, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 52 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
52 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Fine-Tuning, Retrieval-Augmented Generation, and Hybrid Adaptation of Language Models for Clinical Decision-Making in Health Care: Systematic Review.Journal of medical Internet research · 2026Pooled it
- PRIME 2.0: Proposed Requirements for Cardiovascular Imaging-Related Multimodal-AI Evaluation: An Updated Checklist.JACC. Cardiovascular imaging · 2026Guideline
- GPT-4.1 and Llama 3.3 70 fail to detect clinically relevant errors in radiology reports in zero-shot evaluation.European radiology · 2026Article
- Exploring Conversational Dynamics in Scientific and Pseudoscientific Health Communities on YouTube: Process Mining and Network Analysis Study.Journal of medical Internet research · 2026Article
- Improving Reliability and Explainability of Medical Question Answering Through Atomic Fact-Checking in Retrieval-Augmented Large Language Models: Creation and Validation Study.Journal of medical Internet research · 2026Article
- MES: A Multi-Agent Evidence Synthesis System for Medical Decision-Making.medRxiv : the preprint server for health sciences · 2026Article
- Validating LLM judges for automated oversight of patient communication.medRxiv : the preprint server for health sciences · 2026Article
- From zero-shot to fine-tuning: optimize large language models for error detection of ultrasound reports.Insights into imaging · 2026Article
- Generative large language models in medicine: a scoping review of recent methodological advances.npj health systems · 2026Review
- An advanced LLM-powered article extraction pipeline delivers a database of buffers for protein studies and protein expression conditions.Briefings in bioinformatics · 2026Article
- Tutorial: guidance on the use of large language models for medical research.Nature protocols · 2026Review
- Review
- A data-driven method for research trend analysis in a scientific discipline: Application to the journal of biomedical informatics.Journal of biomedical informatics · 2026Article
- Comparing performance of seven fine-tuned open-source large language models in summarizing and predicting outcome-relevant information from mechanical thrombectomy reports in patients with acute ischemic stroke.European radiology · 2026Article
- Artificial intelligence-assisted quality assessment of mid-trimester ultrasound examinations using large vision-language models.BMC pregnancy and childbirth · 2026Article
- Multidisciplinary expert evaluation of large language models on questions regarding bariatric surgery: a comparative analysis of ERNIE Bot 4.0, ChatGPT-4, Claude 3 Opus, and Gemini Pro.Scientific reports · 2026Article
- Innovating global regulatory frameworks for generative AI in medical devices is an urgent priority.NPJ digital medicine · 2026Review
- Cardiology knowledge assessment of retrieval-augmented open versus proprietary large language models.PLOS digital health · 2026Article
- Automating clinical phenotyping using natural language processing.Communications medicine · 2026Article
- Fine-tuning and structured prompting strategies for question answering over full-text biomedical research articles.PloS one · 2026Article
Corrections and comments
- Update of
Authors and funding
14 authors.
Funding
Abstract
Large language models (LLMs) hold great promise in summarizing medical evidence. Most recent studies focus on the application of proprietary LLMs. Using proprietary LLMs introduces multiple risk factors, including a lack of transparency and vendor dependency. While open-source LLMs allow better transparency and customization, their performance falls short compared to the proprietary ones. In this study, we investigated to what extent fine-tuning open-source LLMs can further improve their performance. Utilizing a benchmark dataset, MedReview, consisting of 8161 pairs of systematic reviews and summaries, we fine-tuned three broadly-used, open-sourced LLMs, namely PRIMERA, LongT5, and Llama-2. Overall, the performance of open-source models was all improved after fine-tuning. The performance of fine-tuned LongT5 is close to GPT-3.5 with zero-shot settings. Furthermore, smaller fine-tuned models sometimes even demonstrated superior performance compared to larger zero-shot models. The above trends of improvement were manifested in both a human evaluation and a larger-scale GPT4-simulated evaluation.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.