Optimising large language models for clinical information extraction: a benchmarking study in the context of ulcerative colitis research.

Objective: Closed-source large language models (LLMs) like generative pre-trained transformer 4o (GPT-4o) have shown promise for clinical information extraction but are potentially limited by cost, data security concerns and inflexibility. Open-source models are an attractive alternative with various adaptation strategies with no consensus on best practices. This study aims to rigorously identify optimal adaptation strategies for open-source models and evaluate their performance relative to closed-source alternatives. Methods and Analysis: We studied three LLM adaptation strategies: chain-of-thought prompting, few-shot prompting and fine-tuning. Our target for information extraction was the Mayo Endoscopic Subscore (MES). We applied those strategies in all combinations to six open-source models (8-70 billion parameters) using an annotated set of colonoscopy procedure reports from the University of California, San Francisco (N=608) and San Francisco General Hospital (N=217). We analysed the relationship of these strategies to several performance metrics with a mixed-effects model, accounting for the variability between centres and LLMs. GPT-4o served as a closed-source oracle and provided in-depth commentary on the cost-effectiveness of these options. Results: Quantised low-rank adaptation (QLoRA) statistically improves ( Conclusion: GPT-4o is currently the most efficient LLM for MES extraction. If unavailable, QLoRA-optimised open-source models are a competitive alternative. However, results also suggest that current instruction-following LLMs including GPT-4o do not fully follow user-provided instructions, leaving room for improvement. More work is needed to achieve consistent, near-perfect performance in clinical information extraction by LLMs.