Prompting large language models and evaluating inter- and intra-rater agreement for cancer progression assessment from radiology reports.

Background: Manual annotation of free-text radiology reports is time-consuming and costly, delaying real-world evidence (RWE) studies in oncology. This study aimed to evaluate the performance of large language models (LLMs) in annotating cancer progression from Danish free-text radiology reports. The objectives were to determine whether human-to-LLM inter-rater agreement was non-inferior to human-to-human agreement, establish human intra-rater agreement, and develop a framework for tuning LLM performance to RWE needs. Materials and methods: We identified 376 radiology reports from 184 patients with metastatic breast cancer from Danish electronic health records. Six human annotators, including two experts, classified radiology reports as progressive disease (PD) or non-PD. A 'reverse questioning' strategy was used to evaluate five LLM model series (Mistral, Gemma, Gemma 2, Llama 3, and Llama 3.1). Bootstrapping estimated confidence intervals (CIs) and assessed non-inferiority of the best-performing LLM ensemble compared with human agreement, using a non-inferiority margin of 0.1. Results: The LLM framework was non-inferior to human annotators with a mean Cohen's kappa of 0.82 (95% CI 0.74-0.89) for human-to-LLM versus 0.79 (95% CI 0.71-0.86) for human-to-human agreement ( Conclusions: The proposed LLM framework was non-inferior to human annotators in classifying cancer progression from free-text radiology reports. This offers significant potential for using LLMs as a tool for identifying tumor progression events in clinical assessment and research.