A comparative performance analysis of regular expressions and a large language model-based approach to extract the BI-RADS score from radiological reports.

Background: Different natural language processing (NLP) techniques have demonstrated promising results for data extraction from radiological reports. Both traditional rule-based methods like regular expressions (Regex) and modern large language models (LLMs) can extract structured information. However, comparison between these approaches for extraction of specific radiological data elements has not been widely conducted. Methods: We compared accuracy and processing time between Regex and LLM-based approaches for extracting Breast Imaging-Reporting and Data System (BI-RADS) scores from 7764 radiology reports (mammography, ultrasound, MRI [magnetic resonance imaging], and biopsy). We developed a rule-based algorithm using Regex patterns and implemented an LLM-based extraction using the Rombos-LLM-V2.6-Qwen-14b model. A ground truth dataset of 199 manually classified reports was used for evaluation. Results: We did not detect a statistically significant difference in the accuracy in extracting BI-RADS scores between Regex and an LLM-based method (accuracy of 89.20% for Regex vs 87.69% for the LLM-based method; Conclusion: For structured, standardized data like BI-RADS, traditional NLP techniques seem to be superior, though future work should explore hybrid approaches combining Regex precision for standardized elements with LLM contextual understanding for more complex information extraction tasks.