RadCLARE: an automated clinical language engine for detecting semantic errors in radiology reports.

backgroundErrors in radiology reports can result in inappropriate/harmful decisions. We investigated whether large language models can reduce the error rate. MATERIALS AND

methodsWe developed the radiology-specific clinical language anomaly recognition engine (RadCLARE) network, an automated engine based on the bidirectional encoder representations from transformers (BERT)-base model, designed to detect semantic errors in Chinese radiology reports and trained using 1.4 million reports, including 615,920 digital radiography, 560,310 computed tomography reports, and 223,480 magnetic resonance reports. One thousand reports were randomly selected for expert manual annotation. Inter-reader agreement for error detection and classification was assessed using Cohen κ and Gwet AC1. The RadCLARE's detection was compared against the expert references. Changes in error rates before (baseline test dataset, BTD) and after (experimental test dataset, ETD) RadCLARE implementation were analyzed. Finally, radiologists were invited to complete questionnaires to evaluate satisfaction and rate the system across five dimensions.

resultsAmong the 1,000 reports, a total of 506 errors were identified as the reference standard. Inter-reader agreement was substantial for error detection (κ = 0.77) and excellent for error classification (Gwet AC1 = 0.94). RadCLARE successfully detected 437/506 errors, with 87.3% accuracy, 88.3% precision, 86.4% recall, and 87.4% F1-score. The BTD comprised 571,264 reports, the ETD 873,030 reports. After RadCLARE implementation, the semantic error rate dropped significantly compared to the BTD (error rate, 0.85% [7408/873,030] versus 4.19% [23,909/571,264]; p < 0.001). The questionnaire results showed that 95.7% (44/46) of radiologists were satisfied with RadCLARE.

conclusionRadCLARE showed the capability for automatic detection of semantic errors in radiology reports. RELEVANCE STATEMENT: RadCLARE demonstrated high performance in detecting semantic errors in radiology reports. Future studies should aim to extend their applicability across multiple languages and institutions. KEY POINTS: We developed the RadCLARE network, a BERT-based engine for detecting semantic errors in Chinese radiology reports. With the aid of RadCLARE, the semantic error rate in radiology reports dropped significantly from 4.19% to 0.85%. The large majority (96%) of radiologists who participated in the test were satisfied with the RadCLARE and felt that it reduced stress.