Multimodal large language models in ultrasound diagnosis of breast masses: a multicenter comparative analysis based on GPT-4o, radiologists, and convolutional neural network (CNN).

Background: An advanced version of large language model (LLM), ChatGPT 4o (GPT-4o), has shown capacity in image-text pair interpretation, yet the performance in medical image analysis remains unclear. This study aimed to evaluate the diagnostic capacity of GPT-4o in breast ultrasound (US) datasets. Methods: US exams including breast images and original reports were respectively included from January 2021 to December 2023 in three hospitals throughout China. The diagnostic performance in distinguishing benign or malignant breast masses of GPT-4o was assessed through two approaches: image-strategy and image-combined-text-strategy. Fleiss kappa was calculated to determine intra-LLM consistency. Thereafter, diagnostic accuracy was evaluated and compared with the convolutional neural network (CNN) model and 95 human experts with various levels of expertise from 60 institutions in China. Responses from GPT-4o were rated by diagnostic confidence and radiologist's evaluation. Results: The observations of 80 breast masses (37 malignant, 43 benign) from 80 patients [median age, 42.5 years; interquartile range (IQR), 37.0-53.0 years] were enrolled. GPT-4o with image-strategy exhibited a fair consistency [0.25, 95% confidence interval (CI): 0.07-0.43], whereas the agreement of image-combined-text-strategy was excellent (0.81, 95% CI: 0.67-0.91). Diagnostic accuracy improved when deploying the image-combined-text-strategy compared to only image [58% (46 of 80) Conclusions: The effectiveness of GPT-4o in interpreting real-world US images was limited, yet improved in image-combined-text-strategy. Deploying LLM warrants scrutiny from radiologists.