Utilization of Image-Based Deep Learning in Multimodal Glaucoma Detection Neural Network from a Primary Patient Cohort.

Purpose: To develop a clinically motivated multimodal neural network glaucoma detection model trained on minimally processed imaging data of time-matched multimodal testing including fundus photographs, OCT scans, and Humphrey visual field (HVF) analysis. Design: Evaluation of a diagnostic technology. Subjects: A total of 716 encounters with time-matched fundus photographs, OCT optic nerve imaging, and HVF testing from 706 eyes (557 nonglaucomatous, 149 glaucomatous) from 571 individual patients seen at a tertiary medical center and 4 external single-modality (fundus photograph and OCT) datasets. Methods: A multimodal neural network model was developed consisting of 2 main components: first, 3 convolutional neural networks to extract semantic features and generate embeddings for each respective modality, followed by a second component consisting of a multilayer perceptron to integrate the individual embeddings and produce a predicted label, glaucomatous or nonglaucomatous. Main Outcome Measures: Single and multimodal performances were evaluated on the internal test set using the area under the receiver operating characteristic curve (AUC), accuracy, recall, and specificity. Fundus photograph and OCT single-modality neural networks were additionally evaluated on external datasets by these metrics. Results: Our results show single-modality models with high performance on curated training datasets perform inferiorly on our primary clinical dataset. Performance metrics, however, can be notably improved through multimodal integration (AUC: 0.86 from 0.57 to 0.74 and specificity: 0.85 from 0.77 to 0.82), suggesting that a holistic approach considering both structural and functional data may enhance the functionality and accuracy of artificial intelligence (AI) model. Conclusions: Clinical implementation of deep learning models for glaucoma detection benefits from multimodal integration, and we demonstrate this approach on a true clinical cohort to obtain a production-level AI solution for glaucoma diagnosis. Financial Disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.