ArticlePloS one2019
Reproduction study using public data of: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.
Article in PloS one, 2019. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 52 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
52 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Machine Learning Approaches in High Myopia: Systematic Review and Meta-Analysis.Journal of medical Internet research · 2025Pooled it
- Performance and Limitation of Machine Learning Algorithms for Diabetic Retinopathy Screening: Meta-analysis.Journal of medical Internet research · 2021Pooled it
- Generative Modeling and Multispectral Imaging for Eye Fundus Classification.Medical sciences (Basel, Switzerland) · 2026Article
- An explainable ensemble for diabetic retinopathy grading with a novel confidence quality factor and configurable heatmaps.Medical & biological engineering & computing · 2026Article
- DPLSeg: unsupervised general segmentation model for retinal images across multiple OCT devices.Biomedical optics express · 2026Article
- ORDER-DR: external validation of severity grading and referable-risk stratification from fundus images.Frontiers in endocrinology · 2026Article
- Feature-reconstructed diabetic retinopathy classification using variational autoencoder with disentanglement factor.Frontiers in medical technology · 2026Article
- Enhanced performance in automated diabetic retinopathy diagnosis achieved through Voronoi diagrams and artificial intelligence.Scientific reports · 2025Article
- Interpreting Deep Neural Networks in Diabetic Retinopathy Grading: A Comparison with Human Decision Criteria.Life (Basel, Switzerland) · 2025Article
- Improving diabetic retinopathy screening using artificial intelligence: design, evaluation and before-and-after study of a custom development.Frontiers in digital health · 2025Article
- Graph-enhanced multimodal fusion of vascular biomarkers and deep features for diabetic retinopathy detection.Frontiers in artificial intelligence · 2025Article
- A preliminary study of artificial intelligence to recognize tessellated fundus in visual function screening of 7-14 year old students.BMC ophthalmology · 2024Article
- Data set terminology of deep learning in medicine: a historical review and recommendation.Japanese journal of radiology · 2024Review
- Advancing healthcare with artificial intelligence: diagnostic accuracy of machine learning algorithm in diagnosis of diabetic retinopathy in the Brazilian population.Diabetology & metabolic syndrome · 2024Article
- Macular edema in retinal fundus images by a computational algorithmRevista medica del Instituto Mexicano del Seguro Social · 2024Article
- Vision transformer with masked autoencoders for referable diabetic retinopathy classification based on large-size retina image.PloS one · 2024Article
- Article
- Using Deep Learning Architectures for Detection and Classification of Diabetic Retinopathy.Sensors (Basel, Switzerland) · 2023Article
- Use of Electronic Medical Records (EMR) in Gerontology: Benefits, Considerations and a Promising Future.Clinical interventions in aging · 2023Review
- Bibliometric analysis of artificial intelligence and optical coherence tomography images: research hotspots and frontiers.International journal of ophthalmology · 2023Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
We have attempted to reproduce the results in Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs, published in JAMA 2016; 316(22), using publicly available data sets. We re-implemented the main method in the original study since the source code is not available. The original study used non-public fundus images from EyePACS and three hospitals in India for training. We used a different EyePACS data set from Kaggle. The original study used the benchmark data set Messidor-2 to evaluate the algorithm's performance. We used another distribution of the Messidor-2 data set, since the original data set is no longer available. In the original study, ophthalmologists re-graded all images for diabetic retinopathy, macular edema, and image gradability. We have one diabetic retinopathy grade per image for our data sets, and we assessed image gradability ourselves. We were not able to reproduce the original study's results with publicly available data. Our algorithm's area under the receiver operating characteristic curve (AUC) of 0.951 (95% CI, 0.947-0.956) on the Kaggle EyePACS test set and 0.853 (95% CI, 0.835-0.871) on Messidor-2 did not come close to the reported AUC of 0.99 on both test sets in the original study. This may be caused by the use of a single grade per image, or different data. This study shows the challenges of reproducing deep learning method results, and the need for more replication and reproduction studies to validate deep learning methods, especially for medical image analysis. Our source code and instructions are available at: https://github.com/mikevoets/jama16-retina-replication.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.