ArticleCureus2023
Bias and Inaccuracy in AI Chatbot Ophthalmologist Recommendations.
Article in Cureus, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 18 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
18 citing papers in PubMed, 1 synthesis or guideline pooled it, 40 citations in OpenAlex.
- Evaluating Large Language Models in Ophthalmology: Systematic Review.Journal of medical Internet research · 2025Pooled it
- Article
- A randomized factorial experiment to optimize the design of a culturally tailored breast cancer screening outreach chatbot intervention.Frontiers in digital health · 2026Article
- An AI-Based Chatbot to Support Health-Related Social Needs among Pediatric Primary Care Population: Protocol for a Pilot Randomized Controlled Trial.medRxiv : the preprint server for health sciences · 2025Article
- Evaluating the performance of AI chatbots in responding to dental implant FAQs: A comparative study.BMC oral health · 2025Article
- Framework for bias evaluation in large language models in healthcare settings.NPJ digital medicine · 2025Article
- Chat GPT vs an experienced ophthalmologist: evaluating chatbot writing performance in ophthalmology.Eye (London, England) · 2025Article
- ChatGPT's Attitude, Knowledge, and Clinical Application in Geriatrics Practice and Education: Exploratory Observational Study.JMIR formative research · 2025Observational
- Accuracy of an artificial intelligence chatbot in identifying congenital glaucoma from other ocular etiologies.Digital journal of ophthalmology : DJO · 2025Article
- Artificial intelligence in plastic surgery, where do we stand?JPRAS open · 2024Review
- Ethical Considerations in Human-Centered AI: Advancing Oncology Chatbots Through Large Language Models.JMIR bioinformatics and biotechnology · 2024Article
- Mixed methods assessment of the influence of demographics on medical advice of ChatGPT.Journal of the American Medical Informatics Association : JAMIA · 2024Article
- Fairness in AI-Driven Oncology: Investigating Racial and Gender Biases in Large Language Models.Cureus · 2024Article
- Evaluation of ChatGPT responses for back pain?European journal of orthopaedic surgery & traumatology : orthopedie traumatologie · 2024Article
- Leveraging human-centered design and causal pathway diagramming toward enhanced specification and development of innovative implementation strategies: a case example of an outreach tool to address racial inequities in breast cancer screening.Implementation science communications · 2024Article
- A Preliminary Checklist (METRICS) to Standardize the Design and Reporting of Studies on Generative Artificial Intelligence-Based Models in Health Care Education and Practice: Development Study Involving a Literature Review.Interactive journal of medical research · 2024Article
- ChatGPT Performance in Diagnostic Clinical Microbiology Laboratory-Oriented Case Scenarios.Cureus · 2023Article
- Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors at 4 institutions in 1 country.
Funding
No grant is acknowledged in the PubMed record.
Abstract
PURPOSE AND
designTo evaluate the accuracy and bias of ophthalmologist recommendations made by three AI chatbots, namely ChatGPT 3.5 (OpenAI, San Francisco, CA, USA), Bing Chat (Microsoft Corp., Redmond, WA, USA), and Google Bard (Alphabet Inc., Mountain View, CA, USA). This study analyzed chatbot recommendations for the 20 most populous U.S. cities.
methodsEach chatbot returned 80 total recommendations when given the prompt "Find me four good ophthalmologists in (city)." Characteristics of the physicians, including specialty, location, gender, practice type, and fellowship, were collected. A one-proportion z-test was performed to compare the proportion of female ophthalmologists recommended by each chatbot to the national average (27.2% per the Association of American Medical Colleges (AAMC)). Pearson's chi-squared test was performed to determine differences between the three chatbots in male versus female recommendations and recommendation accuracy.
resultsFemale ophthalmologists recommended by Bing Chat (1.61%) and Bard (8.0%) were significantly less than the national proportion of 27.2% practicing female ophthalmologists (p<0.001, p<0.01, respectively). ChatGPT recommended fewer female (29.5%) than male ophthalmologists (p<0.722). ChatGPT (73.8%), Bing Chat (67.5%), and Bard (62.5%) gave high rates of inaccurate recommendations. Compared to the national average of academic ophthalmologists (17%), the proportion of recommended ophthalmologists in academic medicine or in combined academic and private practice was significantly greater for all three chatbots.
conclusionThis study revealed substantial bias and inaccuracy in the AI chatbots' recommendations. They struggled to recommend ophthalmologists reliably and accurately, with most recommendations being physicians in specialties other than ophthalmology or not in or near the desired city. Bing Chat and Google Bard showed a significant tendency against recommending female ophthalmologists, and all chatbots favored recommending ophthalmologists in academic medicine.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.