ArticleDrug, healthcare and patient safety2023
Evaluating the Sensitivity, Specificity, and Accuracy of ChatGPT-3.5, ChatGPT-4, Bing AI, and Bard Against Conventional Drug-Drug Interactions Clinical Tools.
Article in Drug, healthcare and patient safety, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 54 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
54 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Large Language Models for Chatbot Health Advice Studies: A Systematic Review.JAMA network open · 2025Pooled it
- Comparative performance of large language models and Drugs.com versus Lexicomp for antiseizure medication drug-drug interactions: A cross-sectional study with iterative prompting analysis.Exploratory research in clinical and social pharmacy · 2026Article
- Performance of Large Language Models in Answering Healthcare Delivery Questions: A Quantitative Cross-Sectional Study.Health science reports · 2026Article
- Prevalence, predictors, and clinical relevance of drug-drug interactions in outpatient prescribing: A national cross-sectional study.PloS one · 2026Article
- Applications of Chatbots in Improving Patient Care Outcomes:Sultan Qaboos University medical journal · 2026Article
- Evaluating a large language model (ChatGPT-5) for detecting potential drug-drug interactions in intensive care: a cross-sectional comparative study with a clinical decision support system.Frontiers in pharmacology · 2026Article
- Closing the loop: human-augmented, mechanistically enhanced AI for proactive management of drug-drug interactions.Frontiers in pharmacology · 2026Article
- Drug-drug interaction identification using large language models.medRxiv : the preprint server for health sciences · 2025Article
- Pharm D student's Knowledge, perception, and practice of CHAT-GPT in clinical training: a web-based cross-sectional survey in India.Scientific reports · 2025Article
- Article
- The application and challenges of ChatGPT in laboratory medicine.Advances in laboratory medicine · 2025Review
- Comparative evaluation of artificial intelligence platforms and drug interaction screening databases using real-world patient data.Exploratory research in clinical and social pharmacy · 2025Article
- HIV Prevention and Treatment Information from Four Artificial Intelligence Platforms: A Thematic Analysis.AIDS and behavior · 2025Article
- Evaluation of the Accuracy and Reliability of Responses Generated by Artificial Intelligence Related to Clinical Pharmacology.Journal of clinical medicine · 2025Article
- ReviewGenie: a novel automated system for systematic reviews-an exploratory study in speech and language disorders.Systematic reviews · 2025Article
- Comparative Analysis of Generative Artificial Intelligence Systems in Solving Clinical Pharmacy Problems: Mixed Methods Study.JMIR medical informatics · 2025Article
- Evaluation of ChatGPT-4 as an Online Outpatient Assistant in Puerperal Mastitis Management: Content Analysis of an Observational Study.JMIR medical informatics · 2025Observational
- Clinical applications of large language models in medicine and surgery: A scoping review.The Journal of international medical research · 2025Article
- AI-driven healthcare innovations for enhancing clinical services during mass gatherings (Hajj): task force insights and future directions.BMC health services research · 2025Article
- Performance of the Large Language Models in African rheumatology: a diagnostic test accuracy study of ChatGPT-4, Gemini, Copilot, and Claude artificial intelligence.BMC rheumatology · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Background: AI platforms are equipped with advanced algorithms that have the potential to offer a wide range of applications in healthcare services. However, information about the accuracy of AI chatbots against conventional drug-drug interaction tools is limited. This study aimed to assess the sensitivity, specificity, and accuracy of ChatGPT-3.5, ChatGPT-4, Bing AI, and Bard in predicting drug-drug interactions. Methods: AI-based chatbots (ie, ChatGPT-3.5, ChatGPT-4, Microsoft Bing AI, and Google Bard) were compared for their abilities to detect clinically relevant DDIs for 255 drug pairs. Descriptive statistics, such as specificity, sensitivity, accuracy, negative predictive value (NPV), and positive predictive value (PPV), were calculated for each tool. Results: When a subscription tool was used as a reference, the specificity ranged from a low of 0.372 (ChatGPT-3.5) to a high of 0.769 (Microsoft Bing AI). Also, Microsoft Bing AI had the highest performance with an accuracy score of 0.788, with ChatGPT-3.5 having the lowest accuracy rate of 0.469. There was an overall improvement in performance for all the programs when the reference tool switched to a free DDI source, but still, ChatGPT-3.5 had the lowest specificity (0.392) and accuracy (0.525), and Microsoft Bing AI demonstrated the highest specificity (0.892) and accuracy (0.890). When assessing the consistency of accuracy across two different drug classes, ChatGPT-3.5 and ChatGPT-4 showed the highest variability in accuracy. In addition, ChatGPT-3.5, ChatGPT-4, and Bard exhibited the highest fluctuations in specificity when analyzing two medications belonging to the same drug class. Conclusion: Bing AI had the highest accuracy and specificity, outperforming Google's Bard, ChatGPT-3.5, and ChatGPT-4. The findings highlight the significant potential these AI tools hold in transforming patient care. While the current AI platforms evaluated are not without limitations, their ability to quickly analyze potentially significant interactions with good sensitivity suggests a promising step towards improved patient safety.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.