ArticleDigital health
Safety and quality of AI chatbots for drug-related inquiries: A real-world comparison with licensed pharmacists.
Article in Digital health. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 16 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
16 citing papers in PubMed.
- Navigating uncertainty matters: Evaluating large language models for drug-drug interaction identification.Journal of managed care & specialty pharmacy · 2026Article
- Article
- Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: Quantitative Pilot Feasibility Study.JMIR formative research · 2026Article
- Evaluating Consistency and Accuracy of GPT-4 Omni to Analyze Thyroid Ultrasound Features and ACR TR Categories to Aid Report GenerationCurrent medical imaging · 2026Article
- Drug-drug interaction identification using large language models.medRxiv : the preprint server for health sciences · 2025Article
- Article
- Comparative evaluation of artificial intelligence platforms and drug interaction screening databases using real-world patient data.Exploratory research in clinical and social pharmacy · 2025Article
- The role of large language models in self-care: a study and benchmark on medicines and supplement guidance accuracy.International journal of clinical pharmacy · 2025Article
- Benchmarking ChatGPT-3.5 and OpenAI o3 Against Clinical Pharmacists: Preliminary Insights into Clinical Accuracy, Sensitivity, and Specificity in Pharmacy MCQs.Healthcare (Basel, Switzerland) · 2025Article
- AI-driven healthcare innovations for enhancing clinical services during mass gatherings (Hajj): task force insights and future directions.BMC health services research · 2025Article
- A scoping review on generative AI and large language models in mitigating medication related harm.NPJ digital medicine · 2025Article
- Diagnostic performance of four AI tools in pharmacology MCQs: Accuracy, sensitivity, and specificity.PloS one · 2025Article
- Advancing drug-drug interactions research: integrating AI-powered prediction, vulnerable populations, and regulatory insights.Frontiers in pharmacology · 2025Review
- Evaluating remote healthcare practices: Experiences and recommendations of healthcare professionals on smart applications.Digital healthArticle
- Exploring evaluation measures of large language models for family caregiver use: A scoping review.Digital healthReview
- Exploring the opportunities of large language models for summarizing palliative care consultations: A pilot comparative study.Digital healthArticle
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
8 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Introduction: Pharmacists play a pivotal role in ensuring patients are administered safe and effective medications; however, they encounter obstacles such as elevated workloads and a scarcity of qualified professionals. Despite the prospective utility of large language models (LLMs), such as Generative Pre-trained Transformers (GPTs), in addressing pharmaceutical inquiries, their applicability in real-world cases remains unexplored. Objective: To evaluate GPT-based chatbots' accuracy in real-world drug-related inquiries, comparing their performance to licensed pharmacists. Methods: In this cross-sectional study, authors analyzed real-world drug inquiries from a Drug Information Inquiry Database. Two independent pharmacists evaluated the performance of GPT-based chatbots (GPT-3, GPT-3.5, GPT-4) against human pharmacists using accuracy, detail, and risk of harm criteria. Descriptive statistics described inquiry characteristics. Absolute proportion comparative analyses assessed accuracy, detail, and risk of harm. Stratified analyses were performed for different inquiry types. Results: Seventy inquiries were included. Most inquiries were received from physicians (41%) and pharmacists (44%). Inquiries type included dosage/administration (34.2%), drug interaction (12.8%) and pregnancy/lactation (15.7%). Majority of inquires included adults (83%) and female patients (54.3%). GPT-4 had 64.3% completely accurate responses, comparable to human pharmacists. GPT-4 and human pharmacists provided sufficiently detailed responses, with GPT-4 offering additional relevant details. Both GPT-4 and human pharmacists delivered 95% safe responses; however, GPT-4 provided proactive risk mitigation information in 70% of the instances, whereas similar information was included in 25.7% of human pharmacists' responses. Conclusion: Our study showcased GPT-4's potential in addressing drug-related inquiries accurately and safely, comparable to human pharmacists. Current GPT-4-based chatbots could support healthcare professionals and foster global health improvements.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.