Evidence map›Paper›PMID 41137795›Full record

ArticleBritish journal of clinical pharmacology2026

Human vs. artificial intelligence: Physicians outperform ChatGPT in real-world pharmacotherapy counselling.

Benjamin Krichevsky, Stefan Engeli, Stefanie M Bode-Böger, Felix Koop, Martin Schulze Westhoff, Sebastian Schröder, Carsten Schumacher, Thorben Pape, Dirk O Stichtenoth, Johannes Heck

Abstract readComparative Study
PubMed Full text at PMC Publisher
In one paragraph

Article in British journal of clinical pharmacology, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 3 papers.

0numbers the graph read from it
0cells of the map it votes in
3citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

3 citing papers in PubMed.

  1. Article
  2. Article
  3. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

10 authors.

Benjamin KrichevskyHannover Medical School, Institute for General Practice and Palliative Care, Hannover, Germany.ORCID https://orcid.org/0000-0001-5899-0113
Stefan EngeliInstitute for Pharmacology, University Medicine Greifswald, Greifswald, Germany.ORCID https://orcid.org/0000-0002-3087-971X
Stefanie M Bode-BögerInstitute for Clinical Pharmacology, Otto von Guericke University of Magdeburg, Magdeburg, Germany.ORCID https://orcid.org/0000-0002-5132-6591
Felix KoopHannover Medical School, Institute for Clinical Pharmacology, Hannover, Germany.ORCID https://orcid.org/0000-0001-7378-7877
Martin Schulze WesthoffDepartment of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hannover, Germany.ORCID https://orcid.org/0000-0003-3262-8137
Sebastian SchröderDepartment of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hannover, Germany.ORCID https://orcid.org/0000-0003-0905-274X
Carsten SchumacherHannover Medical School, Institute for Clinical Pharmacology, Hannover, Germany.ORCID https://orcid.org/0000-0002-3012-4245
Thorben PapeDepartment of Respiratory Medicine and Infectious Diseases, Hannover Medical School, Hannover, Germany.ORCID https://orcid.org/0000-0003-1031-1913
Dirk O StichtenothHannover Medical School, Institute for Clinical Pharmacology, Hannover, Germany.ORCID https://orcid.org/0000-0002-6526-0917
Johannes HeckHannover Medical School, Institute for Clinical Pharmacology, Hannover, Germany.ORCID https://orcid.org/0000-0002-5382-3014

Funding

DFG DFG ME 3696/3Hannover Medical School
6 · The paper itself

Abstract

aimsTo assess the utility of the artificial intelligence (AI) chatbot ChatGPT (openly available version 3.5) in responding to real-world pharmacotherapeutic queries from healthcare professionals.

methodsThree independent and blinded evaluators with different levels of medical expertise and professional experience (beginner, advanced, and expert) compared AI chatbot- and physician-generated responses to 70 real-world pharmacotherapeutic queries submitted to the clinical-pharmacological drug information centre of Hannover Medical School between June and October 2023 with regard to quality of information, answer preference, answer correctness and quality of language. Inter-rater reliability was assessed with Krippendorff's alpha. Two separate investigators not otherwise involved in the conduct or analysis of the study selected the top three clinically relevant errors in chatbot- and physician-generated responses.

resultsAll three evaluators rated the quality of information of physician-generated responses higher than the quality of information of AI chatbot-generated responses and, accordingly, thought that the physician-generated responses were better than the chatbot-generated responses (answer preference). All evaluators detected factually wrong information more frequently in chatbot-generated responses than in physician-generated responses. Although the beginner and expert evaluators rated the quality of language of physician-generated responses higher than the quality of language of chatbot-generated responses, there was no significant difference according to the advanced evaluator.

conclusionsChatGPT's responses to real-world pharmacotherapeutic queries were substantially inferior compared to conventional physician-generated responses with regard to quality of information and factual correctness. Our study suggests that to date it must be strongly cautioned against the use of ChatGPT in pharmacotherapy counselling.

Indexed as

Artificial IntelligenceCounselingDrug Information ServicesPhysiciansGenerative Artificial IntelligenceHumansReproducibility of Resultsartificial intelligencechatbotChatGPTclinical pharmacologydrug information centrepatient safety

Identifiers

PMID41137795
PMCPMC12930016

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.