Evidence map›Paper›PMID 39150749›Full record

ArticleJMIR medical education2024

A Language Model-Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study.

Friederike Holderried, Christian Stegemann-Philipps, Anne Herrmann-Werner, Teresa Festl-Wietek, Martin Holderried, Carsten Eickhoff, Moritz Mahling

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in JMIR medical education, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 59 papers, 3 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
59citing papers in PubMed, 3 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

59 citing papers in PubMed, 3 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Trial
  5. Trial
  6. Trial
  7. Trial
  8. Trial
  9. Trial
  10. Trial
  11. Trial
  12. Trial
  13. Article
  14. Article
  15. Review
  16. Article
  17. Article
  18. Article
  19. Article
  20. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

7 authors.

Friederike HolderriedTübingen Institute for Medical Education (TIME), Medical Faculty, University of Tübingen, Tübingen, Germany.ORCID 0000-0003-1828-0920
Christian Stegemann-PhilippsTübingen Institute for Medical Education (TIME), Medical Faculty, University of Tübingen, Tübingen, Germany.ORCID 0000-0002-8122-5724
Anne Herrmann-WernerTübingen Institute for Medical Education (TIME), Medical Faculty, University of Tübingen, Tübingen, Germany.ORCID 0000-0003-2413-7047
Teresa Festl-WietekTübingen Institute for Medical Education (TIME), Medical Faculty, University of Tübingen, Tübingen, Germany.ORCID 0000-0003-1450-1757
Martin HolderriedDepartment of Medical Development, Process and Quality Management, University Hospital Tübingen, Tübingen, Germany.ORCID 0000-0002-1337-0516
Carsten EickhoffInstitute for Applied Medical Informatics, University of Tübingen, Tübingen, Germany.ORCID 0000-0001-9895-4061
Moritz MahlingTübingen Institute for Medical Education (TIME), Medical Faculty, University of Tübingen, Tübingen, Germany.ORCID 0000-0001-7960-4015

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

backgroundAlthough history taking is fundamental for diagnosing medical conditions, teaching and providing feedback on the skill can be challenging due to resource constraints. Virtual simulated patients and web-based chatbots have thus emerged as educational tools, with recent advancements in artificial intelligence (AI) such as large language models (LLMs) enhancing their realism and potential to provide feedback.

objectiveIn our study, we aimed to evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model to provide structured feedback on medical students' performance in history taking with a simulated patient.

methodsWe conducted a prospective study involving medical students performing history taking with a GPT-powered chatbot. To that end, we designed a chatbot to simulate patients' responses and provide immediate feedback on the comprehensiveness of the students' history taking. Students' interactions with the chatbot were analyzed, and feedback from the chatbot was compared with feedback from a human rater. We measured interrater reliability and performed a descriptive analysis to assess the quality of feedback.

resultsMost of the study's participants were in their third year of medical school. A total of 1894 question-answer pairs from 106 conversations were included in our analysis. GPT-4's role-play and responses were medically plausible in more than 99% of cases. Interrater reliability between GPT-4 and the human rater showed "almost perfect" agreement (Cohen κ=0.832). Less agreement (κ<0.6) detected for 8 out of 45 feedback categories highlighted topics about which the model's assessments were overly specific or diverged from human judgement.

conclusionsThe GPT model was effective in providing structured feedback on history-taking dialogs provided by medical students. Although we unraveled some limitations regarding the specificity of feedback for certain feedback categories, the overall high agreement with human raters suggests that LLMs can be a valuable tool for medical education. Our findings, thus, advocate the careful integration of AI-driven feedback mechanisms in medical training and highlight important aspects when LLMs are used in that context.

Indexed as

Medical History TakingPatient SimulationStudents, MedicalArtificial IntelligenceClinical CompetenceEducation, Medical, UndergraduateFeedbackFemaleHumansMaleProspective StudiesReproducibility of Resultsartificial intelligencechatbotchatbotsChatGPTcommunicationcommunication skillsconversational agentconversational agentsGPT: LLMhistorieshistoryinteractioninteractionslanguage modellanguage modelsLLMsmachine learningmedical educationnatural language processingNLPrelationshiprelationshipssimulatedstudentstudentstechnology enhanced educationTELvirtual patients communication

Identifiers

PMID39150749
PMCPMC11364946

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.