ArticleNPJ digital medicine2025
Evaluating evidence-based health information from generative AI using a cross-sectional study with laypeople seeking screening information.
Article in NPJ digital medicine, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 20 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
20 citing papers in PubMed.
- What information is available online for adults with distal radius fracture? Online literature search and synthesis using generative AI.Hand therapy · 2026Article
- Feasibility of Tailoring Artificial Intelligence-Assisted Ambient Scribes for Intensive Care Unit Rounds: Algorithm Development and Validation.JMIR medical informatics · 2026Article
- Leveraging Large Language Models to Address Common Vaccination Myths and Misconceptions.Vaccines · 2026Article
- Evaluating the quality, reliability and readability of digital and artificial intelligence resources for adults with cancer who have significant caregiving responsibilities for children.PLOS digital health · 2026Article
- "MELMA" in otolaryngology: Medical evaluation of large language model answers. Clinician-rated scoring (MELMA-Q) and web-based auditing (MELMA-W) novel tools for AI assessment.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- Evaluating the Quality of Health Information: Comparison of Human and Artificial Intelligence.Neurogastroenterology and motility · 2026Article
- Differences in Safety Risks Across Languages in Health-Relevant Queries: Vulnerability Analysis of Large Language Model Responses.JMIR formative research · 2026Article
- AI-based diagnostic evaluation of GPT-4o for crown-fracture detection on maxillary periapical radiographs: effects of prompt detail and customization.BMC oral health · 2026Article
- A Minimum Safety Case for Record-Connected Consumer Health Assistants.Journal of medical systems · 2026Article
- Co-Creation of Interventions to Promote Critical Health Literacy in the Community: Study Protocol.Health expectations : an international journal of public participation in health care and health policy · 2026Article
- Comparing five generative AI chatbots' answers to LLM-generated clinical questions with medical information scientists' evidence summaries.Journal of the Medical Library Association : JMLA · 2026Article
- Prospective Evaluation of Large Language Model Integration Into a Classical Hematology Case Conference.JMIR formative research · 2026Article
- Review
- LLM-as-a-judge for infection prevention and control and antimicrobial resistance impact: comparing three main LLMs vs. human experts' assessment.Frontiers in public health · 2026Article
- Benchmarking publicly accessible large language models for English-language patient-facing acute pancreatitis information: a cross-sectional study of quality, transparency, and readability.Frontiers in public health · 2026Article
- Information Quality Requirements for AI-Generated Patient Education Materials in Diabetic Foot Ulcer Care: A Directed Qualitative Content Analysis of Patient and Nurse Perspectives.Patient preference and adherence · 2026Article
- Patient-facing diabetic foot information from large language models: a domain- and source-balanced prompt framework for public-interface benchmarking.Frontiers in endocrinology · 2026Article
- Mapping applications and evaluations of LLM-enabled AI chatbots for health purposes: a scoping review.Frontiers in public health · 2026Article
- Comparing Five Generative AI Chatbots' Answers to LLM-Generated Clinical Questions with Medical Information Scientists' Evidence Summaries.medRxiv : the preprint server for health sciences · 2025Article
- In the search for the perfect prompt in medical AI queries.Frontiers in artificial intelligence · 2025Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Large language models (LLMs) are used to seek health information. Guidelines for evidence-based health communication require the presentation of the best available evidence to support informed decision-making. We investigate the prompt-dependent guideline compliance of LLMs and evaluate a minimal behavioural intervention for boosting laypeople's prompting. Study 1 systematically varied prompt informedness, topic, and LLMs to evaluate compliance. Study 2 randomized 300 participants to three LLMs under standard or boosted prompting conditions. Blinded raters assessed LLM response with two instruments. Study 1 found that LLMs failed evidence-based health communication standards. The quality of responses was found to be contingent upon prompt informedness. Study 2 revealed that laypeople frequently generated poor-quality responses. The simple boost improved response quality, though it remained below required standards. These findings underscore the inadequacy of LLMs as a standalone health communication tool. Integrating LLMs with evidence-based frameworks, enhancing their reasoning and interfaces, and teaching prompting are essential. Study Registration: German Clinical Trials Register (DRKS) (Reg. No.: DRKS00035228, registered on 15 October 2024).
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.