ArticleBMC public health2026
Evaluating the reliability and information quality of ChatGPT responses on vaccine hesitancy: an expert panel study.
Article in BMC public health, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Not yet cited in PubMed.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
0 citing papers in PubMed.
No citing paper in PubMed yet.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundVaccine hesitancy is a major public health problem that threatens global immunisation programmes. The use of artificial intelligence-based chatbots to access health information is increasing. This study aimed to evaluate the responses of ChatGPT-4o to frequently asked questions about vaccine hesitancy in terms of health information quality, based on expert opinion.
methodsThis cross-sectional expert evaluation study included 20 questions on vaccine hesitancy. These questions were submitted to ChatGPT-4o, and the responses were evaluated by nine experts from the fields of public health, paediatrics, and infectious diseases. The evaluation was based on six criteria synthesised from HONcode, DISCERN, JAMA Benchmarks, CRAAP, GQS, and QUEST: scientific accuracy, comprehensiveness, understandability, correction of misinformation, source attribution, and actionability. Each criterion was scored on a scale from 1 to 10. Inter-rater reliability was assessed using the intraclass correlation coefficient (ICC) and, as a prevalence-robust sensitivity analysis, Gwet's AC2 with quadratic weights.
resultsThe highest mean scores were found for understandability (9.4 ± 1.1) and scientific accuracy (8.9 ± 1.3), while the lowest mean score was found for source attribution (4.5 ± 3.4). At the question level, the highest scores were obtained for Question 4, on aluminium in vaccines, and Question 14, on claims that pharmaceutical companies endanger children's health by producing and promoting vaccines. The lowest scores were obtained for Question 5, on live vaccines during pregnancy, and Question 6, on natural immunity. ICC analysis showed moderate agreement only for source attribution (ICC = 0.582; p < 0.001); the low ICC values for the other criteria were attributable to a ceiling effect, as confirmed by Gwet's AC2, which indicated substantially higher agreement for the high-scoring criteria.
conclusionsChatGPT-4o can generate scientifically accurate and understandable responses to questions about vaccine hesitancy. However, it shows a systematic shortcoming in source attribution. Although the model has potential as a supportive tool in public health communication, its outputs should be reviewed by experts and supported with verifiable sources.
Indexed as
Identifiers
42482189What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.