ArticleBMC infectious diseases2025
Comparison of the accuracy and reliability of ChatGPT-4o and Gemini in answering HIV-related questions.
Article in BMC infectious diseases, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 4 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
4 citing papers in PubMed.
- Artificial Intelligence in Plastic Surgery Education: A Global Multimodel Benchmark of Large Language Models on the Plastic Surgery In-Service Training Examination.Aesthetic surgery journal. Open forum · 2026Article
- Performance comparison of large language models for medication counseling in people living with HIV.Frontiers in public health · 2026Article
- Artificial intelligence in HIV research: a structured review and task-oriented clinical framework.Frontiers in digital health · 2026Review
- Evaluation of generative AI-driven chatbots as sources of consumer health information on hand, foot, and mouth disease: a cross-sectional comparative study of safety, accuracy, information quality, readability, and empathy.Frontiers in public health · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
2 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundLarge language models (LLMs) such as ChatGPT and Gemini are increasingly being used to obtain health information, including topics such as HIV. This study aims to comparatively evaluate the accuracy, reliability, and reproducibility of ChatGPT and Gemini in answering HIV-related questions obtained from official public health sources, clinical guidelines, and social media.
methodsA total of 156 HIV-related questions were asked to ChatGPT-4o and Google Gemini 1.5 Flash across three categories: questions derived from the United States Centers for Disease Control and Prevention (CDC) resources (44.2%, n = 69), guidelines (30.8%, n = 48), and social media (25.0%, n = 39). Responses were rated on a 4-point scale (1 = completely wrong, 4 = completely correct) by two infectious disease specialists. The reproducibility of both LLMs was also evaluated.
resultsThe median score (IQR) of the answers generated for all questions was 4.00 (0.00) for ChatGPT and 4.00 (1.00) for Gemini (p = 0.051). The rate of completely correct answers was 81.4% for ChatGPT and 71.8% for Gemini (p = 0.045). ChatGPT demonstrated significantly lower accuracy in guideline-based questions (47.9%) than in CDC-related (97.1%) and social media-derived (94.9%) questions (p < 0.001 for both). Similarly, Gemini demonstrated significantly lower accuracy in guideline-based questions (35.4%) compared to CDC-related (88.4%) and social media-derived (87.2%) questions (p < 0.001 for both). Considering the questions according to the topics, the lowest accuracy rate for both LLMs was in the subject of ‘Prevention and Treatment’ (67.2% for ChatGPT, 54.7% for Gemini). The reproducibility of the answers was 94.8% for ChatGPT and 90.3% for Gemini.
conclusionChatGPT and Gemini, answered CDC- and social media–based questions with high accuracy. However, both LLMs showed lower accuracy for guideline-based and “Prevention and Treatment” questions. These findings suggest that while such models may provide useful general information, they are not yet reliable for clinical decision-making, and their outputs should be verified against evidence-based clinical guidelines.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.