ArticleJMIR mHealth and uHealth2024
The Evaluation of Generative AI Should Include Repetition to Assess Stability.
Article in JMIR mHealth and uHealth, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 15 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
15 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Clinical applications of large language models in knee osteoarthritis: a systematic review.Frontiers in medicine · 2025Pooled it
- ChatGPT improves readability in validated spine patient-reported outcome measures.North American Spine Society journal · 2026Article
- Impact of guideline-based prompting on the large language model performance in dental trauma management clinical decision-making.Odontology · 2026Article
- Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing.Journal of the American Medical Informatics Association : JAMIA · 2026Article
- Article
- Tailoring Discharge Summaries to Health Care Providers' Needs (Part 1 of the Framework and Implementation of AI Tools Project): User-Centered Design Approach.JMIR medical informatics · 2026Article
- From risk classification to clinical action: a prespecified paired pilot benchmark of public large language model interfaces for diabetes-related foot ulcer prevention.Frontiers in endocrinology · 2026Article
- Re: Pushpanathan et al.: Can OpenAI's New o1 Model Outperform Its Predecessors in Common Eye Care Queries?Ophthalmology science · 2026Article
- Emotion recognition in cross-linguistic legal context: a comparison between human-based and computational approaches.Frontiers in psychology · 2026Article
- Rethinking AI Workflows: Guidelines for Scientific Evaluation in Digital Health Companies.JMIR AI · 2025Article
- Evaluation of the readability, quality, and accuracy of AI chatbot responses to questions about deleterious oral habits.BMC oral health · 2025Article
- Large language models in clinical trials: applications, technical advances, and future directions.BMC medicine · 2025Review
- ChatGPT for Univariate Statistics: Validation of AI-Assisted Data Analysis in Healthcare Research.Journal of medical Internet research · 2025Article
- A Qualitative Evaluation of ChatGPT4 and PaLM2's Response to Patient's Questions Regarding Age-Related Macular Degeneration.Diagnostics (Basel, Switzerland) · 2024Article
- Article
Corrections and comments
- Comment on
Authors and funding
9 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
The increasing interest in the potential applications of generative artificial intelligence (AI) models like ChatGPT in health care has prompted numerous studies to explore its performance in various medical contexts. However, evaluating ChatGPT poses unique challenges due to the inherent randomness in its responses. Unlike traditional AI models, ChatGPT generates different responses for the same input, making it imperative to assess its stability through repetition. This commentary highlights the importance of including repetition in the evaluation of ChatGPT to ensure the reliability of conclusions drawn from its performance. Similar to biological experiments, which often require multiple repetitions for validity, we argue that assessing generative AI models like ChatGPT demands a similar approach. Failure to acknowledge the impact of repetition can lead to biased conclusions and undermine the credibility of research findings. We urge researchers to incorporate appropriate repetition in their studies from the outset and transparently report their methods to enhance the robustness and reproducibility of findings in this rapidly evolving field.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.