ArticleCureus2024
Accuracy and Repeatability of ChatGPT Based on a Set of Multiple-Choice Questions on Objective Tests of Hearing.
Article in Cureus, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 22 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
22 citing papers in PubMed.
- Generative AI for Personalized Learning Platforms in Otolaryngology-Head and Neck Surgery Education: Scoping Review.JMIR medical education · 2026Review
- Audiologist-Guided Multimodal AI for Pure-Tone Audiometry and Tympanometry Interpretation and Reporting.Journal of medical systems · 2026Article
- Performance of large language models on undergraduate endodontic multiple-choice questions.BMC oral health · 2026Article
- A validity-guided workflow for robust large language model research in psychology.Behavior research methods · 2026Article
- Evaluation of the performance of ChatGPT-4 and ChatGPT-4o as a learning tool in endodontics.International endodontic journal · 2026Article
- Is one run enough? Reproducibility of flagship large language models across temperature and reasoning settings in biomedical text processing.Journal of the American Medical Informatics Association : JAMIA · 2026Article
- Diagnostic interpretation of pure tone audiograms by multimodal LLMs: A comparative study of ChatGPT-5.0 and Gemini 2.5.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- Variation in Large Language Model Recommendations in Challenging Inpatient Management Scenarios.Journal of general internal medicine · 2026Article
- Evaluation of ChatGPT's Accuracy, Repeatability, and Reasoning Ability in Prosthodontics Education: A Cross-Sectional Comparative Study with Prosthodontists.Journal of clinical and experimental dentistry · 2026Review
- Reference Accuracy in Large Language Model Chatbots: A Metric for Inherent Misinformation?Medical science monitor : international medical journal of experimental and clinical research · 2026Article
- Performance of deepseek-R1 and ChatGPT-5.4 thinking in the medical laboratory professional title examination: accuracy, stability, and comparison with interns.Frontiers in digital health · 2026Article
- The application and challenges of ChatGPT in laboratory medicine.Advances in laboratory medicine · 2025Review
- Article
- Performance of the Large Language Models on the Chinese National Nurse Licensure Examination: Cross-Sectional Evaluation Study.JMIR medical informatics · 2025Article
- An Evaluation of Current Trends in AI-Generated Text in Otolaryngology Publications.The Laryngoscope · 2025Article
- Evaluation of deepseek, gemini, ChatGPT-4o, and perplexity in responding to salivary gland cancer.BMC oral health · 2025Article
- The Accuracy of ChatGPT-4o in Interpreting Chest and Abdominal X-Ray Images.Journal of personalized medicine · 2025Article
- ChatGPT Responses to Clinical Questions in the Japan Atherosclerosis Society Guidelines for Prevention of Atherosclerotic Cardiovascular Disease 2022.Journal of atherosclerosis and thrombosis · 2025Article
- Performance of ChatGPT in Pediatric Audiology as Rated by Students and Experts.Journal of clinical medicine · 2025Article
- A Performance Evaluation of Large Language Models in Keratoconus: A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity.Journal of clinical medicine · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
introductionChatGPT has been tested in many disciplines, but only a few have involved hearing diagnosis and none to physiology or audiology more generally. The consistency of the chatbot's responses to the same question posed multiple times has not been well investigated either. This study aimed to assess the accuracy and repeatability of ChatGPT 3.5 and 4 on test questions concerning objective measures of hearing. Of particular interest was the short-term repeatability of responses which was here tested on four separate days extended over one week.
methodsWe used 30 single-answer, multiple-choice exam questions from a one-year course on objective methods of testing hearing. The questions were posed five times to both ChatGPT 3.5 (the free version) and ChatGPT 4 (the paid version) on each of four days (two days one week and two days the following week). The accuracy of the responses was evaluated in terms of a response key. To evaluate the repeatability of the responses over time, percent agreement and Cohen's Kappa were calculated. Results: The overall accuracy of ChatGPT 3.5 was 48-49%, while that of ChatGPT 4 was 65-69%. ChatGPT 3.5 consistently failed to pass the threshold of 50% correct responses. Within a single day, the percent agreement was 76-79% for ChatGPT 3.5 and 87-88% for ChatGPT 4 (Cohen's Kappa 0.67-0.71 and 0.81-0.84 respectively). The percent agreement between responses from different days was 75-79% for ChatGPT 3.5 and 85-88% for ChatGPT 4 (Cohen's Kappa 0.65-0.69 and 0.80-0.85 respectively).
conclusionChatGPT 4 outperforms ChatGPT 3.5 both in accuracy and higher repeatability over time. However, the great variability of the responses casts doubt on possible professional applications of both versions.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.