ArticleEuropean archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery2024
Validity and reliability of an instrument evaluating the performance of intelligent chatbot: the Artificial Intelligence Performance Instrument (AIPI).
Article in European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 37 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
37 citing papers in PubMed, 2 syntheses or guidelines pooled it, 77 citations in OpenAlex.
- Testing and evaluation of generative large language models in electronic health record applications: a systematic review.Journal of the American Medical Informatics Association : JAMIA · 2026Pooled it
- Clinical decision support using large language models in otolaryngology: a systematic review.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2025Pooled it
- ChatGPT Versus DeepSeek in Resectability Categorization of Chronic Rhinosinusitis: A Comparison Based on EPOS 2020.World journal of otorhinolaryngology - head and neck surgery · 2026Article
- "MELMA" in otolaryngology: Medical evaluation of large language model answers. Clinician-rated scoring (MELMA-Q) and web-based auditing (MELMA-W) novel tools for AI assessment.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- Optimizing a Multimodal Large Language Model for Ultrasound-Based Thyroid Nodule Malignancy Classification: A Comparative Study of Few-Shot Learning, Prompt Engineering, and Fine-Tuning.Diagnostics (Basel, Switzerland) · 2026Article
- Use of Commercially Available Large Language Models to Generate Information Leaflets on Post-Intensive Care Syndrome: Clinical Utility Assessment.JMIR formative research · 2026Article
- Comparative evaluation of four large language models (ChatGPT-5, Gemini 2.5 Pro, Claude Sonnet-4, and DeepSeek-R1) for diagnostic and therapeutic decision-making in real-world otolaryngology scenarios.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- Assessing ChatGPT Responses to Commonly Asked Questions About Hypoglossal Nerve Stimulation.World journal of otorhinolaryngology - head and neck surgery · 2026Article
- Evaluation of ChatGPT-4o as a text-based clinical decision support tool for junior otolaryngology residents: A randomized controlled trial.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- Evaluation of ChatGPT-4o as a text-based clinical decision support tool for junior otolaryngology residents: A randomized controlled trial.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- Safety-First Framework for AI-Enabled Anamnesis in Head and Neck Surgery: Evidence Synthesis from a Narrative Review.Journal of clinical medicine · 2026Review
- Generative Artificial Intelligence Methodology Reporting in Otolaryngology: A Scoping Review.The Laryngoscope · 2026Article
- Artificial Intelligence in Rhinoplasty Recovery: Linguistic Intelligence and Machine Learning-Driven Insights.Journal of clinical medicine · 2026Article
- AI at the Sella Turcica: Multi-Model Large Language Model Evaluation in Pituitary Adenomas.Brain & spine · 2026Article
- Applications of Natural Language Processing in Otolaryngology: A Scoping Review.The Laryngoscope · 2025Article
- A radiopathomics model for predicting large-number cervical lymph node metastasis in clinical N0 papillary thyroid carcinoma.European radiology · 2025Article
- Diagnostic Performance of ChatGPT-4o in Analyzing Oral Mucosal Lesions: A Comparative Study with Experts.Medicina (Kaunas, Lithuania) · 2025Article
- Assessing the value of artificial intelligence-based image analysis for pre-operative surgical planning of neck dissections and iENE detection in head and neck cancer patients.Discover oncology · 2025Article
- Advancing Clinical Chatbot Validation Using AI-Powered Evaluation With a New 3-Bot Evaluation System: Instrument Validation Study.JMIR nursing · 2025Article
- Artificial intelligence for image recognition in diagnosing oral and oropharyngeal cancer and leukoplakia.Scientific reports · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors at 5 institutions in 6 countries.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectivesTo evaluate the reliability and validity of the Artificial Intelligence Performance Instrument (AIPI).
methodsMedical records of patients consulting in otolaryngology were evaluated by physicians and ChatGPT for differential diagnosis, management, and treatment. The ChatGPT performance was rated twice using AIPI within a 7-day period to assess test-retest reliability. Internal consistency was evaluated using Cronbach's α. Internal validity was evaluated by comparing the AIPI scores of the clinical cases rated by ChatGPT and 2 blinded practitioners. Convergent validity was measured by comparing the AIPI score with a modified version of the Ottawa Clinical Assessment Tool (OCAT). Interrater reliability was assessed using Kendall's tau.
resultsForty-five patients completed the evaluations (28 females). The AIPI Cronbach's alpha analysis suggested an adequate internal consistency (α = 0.754). The test-retest reliability was moderate-to-strong for items and the total score of AIPI (r
conclusionsAIPI is a valid and reliable instrument in assessing the performance of ChatGPT in ear, nose and throat conditions. Future studies are needed to investigate the usefulness of AIPI in medicine and surgery, and to evaluate the psychometric properties in these fields.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.