ArticleJournal of pathology informatics2026
How does AI perform compared to human expert panels in medical Delphi studies? A pilot study through the lens of pathology.
Article in Journal of pathology informatics, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. An erratum has been issued. Not yet cited in PubMed.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
0 citing papers in PubMed.
No citing paper in PubMed yet.
Corrections and comments
- Erratum issued
Authors and funding
31 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Background: Since their inception, Delphi studies have been a key part of medical literature. They consist of an expert panel tasked with coming to consensus on answers to various questions where obtaining objective results is difficult or impossible, with ranked responses based on a Likert scale. The ability of artificial intelligence (AI), particularly large language models (LLMs), to perform this role traditionally assigned to a panel of experts has been scarcely explored in medicine. This study accordingly aimed to explore the feasibility of an "AI-run" Delphi study applied to the practice of pathology. Methods: A prior human-based Delphi study (PMID: 36603288) employed to forecast the future role of AI in pathology was repeated, but this time with LLMs (Llama 3, ChatGPT-4, and ChatGPT-3.5 based on availability at the time of the study). This was done at various temperature settings (0, 0.7, and 1.0), a measurement of how much an LLM prioritizes determinism versus creativity. Low temperature caused the models to be more deterministic and focused, whereas high temperature increased creativity. "Delphi-GPT" was created to automate prompts that entailed 5 trials for 180 questions, leading to data that were compared to the original human expert panel. Findings: All LLM and temperature combinations were able to reach consensus for a greater percentage of the 180 questions posed than human experts. Newer ChatGPT-4 and Llama 3 models performed better than ChatGPT-3.5. Whereas AI models and human experts did not always agree, the amount of agreement increased when the temperature setting was increased across all LLMs. Interpretation: LLMs are shown here to successfully be able to simulate a Delphi study in medicine. The data show that generative AI models were consistently able to reach greater degrees of consensus than human experts in their responses to 180 prompts related to the future practice of pathology. This serves as a proof-of-concept that one day, pending further robust methodological validation, AI could even serve as a surrogate for de novo Delphi studies that ordinarily would have relied on feedback from a panel of experts. The reliability of consensus/concordance achieved will depend upon the combination of LLM and temperature setting selected.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.