Complementary Error Profiles of Human Readers and Large Language Models in Detecting AI-Generated Chest Radiographs: A Comparative Reader Study.

Frontier large language models (LLMs) and diffusion models can generate realistic chest radiographs, raising concerns for dataset integrity when provenance metadata are absent. We compared authenticity judgments by two radiologists, two nonradiologist physicians, and three frontier LLMs (GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8) on 160 posteroanterior radiographs: 40 real NIH ChestX-ray14 images and 120 synthetic images generated from text by three pipelines. Each evaluator judged every image once; the LLM evaluation was repeated twice 3 months later. Balanced accuracy was the primary metric, given 75% synthetic prevalence. The two best evaluators reached nearly identical balanced accuracy from opposite operating points: Gemini 3.5 Flash had the highest point estimate (77.9%; specificity 100%, sensitivity 55.8%) and RAD1 was a close second (75.8%; specificity 55.0%, sensitivity 96.7%), a paired difference of - 2.1 percentage points (95% CI, - 5.8 to 1.2). On repetition, LLM judgments were not fully reproducible and this ordering did not hold. Nonradiologists performed near chance. All four human readers detected diffusion-generated images at 82-100%, whereas the radiologists misclassified 38-45% of real images and the conservative LLMs reached perfect specificity yet detected only 32-72% of LLM-generated images. With laterality markers and borders masked, specificity was 62.5% (Gemini) and 85.0% (GPT-5.5) against 90-100% in same-day unmasked runs. Interevaluator agreement was low throughout (Fleiss' kappa among LLMs 0.213). A frequency-domain baseline reached 98.3% balanced accuracy within this dataset but 12% sensitivity on a held-out generator. Human readers and LLMs failed on different images; visual authenticity assessment alone may be insufficient for provenance verification.