Pediatric Autism Diagnosis Accuracy and Confidence: A Comparison of Experienced and Inexperienced Clinicians Making Decisions with and without AI Decision Support.

Background: Autism Spectrum Disorders (ASD) is a neurodevelopmental condition where early diagnosis is extremely important for optimal treatment effects. Unfortunately, the current age of diagnosis is made late due to a variety of factors, including a lack of clinicians with ASD expertise. We developed Autism Diagnostic Identification Software (ADIS), an AI-based clinical decision support tool that identifies autism-relevant behaviors in narrative clinical text and labels them with DSM5 diagnostic criteria. Using current DSM5 rules, an autism diagnosis is then suggested. Our aim is to provide pre-decision support, which differs from explainable AI approaches, in which AI decisions are clarified post hoc. Methods: To evaluate the impact of AI suggestions, we conducted a vignette-based user study with 21 clinicians (48% with completed medical training), each of whom reviewed four real pediatric cases (two with ADIS support and two without), resulting in 84 diagnostic decisions. The cases were chosen so that ADIS also suggested correct and incorrect decisions to study participants. ResuIts: Overall diagnostic accuracy was 57.14% when ADIS was active versus 66.67% without ADIS, a nonsignificant difference. Decision confidence was higher with ADIS (F(1,80) = 3.71, p = .058), and significantly higher among clinicians who had completed their medical training (F(1,80) = 18.45, p < .001). There was also significant interaction between ADIS correctness and training completion (F(1,38) = 5.05, p = .031): trainees showed 0% accuracy when ADIS was wrong versus 68.75% when it was correct, whereas trained clinicians achieved 66.67% and 56.25% accuracy in those conditions, respectively. Despite these differences in accuracy, confidence was high regardless of ADIS correctness. Most participants (90%) reported that ADIS use was learned quickly and rated its features as well integrated. Conclusions: These findings suggest that AI support can help inexperienced clinicians, but also highlight the importance of AI accuracy, given the observed overreliance on AI. ADIS is usable and perceived as helpful, but its deployment will be most effective when paired with AI literacy training that mitigates automation bias.