The explainable AI dilemma under knowledge imbalance in specialist AI for glaucoma referrals in primary care.

Primary eye care providers refer glaucoma patients using their clinical experience and context. Specialized Artificial Intelligence (AI) excels in referrals trained on clinical data but relies on assumptions that may not hold in practice. To address this knowledge imbalance, we proposed using AI explanations to help providers assess AI predictions against their experience and improve referrals. We developed AI models to identify patients needing urgent referrals from routine eye care data and created intrinsic and post-hoc explanations. In a user study with 87 optometrists, human-AI teams achieved higher accuracy (60%) than humans alone (51%), but explanations did not enhance performance. Instead, they introduced uncertainty about when to trust AI. Post-hoc explanations increased over-reliance on incorrect AI recommendations, and both explanation types contributed to anchoring bias, making participants align more closely with AI referrals than without explanations. Overall, human-AI teams still underperformed the AI alone (80% accuracy). Challenges persist in designing effective support mechanisms to surpass AI performance while preserving human agency.