Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals' perspectives.

Explainable Artificial Intelligence (XAI) has the potential to enhance clinical decision support (CDS) systems however, it remains unclear how XAI systems are perceived by healthcare professionals in hospital settings, and if new challenges arise as a result of explanations. This scoping review aimed to understand healthcare professionals' perceptions of CDS systems with XAI in the hospital setting; specifically, the drivers of acceptance and use, explainability needs, and design preferences. Databases were searched. Studies were included if they reported qualitative findings on health professionals' perceptions of XAI-enabled CDS systems used in hospital settings. MEDLINE, Embase, and Web of Science were searched, and reference lists were screened for additional papers. Study characteristics and health professional perceptions were extracted and inductively coded. A quality assessment was performed using the CASP checklist. Sixteen studies were identified. Included studies primarily focused on ML-based CDS systems for predicting various clinical outcomes. Most studies used feature importance or model agnostic techniques like SHapley Additive exPlanations (SHAP). Overall, healthcare professionals perceived CDS systems with XAI as useful for supporting clinical tasks, decision-making, and teamwork. Acceptance was influenced by integration into workflows, performance, data quality, and alignment with clinical knowledge. Concerns were raised about overreliance and reduced professional autonomy. Health professionals predominantly used explanations to validate outputs, and desired actionable information from systems. SHAP plots and visualizations were difficult to interpret. Participants preferred explanation designs that included concise, high-level information, and simple plots for quick interpretation. Clear visual indicators such as colour coding, contextual patient data, and aggregating similar features also aided interpretation. Poorly designed XAI explanations can hinder understanding and increase cognitive burden in busy clinical settings. Future research should optimise the design and delivery of explanations, so clinicians can appropriately trust in XAI CDS systems and feel confident in their clinical decision making.