Evaluation of the explainability and reliability of two surrogate models for predicting outflow of combined sewer systems.

Urban drainage networks (UDNs) are designed to reliably discharge wastewater and stormwater, with machine learning (ML) increasingly leveraged for a wide range of applications. A legal assessment of the European 'Artificial Intelligence Act' (AI Act) indicates that, while UDNs are considered part of critical infrastructure, ML applications in this field are generally not classified as high-risk AI systems, being subject to specific implementation requirements. Nevertheless, aspects such as model explainability are important to align with best practices in ML model development. In this context, two ML models (i.e., purely data-driven and physics-leveraged) used herein as surrogates for hydrodynamic modelling to predict combined sewer flow, were adapted from the authors' previous work and enhanced with various explainable AI techniques such as SHapley Additive exPlanations and robustness scores for a comprehensive understanding of the models' decision-making processes. The analysis revealed that the engineered classification into dry or wet flow is the most important feature for both ML models, followed by historical rainfall features for the purely data-driven ML model and hydrodynamic model outputs for the physics-leveraged ML model. Additionally, the achievable accuracy of both utilised ML models was highly sensitive to deviations in rainfall data, further limiting their reliability and trustworthiness.