An explainable GeoAI approach for the multimodal analysis of urban human dynamics: a case study for the COVID-19 pandemic in Rio de Janeiro.

The recent COVID-19 pandemic has underscored the need for effective public health interventions during infectious disease outbreaks. Understanding the spatiotemporal dynamics of urban human behaviour is essential for such responses. Crowd-sourced geo-data can be a valuable data source for this understanding. However, previous research often struggles with the complexity and heterogeneity of such data, facing challenges in the utilisation of multiple modalities and explainability. To address these challenges, we present a novel approach to identify and rank multimodal time series features derived from mobile phone and geo-social media data based on their association with COVID-19 infection rates in the municipality of Rio de Janeiro. Our analysis spans from April 6, 2020, to August 31, 2021, and integrates 59 time series features. We introduce a feature selection algorithm based on Chatterjee's Xi measure of dependence to identify relevant features on an Área Programática da Saúde (health area) and city-wide level. We then compare the predictive power of the selected features against those identified by traditional feature selection methods. Additionally, we contextualise this information by correlating dependence scores and model error with 15 socio-demographic variables such as ethnic distribution and social development. Our results show that social media activity related to COVID-19, tourism and leisure activities was associated most strongly with infection rates, indicated by high dependence scores up to 0.88. Mobility data consistently yielded low to intermediate dependence scores, with the maximum being 0.47. Our feature selection approach resulted in better or equivalent model performance when compared to traditional feature selection methods. At the health-area level, local feature selection generally yielded better model performance compared to city-wide feature selection. Finally, we observed that socio-demographic factors such as the proportion of the Indigenous population or social development correlated with the dependence scores of both mobility data and health- or leisure-related semantic topics on social media. Our findings demonstrate the value of integrating localised multimodal features in city-level epidemiological analysis and offer a method for effectively identifying them. In the broader context of GeoAI, our approach provides a framework for identifying and ranking relevant spatiotemporal features, allowing for concrete insights prior to model building, and enabling more transparency when making predictions.