System identifiability in a time-evolving agent-based model.

Mathematical models are a valuable tool for studying and predicting the spread of infectious agents. The accuracy of model simulations and predictions invariably depends on the specification of model parameters. Estimation of these parameters is therefore extremely important; however, while some parameters can be derived from observational studies, the values of others are difficult to measure. Instead, models can be coupled with inference algorithms (i.e., data assimilation methods, or statistical filters), which fit model simulations to existing observations and estimate unobserved model state variables and parameters. Ideally, these inference algorithms should find the best fitting solution for a given model and set of observations; however, as those estimated quantities are unobserved, it is typically uncertain whether the correct parameters have been identified. Further, it is unclear what 'correct' really means for abstract parameters defined based on specific model forms. In this work, we explored the problem of non-identifiability in a stochastic system which, when overlooked, can significantly impede model prediction. We used a network, agent-based model to simulate the transmission of Methicillin-resistant staphylococcus aureus (MRSA) within hospital settings and attempted to infer key model parameters using the Ensemble Adjustment Kalman Filter, an efficient Bayesian inference algorithm. We show that even though the inference method converged and that simulations using the estimated parameters produced an agreement with observations, the true parameters are not fully identifiable. While the model-inference system can exclude a substantial area of parameter space that is unlikely to contain the true parameters, the estimated parameter range still included multiple parameter combinations that can fit observations equally well. We show that analyzing synthetic trajectories can support or contradict claims of identifiability. While we perform this on a specific model system, this approach can be generalized for a variety of stochastic representations of partially observable systems. We also suggest data manipulations intended to improve identifiability that might be applicable in many systems of interest.