Leveraging transformer models to predict cognitive impairment: accuracy, efficiency, and interpretability.

objectiveThis study aims to develop an enhanced Transformer model for predicting mild cognitive impairment (MCI) using data from the China Health and Retirement Longitudinal Study (CHARLS), focusing on handling mixed data types and improving predictive accuracy.

methodsThe Transformer integrates categorical (integer-encoded) and continuous (floating-point) data, using multi-head attention with four heads to capture complex relationships. Preprocessing involved separate embedding layers for categorical data and feed-forward networks for continuous data. The model was compared with SVM and XGBoost, trained for 150 epochs with RMSProp and a cosine annealing scheduler. Key metrics included accuracy, Mean Absolute Error (MAE) tolerance, and training loss. An attention heatmap was generated to visualize feature importance.

resultsThe Transformer outperformed SVM and XGBoost, achieving over 90% accuracy at an MAE tolerance of 3.5. The model showed rapid convergence, with loss stabilizing within 20 epochs. The attention heatmap highlighted key features, confirming the effectiveness of the multi-head attention mechanism in identifying relevant variables.

conclusionThe enhanced Transformer model offers superior accuracy and efficiency in predicting cognitive decline compared to traditional models. Its capacity to process both continuous and categorical data and its interpretability through attention mechanisms make it a promising tool for early detection of neurodegenerative diseases, potentially improving clinical decision-making and interventions.