A memory-driven reinforcement learning model of phenotypic adaptation for anticipating therapeutic resistance in prostate cancer.

While contemporary cancer treatment strategies have significantly prolonged the lives of patients, therapeutic resistance remains a predominant cause of disease progression and cancer-related deaths. Cancer therapy often induces gene regulatory responses that promote cell survival in the face of this therapy. Herein, we sought to develop a stochastic model of the response to repeat therapeutic challenge. This model integrates reinforcement learning to account for environmental history-dependent cellular transitions and growth dynamics. When applied to prostate cancer, this memory-driven adaptive model successfully captures the experimentally-observed dynamics of drug-sensitive and drug-resistant LNCaP cells under varying dosing schedules of androgen receptor blockade with enzalutamide (enza), significantly outperforming traditional transition models that lack history dependence. This performance is especially evident in the ability of our approach to robustly predict stochastic fluctuations in cancer cell population sizes across the entire disease trajectory, including subtle, later-emerging responses following initial therapy. The model was further evaluated by predicting the control of resistant cells in an enza environment by modeling inhibition of the p38/MAPK pro-survival stress axis, which was then validated experimentally. Lastly, we developed and applied a patient-calibrated model using prostate-specific antigen (PSA) data from clinical patient cohorts undergoing intermittent androgen deprivation therapy. Our model accurately predicts the PSA dynamics under repeated treatment cycles and effectively distinguishing between patients who respond and those who do not respond to treatment, thereby providing quantitative insight into prostate cancer progression. We anticipate that such adaptive modeling frameworks will be broadly useful for predicting cancer treatment outcomes and developing optimized adaptive therapeutic strategies tailored to patient-specific disease dynamics in additional cancer contexts.