Reinforcement learning for proposing smoking cessation activities that build competencies: Combining two worldviews in a virtual coach.

backgroundReaching personal goals typically requires building competencies (e.g., insights into personal strengths), but expert health professionals and non-expert clients often think differently about which competencies are needed. Just having a virtual coach advise activities for "expert-devised" competencies may not motivate clients to carry them out, while advising only "non-expert devised" activities may not result in all required competencies being built.

methodsWe integrated the client and health expert worldviews in our modeling method for informing the activity selection by a virtual coach: We created a pipeline to build a reinforcement learning model for proposing activities in the context of preparing for quitting smoking. This model considers smokers’ current and future levels for expert-devised competencies as well as their beliefs about the usefulness of different competencies when choosing activities. To train the model, we conducted a micro-randomized trial in which 542 smokers interacted with a virtual coach in five sessions spread over at least nine days and received a randomly chosen activity in each session. Using data from this study, we performed simulations to systematically assess the impact of the different model components on the competencies built by smokers. Moreover, we performed paired Bayesian t-tests to determine the effect of persuasive activities on smokers’ usefulness beliefs.

resultsOur simulations show that smokers’ current levels for the expert competencies and their usefulness beliefs are important to consider when building expert competencies. In fact, we saw improvements of up to 22% when considering current competencies, and an additional 13% when also accounting for usefulness beliefs. Furthermore, although we found credible evidence that persuasive activities changed smokers’ usefulness beliefs, the effects might be too small to contribute in an optimal strategy for building competencies.

conclusionThe worldviews of both health experts and smokers are important to consider when proposing activities for preparing for quitting smoking. We have presented a reinforcement learning model that combines these worldviews, and we hope that our work can be an example of incorporating different worldviews in a reinforcement learning model for building competencies. Our code and dataset are publicly available.