Sequential Decision Making

STAT 234 Subject & Catalog Number

Description

This graduate course will focus on reinforcement learning algorithms and sequential decision making methods with special attention to how these methods can be used in mobile health.  Reinforcement learning is the area of machine learning which is concerned with sequential decision making.  We will focus on the areas of sequential decision making that concern both how to select optimal actions as well as how to evaluate the impact of these actions.  The choice of action is operationalized via a policy.  A policy is a (stochastic) deterministic mapping from the available data at each time t into (a probability space over) the set of actions. We will consider both off-line and on-line methods for learning good policies. 

 Mobile health is an area that lies within multiple scientific disciplines including: statistical science, computer science, behavioral science and cognitive neuroscience. This makes for very exciting interdisciplinary science! Smartphones and wearable devices have remarkable sensing capabilities allowing us to understand the context in which a person is at a given moment. These devices also have the ability to deliver treatment actions tailored to the specific needs of users in a given location at a given time. Figuring out when and in which context, which treatment actions to deliver can assist people in achieving their longer term health goals.  In multiple classes we will brainstorm about how the methods we discussed during that class might be useful in mobile health.

 This course will cover the following topics:  Markov Decision Processes, on-policy and off-policy RL, least squares methods in RL and Bayesian RL, namely posterior sampling.  Most of the course will focus on Bayesian RL via posterior sampling.  This is particularly useful in mobile health as posterior sampling facilitates off-policy and continual learning.    Also the Bayesian paradigm facilitates use of prior data in initializing an RL algorithm.    Other topics from statistics, machine learning and RL that I think are interesting (you could consider in your class project) include: 1) transfer learning (using data on other similar users to enable faster learning); 2) RL for steering LLM behavior; 3) interpretability of policies (enabling communication with behavioral scientists by making connections to behavioral theories); 4) methods for warm-starting RL algorithms,  5) hierarchical RL, 6) experience replay and 7) multi-task learning.

Recommended Prep

Recommended prerequisites are the equivalent of Stat 210, Stat 211, and Compsci 181