publications

2025

1. Leveraging Offline Data in Linear Latent Contextual Bandits

   Chinmaya Kausik, Kevin Tan, and Ambuj Tewari

   In Forty-second International Conference on Machine Learning, 2025
2. Actor-Critics Can Achieve Optimal Sample Efficiency

   Kevin Tan, Wei Fan, and Yuting Wei

   In Forty-second International Conference on Machine Learning, 2025

2024

1. Informing policy via dynamic models: Cholera in Haiti

   Jesse Wheeler, AnnaElaine Rosengart, Zhuoxun Jiang, and 3 more authors

   PLOS Computational Biology, Apr 2024

   DOI
2. Offline Policy Evaluation and Optimization Under Confounding

   Chinmaya Kausik, Yangyi Lu, Kevin Tan, and 3 more authors

   In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, 02–04 may 2024

   Abs PDF

   Evaluating and optimizing policies in the presence of unobserved confounders is a problem of growing interest in offline reinforcement learning. Using conventional methods for offline RL in the presence of confounding can not only lead to poor decisions and poor policies, but also have disastrous effects in critical applications such as healthcare and education. We map out the landscape of offline policy evaluation for confounded MDPs, distinguishing assumptions on confounding based on whether they are memoryless and on their effect on the data-collection policies. We characterize settings where consistent value estimates are provably not achievable, and provide algorithms with guarantees to instead estimate lower bounds on the value. When consistent estimates are achievable, we provide algorithms for value estimation with sample complexity guarantees. We also present new algorithms for offline policy improvement and prove local convergence guarantees. Finally, we experimentally evaluate our algorithms on both a gridworld environment and a simulated healthcare setting of managing sepsis patients. In gridworld, our model-based method provides tighter lower bounds than existing methods, while in the sepsis simulator, we demonstrate the effectiveness of our method and investigate the importance of a clustering sub-routine.

3. Hybrid Reinforcement Learning Breaks Sample Size Barriers In Linear MDPs

   Kevin Tan, Wei Fan, and Yuting Wei

   In Advances in Neural Information Processing Systems, 02–04 may 2024
4. Accelerated Inference for Partially Observed Markov Processes using Automatic Differentiation

   Kevin Tan, Giles Hooker, and Edward L. Ionides

   02–04 may 2024
5. Distributed Least Squares in Small Space via Sketching and Bias Reduction

   Sachin Garg, Kevin Tan, and Michał Dereziński

   In Advances in Neural Information Processing Systems, 02–04 may 2024
6. A Natural Extension To Online Algorithms For Hybrid RL With Limited Coverage

   Kevin Tan and Ziping Xu

   Reinforcement Learning Journal, 02–04 may 2024

2023

1. Learning Mixtures of Markov Chains and MDPs

   Chinmaya Kausik, Kevin Tan, and Ambuj Tewari

   In Proceedings of the 40th International Conference on Machine Learning, 23–29 jul 2023

   Abs PDF

   We present an algorithm for learning mixtures of Markov chains and Markov decision processes (MDPs) from short unlabeled trajectories. Specifically, our method handles mixtures of Markov chains with optional control input by going through a multi-step process, involving (1) a subspace estimation step, (2) spectral clustering of trajectories using "pairwise distance estimators," along with refinement using the EM algorithm, (3) a model estimation step, and (4) a classification step for predicting labels of new trajectories. We provide end-to-end performance guarantees, where we only explicitly require the length of trajectories to be linear in the number of states and the number of trajectories to be linear in a mixing time parameter. Experimental results support these guarantees, where we attain 96.6% average accuracy on a mixture of two MDPs in gridworld, outperforming the EM algorithm with random initialization (73.2% average accuracy). We also significantly outperform the EM algorithm on real data from the LastFM song dataset.