Restless Multi-Armed Bandits

Scalable resource allocation algorithms for health, conservation, and beyond.

Restless Multi-Armed Bandits (RMABs) model the problem of allocating scarce resources (interventions, attention, budget) across many agents whose states evolve over time. We develop scalable algorithms and deploy them in real-world systems.

Key directions:

  • Whittle index policies and online learning for unknown dynamics
  • Decision-focused learning of RMAB transition parameters
  • Neural index policies for multi-action, heterogeneous-budget settings
  • Non-stationary and networked RMAB extensions

Deployments:

  • Maternal and child health outreach scheduling (ARMMAN, India)
  • Pediatric early warning systems (Children’s Healthcare of Atlanta)
  • Green security patrol scheduling (wildlife conservation)

Selected publications:

(Wang et al., 2023; Wang et al., 2023; Verma et al., 2023; Xu et al., 2021; Pandey et al., 2025)

References

2025

  1. Neural Index Policies for Restless Multi-Action Bandits with Heterogeneous Budgets
    Himadri S Pandey, Kai Wang, and Gian-Gabriel P. Garcia
    2025
    In submission
    arXiv

2023

  1. Optimistic Whittle Index Policy: Online Learning for Restless Bandits
    Kai Wang, Lily Xu, Aparna Taneja, and Milind Tambe
    In AAAI Conference on Artificial Intelligence (AAAI), 2023
    arXiv
  2. Scalable Decision-Focused Learning in Restless Multi-Armed Bandits with Application to Maternal and Child Health
    Kai Wang, Shresth Verma, Aditya Mate, Sanket Shah, Aparna Taneja, and 3 more authors
    In AAAI Conference on Artificial Intelligence (AAAI), 2023
    arXiv
  3. Restless Multi-Armed Bandits for Maternal and Child Health: Results from Decision-Focused Learning
    Shresth Verma, Aditya Mate, Kai Wang, Neha Madhiwalla, Aparna Hegde, and 2 more authors
    In International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), 2023
    HTML

2021

  1. Dual-Mandate Patrols: Multi-Armed Bandits for Green Security
    Lily Xu, Elizabeth Bondi, Fei Fang, Andrew Perrault, Kai Wang, and 1 more author
    In AAAI Conference on Artificial Intelligence (AAAI), 2021
    Best Paper Runner-Up
    arXiv