Introduction to Online Control

This tutorial guide introduces online nonstochastic control, an emerging paradigm in control of dynamical systems and differentiable reinforcement learning that applies techniques from online convex optimization and convex relaxations to obtain new methods with provable guarantees for classical settings in optimal and robust control. In optimal control, robust control, and other control methodologies that assume stochastic noise, the goal is to perform comparably to an offline optimal strategy. In online control, both cost functions and perturbations from the assumed dynamical model are chosen by an adversary. Thus, the optimal policy is not defined a priori and the goal is to attain low regret against the best policy in hindsight from a benchmark class of policies. The resulting methods are based on iterative mathematical optimization algorithms and are accompanied by finite-time regret and computational complexity guarantees. This book is ideal for graduate students and researchers interested in bridging classical control theory and modern machine learning.

Table of Contents:
Symbols; Part I. Background in Control and RL: 1. Introduction; 2. Dynamical systems; 3. Markov decision processes; 4. Linear dynamical systems; 5. Optimal control of linear dynamical systems; Part II. Basics of Online Control: 6. Regret in control; 7. Online nonstochastic control; 8. Online nonstochastic system identification; Part III. Learning and Filtering: 9. Learning in unknown linear dynamical systems; 10. Kalman filtering; 11. Spectral filtering; Part IV. Online Control with Partial Observation: 12. Policy classes for partially observed systems; 13. Online nonstochastic control with partial observation; References; Index.

Review :
'We are in a golden age for control and decision making. A proliferation of new applications including self-driving vehicles, humanoid robots, and artificially intelligent drones opens a new set of challenges for control theory to address. Hazan and Singh have written the definitive book on the New Control Theory - non-stochastic control. The phrase 'a paradigm shift' has become cliche from overuse, but here it is truly well deserved; the authors have revisited the foundations by focusing on building controllers that perform nearly as well as if they knew future disturbances in advance, rather than relying on probabilistic or worst-case models. The non-stochastic control approach has extended one of the most profound ideas in mathematics of the 20th century, online (no-regret) learning, to master sequential decision making with continuous actions. This leads to high performance in benign environments and resilience in adversarial ones. The book, authored by pioneers in the field, presents both foundational concepts and the latest research, making it an invaluable resource.' Drew Bagnell, Carnegie Mellon University and Aurora 'As someone who has worked extensively on learning theory and online learning, and later applied these ideas in domains such as autonomous driving and humanoid robotics, I find this book both timely and inspiring. It introduces a regret-minimization framework for control that draws on the elegance and power of online learning. Traditional control theory often models noise either as stochastic-sometimes unrealistically optimistic-or adversarial-often overly conservative. This book charts a new path by asking a deeper question: while we cannot predict noise, can we perform nearly as well as if we could? The answer, developed here, is a novel and exciting paradigm that bridges learning theory and control, and I believe it will have a lasting impact on both research and practice.' Shai Shalev-Shwartz, Hebrew University of Jerusalem