Current Project:
Rethinking Optimization for Reinforcement Learning (Luca Viano)
While promising automated solutions beyond human performance to many real-world tasks, including continuous control, robotics, and autonomous driving, reinforcement learning (RL) and its variations, such as inverse RL, imitation learning, and behavioral cloning, appear to be extremely fragile to mismatches in practice. Existing optimization approaches are motivated from the naive tabular settings and cannot extend to the contemporary neural network representations as underlying formulations are non-convex and non-concave minimax optimization problems blocked by hardness results. To this end, we propose a paradigm shift in how we handle RL and its variants via how we set up problems with neural networks for continuous state and action spaces, how to exploit new key structures in minimax problems, such as weak Minty inequalities, with new algorithms such as adaptive double time-scale extragradient methods to overcome hardness results, and how to exploit new universal second order methods in order to close the gap between convergence rate of algorithms (i.e., numerical efficiency) vs. their sample efficiency, which is often more critical for RL. We contend that optimization algorithms cannot be developed in isolation from the context in which RL formulations are proposed. By taking a joint perspective between RL and optimization, we expect our work to make RL more robust, more scalable, and more sample efficient, which we will illustrate with real-life applications.
Past Project:
Robust deep learning and generative models (Fabian Latorre)