Slides 2021 Outline The 2021 course consists of the following topics: Lecture 1 Overview of Mathematics of Data Empirical Risk Minimization Statistical Learning with Maximum Likehood Estimators Decomposition of error Lecture 2 Principles of iterative descent methods Gradient descent for smooth convex problems Gradient descent for smooth non-convex problems Lecture 3 Accelerated gradient descent Adaptive gradient methods Stochastic gradient descent for convex problems Stochastic gradient descent for non-convex problems Lecture 4 Deficiency of smooth models Sparsity and compressive sensing Atomic norms Non-smooth minimization via Subgradient descent Lecture 5 Composite minimization Proximal gradient methods Introduction to Frank-Wolfe method Lecture 6 Time-data trade-offs Rate iteration-cost trade-offs Variance reduction Lecture 7 Introduction to Deep Learning The Deep Learning Paradigm Challenges in Deep Learning Theory and Applications Introduction to Generalization error bounds Uniform Convergence and Rademacher Complexity Lecture 8 The classical trade-off between model complexity and risk The generalization mystery in Deep Learning Implicit regularization of optimization algorithms Double Descent curves Generalization bounds based on Algorithmic Stability Lecture 9 Scalable non-convex optimization with emphasis on deep learning Lecture 10 Adversarial Machine Learning (minmax) Adversarial training Generative adversarial networks Difficulty of minmax Lecture 11 Min-max optimization (continued). Primal-dual optimization (Part 1) Lecture 12 Primal-dual optimization (Part 2) Algorithms for solving min-max optimization Extra-gradient algorithms Lecture 13 Primal-dual optimization (Part 3) Lagrangian methods for constrained minimization
