Slides 2022 Outline The 2022 course consists of the following topics Lecture 01 Introduction. The role of models and data Maximum-likelihood formulation Sample complexity bound for estimation and prediction Lecture 02 Generalized linear model Logistic regression Lecture 03 Linear algebra reminder Gradients Reading convergence plots Lecture 04 Optimization algorithms Optimality measures. Structures in optimization Gradient descent. Gradient descent for smooth functions Lecture 05 Optimality of convergence rates Lower bounds Accelerated gradient descent Concept of total complexity Stochastic gradient descent Lecture 06 Concise signal models Compressive sensing Sample complexity bounds for estimation and prediction Challenges to optimization algorithms for non-smooth optimization Subgradient method Lecture 07 Introduction to proximal-operators Proximal gradient methods Linear minimization oracles Conditional gradient method for constrained optimization Lecture 08 Time-data trade-offs Variance reduction for improving trade-offs Introduction to deep learning Lecture 09 Generalization through uniform convergence bounds Double descent curves and overparameterization Implicit regularization Lecture 10 Optimization in deep learning Escaping saddle points Adaptive gradient methods Lecture 11 Adversarial machine learning and generative adversarial networks (GANs) Wasserstein GAN Sharpness Aware minimization. Lecture 12 Robustness: interplay between width, depth and initialization Difficulty of minimax optimization Diffusion models Lecture 13 Primal-dual optimization-I: Fundamentals of minimax problems Fenchel conjugates Duality Lecture 14 Primal-dual optimization-II: Extra gradient method Chambolle-Pock algorithm Stochastic primal-dual methods Lecture 15 Primal-dual III: Lagrangian gradient methods
