Instructors
Prof. Volkan Cevher
Prof. Matthias Seeger
Description
The course focuses on providing diverse mathematical tools for graduate students from statistical inference and learning; graph theory, signal processing and systems; coding theory and communications, and information theory. We will discuss exact and approximate statistical inference over large number of interacting variables, and develop probabilistic and optimization-based computational methods. We will cover hidden Markov models, belief propagation, and variational Bayesian methods (e.g., expectation maximization algorithm). We will read research papers and book chapters to understand the benefits and limitations of such algorithms.
Recommended Reading Material
Textbooks:
- Christopher M. Bishop, Pattern Recognition and Machine Learning.
- M.J. Wainwright and M.I. Jordan, Graphical Models, exponential families, and variational inference [advanced].
- S.L. Lauritzen, Graphical Models [advanced].
- M.I. Jordan (ed.), Learning in Graphical Models.
- Daphne Koller and Nir Friedman, Probabilistic Graphical Models [advanced].
- J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference.
Tutorial and Research Papers:
- F.R. Kschischang, B.J. Frey, and H.-A. Loeliger, Factor Graphs and the Sum-Product Algorithm, IEEE Transactions on Information Theory, Vol. 47, No. 2, February 2001.
- W. Xu, Q. Zhu and M. I. Jordan, The Junction Tree Algorithm, Class notes, UC Berkeley, CS281A/Stat241A, Fall 2004.
- R. Cowell, Introduction to Inference for Bayesian Networks.
- M. I. Jordan, Z. Ghahramani, T. S. Jaakkola and L. K. Saul, An Introduction to Variational Methods for Graphical Models, Machine Learning vol. 37, 1999.
- T. Minka, Divergence Measures and Message Passing, Microsoft Research Ltd. Tech. Report MSR-TR-2005-173, December 2005.
- V. Cevher, M. Duarte, C. Hegde, and R. Baraniuk, Sparse Signal Recovery Using Markov Random Fields, 2008.
- R.G. Baraniuk, V. Cevher, M.F. Duarte and C. Hegde, Model-Based Compressive Sensing, 2008.
- V. Cevher, M.F. Duarte, C. Hegde and R.G. Baraniuk, Sparse Signal Recovery Using Markov Random Fields, 2008.
- Seeger, M. and Wipf, D, Variational Bayesian Inference Techniques, IEEE SPM 2010.
- Seeger, M., Tutorial on Sparse Linear Models: Reconstruction and Approximate Inference (http://ipg.epfl.ch/~seeger/lapmalmainweb/teaching/dagm10/index.html).
Available Codes
- Infer.net — Microsoft Research UK’s graphical model library http://research.microsoft.com/en-us/um/cambridge/projects/infernet/
- PNL — Intel’s Probabilistic Network Library http://sourceforge.net/projects/openpnl/
- OpenCV (open source library for computer vision) http://opencv.org/
- Image classification using bag-of-words http://vision.ucla.edu/~vedaldi/code/bag/bag.html
- Kevin Murphy’s Bayes net toolbox in Matlab https://code.google.com/p/bnt/
- Mark Steyvers and Tom Griffiths Matlab Topic Modelling Toolbox http://psiexp.ss.uci.edu/research/programs_data/toolbox.htm
- David Blei: Latent Dirichlet allocation (LDA) for topic modeling http://www.cs.princeton.edu/~blei/lda-c/index.html
- Y.W. Teh. Nonparametric Bayesian Mixture Models – release 2.1.
- Hal Daume III . Fast search for Dirichlet process mixture models
- Kenichi Kurihara. Variational Dirichlet Process Gaussian Mixture Model
- Adnan Darwiche’s software for compiling Bayes nets into algebraic circuits http://reasoning.cs.ucla.edu
- JavaBayes (contains example Bayes nets) http://www.cs.cmu.edu/~javabayes/Home/
- WinBugs (Bayesian inference using Gibbs sampling) http://www.mrc-bsu.cam.ac.uk/software/bugs/
- Gaussian Process Code repository http://www.gaussianprocess.org/#code
- Inference with Gaussian processes http://www.gaussianprocess.org/gpml/code/matlab/doc/