520.674 Information Theoretic Methods in Statistics

Notes from lectures will provided through this web page for the private use of the students enrolled in class.

Portions of these notes may have restricted access. If you encounter a page which requires a password, contact the instructor in class or via electronic mail. Please do not distribute notes without permission from the instructor as some of them contain copyright-protected material.

Notes from Past Offerings of the Course

Course Description with References [pdf, ps]

Information Geometry

• I-Divergence: Definition and Properties [pdf, ps]

• I-Divergence Geometry and Minimization Problems [pdf, ps]

• I-Projection on Linear Families of Distibutions [pdf, ps]

• The Method of Iterative Proportional Fitting [pdf, ps]

• Generalized Iterative Scaling [pdf, ps]

• Image Reconstruction by I-Divergence Minimization [pdf, ps]

• Maximum Likelihood from Incomplete Data (EM Algorithms) [pdf, ps]

Large Deviations

• The Markov, Chebychev and Chernoff Inequalities [pdf, ps]

• Large Deviations of the Empirical Mean [pdf, ps]

• Large Deviations of the Empirical Measure [pdf, ps]

• Sanov's, Stein's and Chernoff's Theorems [pdf, ps]

Universal Data Compression

• Redundancy in Data Compression [pdf, ps]

• Universal Compression by the Method of Mixtures [pdf, ps]

• Rissanen's Theorem and MDL [pdf, ps]

Sample Examinations

Spring 2002 Take-home Exam [pdf, ps]

Spring 1999 Take-home Exam [pdf, ps]

Spring 1997 Take-home Exam [pdf, ps]