ECE 520.447 Introduction to Information Theory and Coding
Spring 2014
Mon Wed Fri 1:302:20 PM in Hackerman 320 (Occasionally 1:302:45)
Information Theory addresses some fundamental questions about systems
that store or communicate data.

Most electronic communication channels are noisy. Telephone and radio
channels are more obviously so, but even coaxial cables and fiber
optic links will occasionally ``flip a 0 to a 1,'' especially if we
try to push data through them at ever increasing rates. How can one
send data in an essentially errorfree manner over such noisy
communication channels? What is the ultimate transmission rate that a
channel can support?

Many electronic storage systems employ some form of data compression.
Data compression is also used for transmission in order to reduce the
amount of data that needs to be transmitted. What is the ultimate
compression factor that can be achieved while guaranteeing perfect
recovery of the compressed data? What more can be achieved if one is
willing to tolerate some prespecified small distortion of the
recovered data (as in images or video)?
The answer to the first set of questions leads to the concept of
Channel Capacity, and the second set to Entropy.
Information Theory is therefore an essential weapon in a
communication engineer's arsenal.
Information Theory, due to the nature of its subject matter, also
makes fundamental contributions to statistical physics
(thermodynamics), computer science (string complexity), economics
(optimal portfolios), probability theory (large deviations) and
statistics (Fisher information, hypothesis testing). This makes
Information Theory a useful tool for students of other disciplines.
Instructor:
E mail:
Telephone:
Fax:
Office Location:
Office Hours:
Course Assistants:
E mail:
Office Location:
Office Hours:

Sanjeev Khudanpur
MyLastName at jhu dot edu
4105167024
4105165050
Hackerman Hall Room 325
By appointment: W 3:004:00. Please email Ms. Ruth Scally [rscally1] in advance.
Congyuan Yang and Dung Tran
[yangcy dot ee at gmail dot com] and [FirstName.n.Lastname at jhu dot edu] respectively
Hackerman 322 and Barton 320 respectively
Monday 3:004:00 PM and Wednesday 10:0011:00 AM respectively

Text and Reference Books
The Fall 2002 offering of the course will utilize the book by Cover
and Thomas as the primary textbook, with some additional reading
material from Csiszár and Körner.

T. M. Cover and J. A. Thomas, Elements of
Information Theory, 2nd Edition, John Wiley & Sons, 2006. (Available
from Barnes & Noble, Amazon and other sellers)

I. Csiszár and J. Körner, Information
Theory: Coding Theorems for Discrete Memoryless Systems,
Akadémiai Kiadó, Budapest, 1981.

S. Lin and D. J. Costello, Jr., Error Control
Coding: Fundamentals and Applications, Prentice Hall, Englewood
Cliffs, NJ, 1983.
Homeworks, Exams and Class Notes
Evaluation will be based on a combination of homework assignments
(20%), two midterm examinations (20% each, on March 14 and April 30), and a comprehensive final examination (40%) on May 14, 9AM 12 Noon.
Homework assignments and notes will be posted here throughout the
semester. Check here frequently!
An Important Note on Academic Ethics:
The strength of the university depends on academic and personal
integrity. In this course, you must be honest and truthful.
Ethical violations include cheating on exams, plagiarism, reuse of
assignments, improper use of the Internet and electronic
devices, unauthorized collaboration, alteration of graded assignments,
forgery and falsification, lying, facilitating academic dishonesty,
and unfair competition. Report any violations you witness to the
instructor. You may consult the associate dean of students and/or the
chairman of the Ethics Board beforehand. See the guide on Academic Ethics for Undergraduates and the Ethics Board web site for more information.
Old Homeworks
Homework assignments from the Fall 2002 offering of the course are
provided below as examples. These or similar homeworks will be
assigned in Spring 2014. See above for the actual homeworks for this semester.

Homework Assignment #1: Due Sep 16, 2002 ( pdf, ps)

Homework Assignment #2: Due Sep 23, 2002 ( pdf, ps)

Homework Assignment #3: Due Sep 30, 2002 ( pdf, ps)

Homework Assignment #4: Due Oct 8, 2002 ( pdf, ps)

Homework Assignment #5: Due Oct 15, 2002 ( pdf, ps)

Midterm Examination #1: Given Oct 16, 2002 ( pdf, ps)

Homework Assignment #6: Due Oct 22, 2002 ( pdf, ps)

Homework Assignment #7: Due Oct 29, 2002 ( pdf, ps)

Reading Assignment: Due Oct 29, 2002 ( pdf, ps)

Homework Assignment #8: Due Nov 12, 2002 ( pdf, ps)

Homework Assignment #9: Due Nov 20, 2002 ( pdf, ps)

Midterm Examination #2: Given Nov 25, 2002 ( pdf, ps)

Homework Assignment #10: Due Dec 3, 2002 ( pdf, ps)

Reading Assignment: Due Dec 9, 2002 ( pdf, ps)

(Optional) Homework Assignment #11: Due Dec 11, 2002 ( pdf, ps)
Old Exams

Fall 2002 Midterm Exam #1 ( pdf, ps)

Fall 2002 Midterm Exam #2 ( pdf, ps)

Fall 2002 Final Exam ( pdf, ps)

Fall 2001 Midterm Exam #1 ( pdf, ps)

Fall 2001 Midterm Exam #2 ( pdf, ps)

Fall 2001 Final Exam ( pdf, ps)

Spring 2000 Midterm Exam #1 ( pdf, ps)

Spring 2000 Midterm Exam #2 ( pdf, ps)

Spring 2000 Final Exam ( pdf, ps)
Old Notes
Some notes on Joint Typicality ( pdf, ps)
Some notes on Rate Distortion ( pdf, ps)
Some notes on Relative Entropy ( pdf, ps)
Some notes on the Method of Types (large pdf file, or even larger ps file)