ECE 520.447 Introduction to Information Theory and Coding
Fall 2002
Mon Tue Wed 9:00-9:50 PM in Barton 117
Information Theory addresses some fundamental questions about systems
that store or communicate data.
-
Most electronic communication channels are noisy. Telephone and radio
channels are more obviously so, but even coaxial cables and fiber
optic links will occasionally ``flip a 0 to a 1,'' especially if we
try to push data through them at ever increasing rates. How can one
send data in an essentially error-free manner over such noisy
communication channels? What is the ultimate transmission rate that a
channel can support?
-
Many electronic storage systems employ some form of data compression.
Data compression is also used for transmission in order to reduce the
amount of data that needs to be transmitted. What is the ultimate
compression factor that can be achieved while guaranteeing perfect
recovery of the compressed data? What more can be achieved if one is
willing to tolerate some prespecified small distortion of the
recovered data (as in images or video)?
The answer to the first set of questions leads to the concept of
Channel Capacity, and the second set to Entropy.
Information Theory is therefore an essential weapon in a
communication engineer's arsenal.
Information Theory, due to the nature of its subject matter, also
makes fundamental contributions to statistical physics
(thermodynamics), computer science (string complexity), economics
(optimal portfolios), probability theory (large deviations) and
statistics (Fisher information, hypothesis testing). This makes
Information Theory a useful tool for students of other disciplines.
Text and Reference Books
The Fall 2002 offering of the course will utilize the book by Cover
and Thomas as the primary textbook, with some additional reading
material from Csiszár and Körner. Chapter 3 from Lin and
Costello will be used to cover linear block codes.
-
T. M. Cover and J. A. Thomas, Elements of
Information Theory, John Wiley & Sons, New York, NY, 1991. (Available
from the campus
bookstore as well as
Bookpool and
Amazon)
-
I. Csiszár and J. Körner, Information
Theory: Coding Theorems for Discrete Memoryless Systems,
Akadémiai Kiadó, Budapest, 1981. (Requested for Course
Reserve at MSEL)
-
S. Lin and D. J. Costello, Jr., Error Control
Coding: Fundamentals and Applications, Prentice Hall, Englewood
Cliffs, NJ, 1983.
Homeworks, Exams and Class Notes
Evaluation will be based on a combination of homework assignments
(20%), two midterm examinations (20% each), and a comprehensive final
examination (40%).
Homework assignments and notes will be posted here throughout the
semester. Check here frequently!
-
Homework Assignment #1: Due Sep 16, 2002 ( pdf, ps)
-
Homework Assignment #2: Due Sep 23, 2002 ( pdf, ps)
-
Homework Assignment #3: Due Sep 30, 2002 ( pdf, ps)
-
Homework Assignment #4: Due Oct 8, 2002 ( pdf, ps)
-
Homework Assignment #5: Due Oct 15, 2002 ( pdf, ps)
-
Midterm Examination #1: Given Oct 16, 2002 ( pdf, ps)
-
Homework Assignment #6: Due Oct 22, 2002 ( pdf, ps)
-
Homework Assignment #7: Due Oct 29, 2002 ( pdf, ps)
-
Reading Assignment: Due Oct 29, 2002 ( pdf, ps)
-
Homework Assignment #8: Due Nov 12, 2002 ( pdf, ps)
-
Homework Assignment #9: Due Nov 20, 2002 ( pdf, ps)
-
Midterm Examination #2: Given Nov 25, 2002 ( pdf, ps)
-
Homework Assignment #10: Due Dec 3, 2002 ( pdf, ps)
-
Reading Assignment: Due Dec 9, 2002 ( pdf, ps)
-
(Optional) Homework Assignment #11: Due Dec 11, 2002 ( pdf, ps)
An Important Note on Academic Ethics:
The strength of the university depends on academic and personal
integrity. In this course, you must be honest and truthful.
Ethical violations include cheating on exams, plagiarism, reuse of
assignments, improper use of the Internet and electronic
devices, unauthorized collaboration, alteration of graded assignments,
forgery and falsification, lying, facilitating academic dishonesty,
and unfair competition. Report any violations you witness to the
instructor. You may consult the associate dean of students and/or the
chairman of the Ethics Board beforehand. See the guide on Academic
Ethics for Undergraduates and the Ethics Board web site for more information.
Old Homeworks
Homework assignments from the Fall 2001 offering of the course are
provided below as examples. These or similar homeworks will be
assigned in Fall 2002. See above for the actual homeworks for this semester.
Homework Assignment #1: Due Sep 19, 2001 ( pdf, ps)
Homework Assignment #2: Due Sep 26, 2001 ( pdf, ps)
Homework Assignment #3: Due Oct 3, 2001 ( pdf, ps)
Homework Assignment #4: Due Oct 9, 2001 ( pdf, ps)
Homework Assignment #5: Due Oct 29, 2001 ( pdf, ps)
Homework Assignment #6: Due Nov 5, 2001 ( pdf, ps)
Homework Assignment #7: Due Nov 12, 2001 ( pdf, ps)
Homework Assignment #8: Due Nov 19, 2001 ( pdf, ps)
Homework Assignment #9: Due Nov 26, 2001 ( pdf, ps)
Homework Assignment #10: Due Dec 3, 2001 ( pdf, ps)
Old Exams
Fall 2002 Midterm Exam #1 ( pdf, ps)
Fall 2002 Midterm Exam #2 ( pdf, ps)
Fall 2002 Final Exam ( pdf, ps)
Fall 2001 Midterm Exam #1 ( pdf, ps)
Fall 2001 Midterm Exam #2 ( pdf, ps)
Fall 2001 Final Exam ( pdf, ps)
Spring 2000 Midterm Exam #1 ( pdf, ps)
Spring 2000 Midterm Exam #2 ( pdf, ps)
Spring 2000 Final Exam ( pdf, ps)
Old Notes
Some notes on Joint Typicality ( pdf, ps)
Some notes on Rate Distortion ( pdf, ps)
Some notes on Relative Entropy ( pdf, ps)
Some notes on the Method of Types (large pdf file, or even larger ps file)
Instructor: Sanjeev Khudanpur
Office: Barton 221, 410-516-7024
email: MyLastName at jhu dot edu