Information theory


Teacher coordinatorAlexandrina Rogozan
Teacher(s)Alexandrina Rogozan
TeachingLectures : 21h Exercises : 31h30
Web site

Aim and objective

Introduction to the field of information theory and its applications to the communication theory. Presentation of the basic quantities for entropy and mutual information. Presentation of the basic algorithms for source coding, channel coding and cryptography.

Outcome learning

  • INSA reference data : 1 - Notion, 2 - Concept, 3 - Application, I - fully, P - incomplete

    Course description

    Mathematical tools :

    • Probabilities, entropy and mutual information,
    • Entropy rate of stochastic processes,
    • Rate distortion theory,
    • Markov models.

    Source coding :

    • Kraft inequality, optimal codes, bounds on the optimal codelenght,
    • Data compression without distortion : Shanon-Fano, Huffman, Run-lenght coding, Lempel- Ziv, arithmetic algorithms,
    • Data compression algorithms with distortion : scalar quantification, vectorial quantification, linear prediction,
    • Audio coding (MP3, AAC), image coding (JPEG) and video coding (MPEG).

    Chanel coding :

    • Channel capacity,
    • Error detection and/or correction algorithms,
    • Hamming codes,
    • Binary cyclic codes and BCH codes
    • Binary convolutional codes
    • Combining codes



    Signal processing, Matlab/Octave programming and probabilities.


    • [Th. Cover et J. Thomas, 1991] "Elements of Information Theory", Wiley Series in Telecommunications ed., 1991, 542 pages
    • [R. Morelos-Zaragoza, 2002] "The art of error correcting coding", Wiley Series, 221 pages.


    • Final exam (60%) : Written exam (3h)
    • Oral exam (40%) : Project presentation