Announcement

  • 2019/02/13 - Website established.
  • 2019/02/18 - Update Week 1 Lecture.
  • 2019/02/25 - Update Week 2 Lecture, Homework 1 announced.
  • 2019/03/11 - Update Week 3 Lecture, Homework 2 announced.
  • 2019/03/18 - Update Week 5 Lecture
  • 2019/03/25 - Update Week 6 Lecture, reading assignment.
  • 2019/04/01 - Update Week 7 Lecture. Midterm presentation registry. Please fill in before 04/08 14:00.
  • 2019/04/08 - Update Week 8 Lecture. Programming homework (Midterm) announced. Midterm presentation schedule 10-15 mins for each person. Start at 2:00 PM April 15. Don't be late.

  • Last Edited Time : 19/04/22 05:00 PM
  • Course Information

  • Lecturer: Prof. 吳家麟 (wjl@csie.ntu.edu.tw) R214
  • TA: 陳品君 (r06944025@ntu.edu.tw) R505
              黃昌第 (r06944017@ntu.edu.tw) R505
  • TA Hour : By appointment via email.
  • itct@cmlab.csie.ntu.edu.tw
  • Course Requirements

  • Homework: 20%
  • In-Class Written Exam: 20%
  • Midterm: 25%
  • Term Project : 35%
  • Outline

  • Basics of Information Theory
    - Entropy, Relative Entropy, Mutual Information.
  • Data Compression / Compaction
    - Kraft Inequality, the prefix condition and Instantaneous decodable codes.
  • Variable Length Codes
    - Huffman code, Arithmetic code and L-Z code.
  • Coding Techniques
    - DPCM (predictive coding).
    - Transform coding (Discrete Cosine Transform).
    - JPEG (JPEG2000).
    - Motion Estimation and Compression.
    - MPEG-1, 2, 4.
    - H.26P, H.264, HEVC.
    - Distributed Video Coding
  • Steganography and Information Hiding
    - Digital Watermarking.
  • Homework

  • For late submission, contact TA to hand-in.
    Score *= (0.8)^(late days)
  • 2019/02/25 - Homework 1
    Due date : 2019/03/18, handwritten or printed and submit in the class.
    You must finish the "Must done" section. The "Recommended" part is NOT included in grading.
  • 2019/03/11 - Homework 2
    Due date : 2018/04/01, handwritten or printed and submit in the class.
  • 2019/04/08 - Programming Homework (Midterm)
    JPEG introduction,
    homework specification,
    testing images,
    JPEG specification,
    parser (for debug),
    practical reference zh 1,
    practical reference zh 2,
    practical reference en,
    Due: 2019/04/28 11:59 PM. Please submit to itct@cmlab.csie.ntu.edu.tw
  • Grades

  • TBC
  • Lecture Notes

    Week 1: Introduction

    Why Information is important.pptx   (Last Modified: 2019/02/18 07:15 PM)
    Lecture 1.1.pdf   (Last Modified: 2019/02/18 03:06 PM)
    Lecture 1.1.mp4 (Last Modified: 2019/02/18 03:06 PM)
    Lecture 1.2.pdf   (Last Modified: 2019/02/18 07:17 PM)
    Lecture 1.2.mp4 (Last Modified: 2019/02/18 03:06 PM)

    Week 2: Entropy, Relative Entropy and Mutual Information

    Lecture 2.pdf   (Last Modified: 2019/02/25 02:49 PM)
    Lecture 2.mp4 (Last Modified: 2019/02/25 06:22 PM)
    Lecture 2_Extra1.pdf (Last Modified: 2019/02/25 04:59 PM)
    Lecture 2_Extra2.pdf (Last Modified: 2019/02/25 05:04 PM)

    Week 3: The Asymptotic Equipartition Property and Entropy Rates

    Lecture 3.pdf   (Last Modified: 2019/03/11 02:11 PM)
    Lecture 3.mp4 (Last Modified: 2019/03/11 02:11 PM)

    Week 4: The Asymptotic Equipartition Property and Entropy Rates - 2

    Same as last week.

    Week 5: Channel and Channel Capacity

    Lecture 4.1.pdf   (Last Modified: 2019/03/18 02:34 PM)
    Lecture 4.1.mp4 (Last Modified: 2019/03/18 02:34 PM)
    Lecture 4.2.pdf   (Last Modified: 2019/03/18 02:34 PM)
    Lecture 4.2.mp4 (Last Modified: 2019/03/18 02:34 PM)
    Lecture 4.3.pdf   (Last Modified: 2019/03/18 02:34 PM)
    Lecture 4.3.mp4 (Last Modified: 2019/03/18 02:34 PM)

    Week 6: Data Compression & Huffman Codes

    Lecture 5.pdf   (Last Modified: 2019/03/25 04:55 PM)
    Lecture 5.mp4 (Last Modified: 2019/03/25 04:55 PM)
    Lecture 6.1.pdf   (Last Modified: 2019/03/25 04:55 PM)
    Lecture 6.1.mp4 (Last Modified: 2019/03/25 04:55 PM)

    Week 7: Image Data Compression

    Lecture 6.2.pdf   (Last Modified: 2019/04/02 12:04 AM)
    Lecture 6.2.mp4 (Last Modified: 2019/04/02 12:04 AM)
    Lecture 9.1.pdf   (Last Modified: 2019/04/02 12:04 AM)
    Lecture 9.1.mp4 (Last Modified: 2019/04/02 12:04 AM)

    Week 8: Image Data Compression - 2

    Same as last week.

    Week 9: Oral Presentation

    Week 10: Arithmetic Coding

    Lecture 7.1.pdf   (Last Modified: 2019/04/22 02:44 PM)
    Lecture 7.1.mp4 (Last Modified: 2019/04/22 02:44 PM)
    Lecture 7.2.pdf   (Last Modified: 2019/04/22 04:58 PM)
    Lecture 7.2.mp4 (Last Modified: 2019/04/22 04:58 PM)
    Lecture 7.3.pdf   (Last Modified: 2019/04/22 04:58 PM)
    Lecture 7.3.mp4 (Last Modified: 2019/04/22 04:58 PM)



    Week 9: Transform Coding Technique(DCT)

    Lecture 9.2.pdf   (Last Modified: 2018/04/16 01:04 AM)
    Lecture 9.2.mp4 (Last Modified: 2018/04/16 01:04 AM)
    Lecture 9.3.pdf   (Last Modified: 2018/04/16 01:04 AM)
    Lecture 9.3.mp4 (Last Modified: 2018/04/16 01:04 AM)
    Lecture 10.1.pdf   (Reading) (Last Modified: 2018/04/23 10:21 PM)
    Lecture 10.1.mp4 (Reading) (Last Modified: 2018/04/23 10:21 PM)
    Lecture 10.2.pdf   (Last Modified: 2018/04/23 10:21 PM)
    Lecture 10.2.mp4 (Last Modified: 2018/04/23 10:21 PM)
    Lecture 10.3.pdf   (Last Modified: 2018/04/23 10:21 PM)
    Lecture 10.3.mp4 (Last Modified: 2018/04/23 10:21 PM)

    Week 10: Oral Presentation


    Week 11: An Overview of Video Coding Algorithms

    Lecture 11.3.pdf   (Last Modified: 2014/05/07 08:21 AM)
    Lecture 11.3.mp4 (Last Modified: 2014/05/07 08:21 AM)

    Week 12: An Overview of Video Coding Algorithms - 2

    Lecture 11.1.pdf   (Last Modified: 2014/05/07 08:21 AM)
    Lecture 11.1.mp4 (Last Modified: 2014/05/07 08:21 AM)
    Lecture 11.2.pdf   (Last Modified: 2014/05/07 08:21 AM)
    Lecture 11.2.mp4 (Last Modified: 2014/05/07 08:21 AM)
    Lecture 11.4.pdf   (Last Modified: 2014/05/14 07:51 PM)

    Week 13: Rate Distortion Function and Optimal Bit-AllocationCoding Algorithms & Arithmetic Code

    Lecture 12.pdf   (Last Modified: 2014/05/20 10:44 PM)
    Lecture 12.mp4 (Last Modified: 2014/05/20 10:44 PM)



    Week 14: Information Theory in Data Visualization

    Information Theory in Data Visualization-1.pdf. (Last Modified: 2016/05/16 01:16 PM)
    Information Theory in Data Visualization-1.mp4. (Last Modified: 2016/05/16 01:14 PM)
    Information Theory in Data Visualization-2.pdf. (Last Modified: 2016/05/16 01:16 PM)
    Information Theory in Data Visualization-2.mp4. (Last Modified: 2016/05/16 01:14 PM)

    Week 15: Arithmetic Code

    Week 16: LZ Code

    Lecture 8.1.pdf. (Last Modified: 2014/04/23 02:07 PM)
    Lecture 8.1.mp4. (Last Modified: 2014/04/23 02:08 PM)
    Lecture 8.2.pdf. (Last Modified: 2014/04/23 02:07 PM)
    Lecture 8.2.mp4. (Last Modified: 2014/04/23 02:07 PM)
    Lecture 8.3.pdf. (Last Modified: 2014/04/23 02:07 PM)
    Lecture 8.3.mp4. (Last Modified: 2014/04/23 02:07 PM)

    Week 17: Encryption then Compression Scheme

    Lecture 13.1.pdf. (Last Modified: 2014/04/23 02:10 PM)
    Lecture 13.1.mp4. (Last Modified: 2014/05/20 10:43 PM)
    Lecture 13.2.pdf. (Last Modified: 2014/04/23 02:10 PM)
    Lecture 13.2.mp4. (Last Modified: 2014/05/20 10:44 PM)
    Lecture 13.3.pdf. (Last Modified: 2014/04/23 02:10 PM)
    Lecture 13.3.mp4. (Last Modified: 2014/05/20 10:44 PM)

    Others:

    Lecture 15.1.pdf. (Last Modified: 2014/05/27 11:11 PM)
    Lecture 15.1.mp4. (Last Modified: 2014/05/27 11:11 PM)
    Lecture 15.2.pdf. (Last Modified: 2014/05/27 11:11 PM)
    Lecture 15.2.mp4. (Last Modified: 2014/05/27 11:11 PM)
    Lecture 15.3.pdf. (Last Modified: 2014/05/27 11:11 PM)
    Lecture 15.3.mp4. (Last Modified: 2014/05/27 11:11 PM)
    Advanced Video Compression.pdf (Last Modified: 2018/04/09 02:49 PM)
    Differential Entropy.pdf (Last Modified: 2018/04/09 02:46 PM)

    Resource & Reference

  • Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, Second Edition, Wiley & Sons, 2006.
  • Khalid Sayood, Introduction to data compression 2ed, Morgan Kaufmann, 2000.
  • JPEG Standard.
  • MPEG-1 Standard.
  • Darrel Hankerson, Greg A. Harris, and Peter D. Johnson Jr., Introduction to information theory and data compression, CRC, 1998.
  • The Minimum Description Length Principle, Peter D. Grunward, the MIT Press, 2007.
  • IEEE Trans. on IT, CSVT, SP, IP, MM, IFS, Comm, Comput
  • Information_Theory_Reference.pdf.
  • Reference from the Journal Entropy.pdf.