Part III — Language Models (TBD)
6
From Transmitters to Transformers (TBD)
From Bits to Embeddings – A Critical Introduction to Information Theory
Part I — Introduction
1
Why Information Theory?
Part II — Information Theory
2
Bit: The Difference which Makes a Difference
3
Entropy: Expected Differences
4
KL Divergence: Differences Between Expected Differences
Part III — Language Models (TBD)
5
Word2Vec as Noisy Channel (TBD)
6
From Transmitters to Transformers (TBD)
Part III — Language Models (TBD)
6
From Transmitters to Transformers (TBD)
6
From Transmitters to Transformers (TBD)
This section remains to be polished enough for publication. Estimating it will be out in early 2026.
5
Word2Vec as Noisy Channel (TBD)