Machine Learning from Perspective of Information Theory

Schematic diagram of a general communication system.

New description of components based on machine learning objectives:

Component Function
information source produce messages based on distribution
transmitter operates on messages to produce signal
channel medium to transmit signal
receiver reconstruct messages from signal
destination consume messages to reconstruct distribution

Specific change:

In general communication system:

destination consumes messages

To learn source distribution:

destination consumes messages to reconstruct distribution

Parallels

New description of components based on machine learning objectives:

Component Function
information source distribution of data to be learned
transmitter draws sample + supervision from distribution
channel operates on samples to produce embeddings
receiver recover supervision from embedding
destination compare tx/rx supervision to learn distribution

Corresponding machine learning concepts:

Component Machine Learning concept
information source dataset
transmitter sampling strategy
channel model (conv layers)
receiver model (fc layers)
destination loss function

Signals and Noise