Understanding quantization from an information-theoretic perspective allows you to analyze how reducing the precision of neural network parameters impacts the network's ability to represent and process information. At the heart of information theory is the concept of entropy, which measures the average amount of information produced by a stochastic source of data. In the context of neural networks, entropy can be used to quantify the uncertainty or information content in the distribution of the network's parameters.

Mathematically, for a discrete random variable $X$ with possible values ${x_1, ..., x_n}$ and probability mass function $P(X)$, the entropy $H(X)$ is defined as:

When you apply quantization, you reduce the number of possible values that each parameter can take. This reduction effectively lowers the entropy of the parameter distribution, as the quantized parameters are now restricted to a smaller set of discrete levels. The process of mapping continuous or high-precision values to fewer quantized levels discards some information, leading to a decrease in entropy.

A key consequence of quantization is the introduction of quantization noise, which affects the signal-to-noise ratio (SNR) in neural network representations. The SNR is a measure of how much useful signal remains relative to the noise introduced by quantization. For a signal $x$ quantized to $Q(x)$, the quantization noise is the difference $x - Q(x)$. The SNR can be calculated as:

If the quantization noise is assumed to be uniformly distributed and uncorrelated with the signal, and the original signal has variance $\sigma_x^2$ while the quantization noise has variance $\sigma_q^2$, then:

and in decibels:

Higher SNR values indicate that the quantized representation retains more of the original signal's fidelity, which is crucial for maintaining model accuracy.

Reducing the precision of neural network parameters inherently limits the information capacity and expressiveness of the model. When you quantize weights and activations to fewer bits, the network's ability to represent subtle patterns or complex relationships in data is diminished. This is because the set of possible values that each parameter can take becomes smaller, shrinking the overall representational space of the network. As a result, certain functions that could be modeled with high-precision parameters may become impossible or less accurate to approximate with quantized parameters. The trade-off between efficiency (from lower precision) and expressiveness (from higher capacity) is a central consideration in quantized model design.