When quantizing neural networks, every layer introduces its own quantization noise, which can accumulate as data passes through the network. To understand this propagation, consider a simple linear layer with input vector $x$, weight matrix $W$, and bias vector $b$. After quantization, the computation becomes:

where $ΔW$, $Δx$, and $Δb$ represent the quantization errors for the weights, inputs, and biases, respectively. Expanding this expression and neglecting higher-order error terms yields:

The total output error is thus:

This shows that the output error from a layer depends on both the quantization errors in the current layer ($ΔW$, $Δb$) and the errors propagated from previous layers ($Δx$). As a result, error accumulates recursively. For a stack of $L$ layers, the error at layer $l$ can be described recursively as:

with the initial error $Δx_0$ introduced at the input. For nonlinear layers, such as those involving activation functions like ReLU, the error propagation becomes more complex. If $f$ is a nonlinear function, the quantized output is:

where $f'(x)$ is the derivative of the activation function. This linear approximation shows that nonlinearities can scale or reshape the error, sometimes amplifying it depending on the activation's slope at $x$.

The sensitivity of different layer types to quantization noise is closely related to their mathematical structure. Multi-Layer Perceptrons (MLPs) consist of dense linear transformations followed by nonlinear activations. Quantization noise in MLPs is mostly shaped by the magnitude of the weights and the nonlinearity of the activation functions. If the weights are large or the activations are highly sensitive, small quantization errors can be amplified, especially in deeper stacks.

Attention layers, such as those used in transformer models, involve operations like scaled dot-product attention. These layers compute attention scores by taking dot products of queries and keys, scaling, and applying softmax. The softmax operation is particularly sensitive to input perturbations: small changes in the dot products, due to quantization, can cause large shifts in the resulting attention weights. This makes attention layers more susceptible to quantization-induced instability than typical MLP layers.

In summary, MLPs and attention layers differ in how quantization noise impacts them because of their distinct mathematical operations. Attention mechanisms, with their reliance on normalization and exponentiation, tend to amplify small errors more unpredictably, while MLPs primarily scale errors according to weight magnitudes and activation slopes.

As neural networks become deeper, the cumulative effect of quantization errors increases. Each layer not only introduces its own quantization noise but also propagates and potentially amplifies errors from all previous layers. In a deep stack, this can result in significant deviations from the intended computation, especially if the weights or activations magnify the errors at each step.

To mitigate this error amplification, several strategies can be employed:

• Choose quantization schemes with higher precision for sensitive layers;
• Apply normalization techniques, such as batch normalization, to control the scale of activations and errors;
• Use quantization-aware training to adapt parameters and activations to their quantized representations during learning;
• Regularize weights to prevent large values that can amplify quantization noise;
• Design architectures with skip connections or residual paths to help preserve information and limit error buildup.

By understanding the sources and propagation of quantization errors, you can design networks and quantization schemes that maintain numerical stability, even as network depth increases.