LLaMA Quantization Report (Part 2)
The LLaMA Quantization Report Series
0. Overview of the Task
- Study the LLM.int8 method in depth
Understand the emergent feature phenomenon https://arxiv.org/pdf/2208.07339.pdfIn the paper “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”, what exactly do int8 absmax and zeropoint mean?Where did the author’s inspiration for this method come from?Does the conjecture hold that “outliers” exist so that the model can extract features? If it does, how much accuracy would be lost by keeping only the outliers? Could one design a method that retains a certain fraction of the non-outliers, so as to balance the performance loss against the model’s demand for compute resources?
- What is the algorithm behind llama.cpp? https://github.com/ggerganov/llama.cpp
8-bit Optimizers via Block-wise Quantization
Author: Tim Dettmers
A method for quantizing the optimizer: without changing any hyperparameters or hurting model accuracy, it quantizes the Adam / momentum state to int8, thereby easing GPU memory pressure during training.
Not closely related to this project.
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Author: Tim Dettmers
0. Background
Absmax quantization:
The simplest quantization scheme, taking fp16$\rightarrow$int8.
In short, it maps fp16 values onto int8 integers in [-127, 127].
For example, given the fp16 vector [-100, 0, 50, 100], absmax quantization maps 100 to 127, 50 to 63.5, 0 to 0, and -100 to -98.73.
The trouble is that if the vector contains large-valued outliers, the other values get wiped out by the quantization.
For instance, take the vector [-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53, -67.0]. After int8 quantization and dequantization it becomes [ -0.00, -0.00, 0.00, -0.53, -0.53, -0.53, -2.11, 0.53, -0.53, -67.00] — clearly most of the information has been lost.
Zeropoint quantization:
The idea is similar to absmax; the difference is that absmax is symmetric while this method is asymmetric.
Taking the vector [-100, 0, 50, 100] as an example, zeropoint quantization maps 100 to 127, 50 to 63.5, 0 to 0, and -100 to -127.
The two methods are equivalent in approach, performance, and accuracy, and neither solves the outlier problem.
LLM.int8:
The author first tried quantizing models with both of the methods above, but accuracy dropped severely, so he set out to find an approach that preserves accuracy while still reducing memory usage.
Through experiments the author found that the matrix $\textbf{X}_{f16}\in\mathbb{R}^{s\times h}$ contains outliers, and that while these outliers occur across all of the sequence dimension s, their distribution along the feature/hidden dimension h follows a clear pattern.
Based on this observation, the author had the idea of separating out the dimensions that contain outliers and computing them in fp16, while computing the remaining dimensions in int8.
How outliers are defined:
The author sets a threshold $\alpha$; any value exceeding $\alpha$ is defined as an outlier, and any dimension containing an outlier must be computed in fp16. Experimentally, the author found that setting $\alpha$ to 6 or above keeps accuracy from degrading. With $\alpha$ set to 6, 99.9% of the dimensions are quantized to int8, while the remaining 0.1% that contain outliers are left unquantized. For a 13B model, no more than 7 dimensions contain outliers. Splitting the matrix costs an extra 0.1% of storage.
1. Emergent Feature
Definition:
The original text reads: “Emergence is a gradual change in a property that suddenly undergoes a phase shift and then changes the quality of its substrate.” In plain terms, the transformer changes gradually, then abruptly undergoes a phase shift, after which a qualitative change takes place — quantitative change leading to qualitative change. What changes over the course of this process is the model’s parameter count.
Findings:
Before explaining this process, the author introduces two things that happen in parallel inside a transformer: the network gradually extracts features, and at the same time it strips away noisy, context-irrelevant ones. An example makes this concrete. To classify cats and dogs, you can apply sharpening, which both accentuates the features that differ between the two animals (eyes, ears) and removes the features they share (color, underlying texture). The transformer works on the same principle. During inference, the transformer discards 99% of the useless information and retains only the 1% that is useful.
Before the qualitative change, different layers use different dimensions to accentuate the important features. After the qualitative change, all layers use the same six dimensions to do so.
In summary:
Below 6.7B parameters, outliers are scattered across different dimensions in individual layers with no discernible pattern.
At 6.7B parameters, the outliers’ values jump sharply and they suddenly appear in all layers, and only in the same six dimensions. This also explains why earlier quantization methods do not carry over to LLMs: from 6.7B onward there are large numbers of outliers, and earlier methods cannot handle them well.
Values jump sharply: the maximum is 15 at 6B, 60 at 13B, and 95 at 66B.
Suddenly appear in all layers: below 6.7B, different layers use different dimensions to amplify features. Above 6.7B, all layers work together to amplify them.
Only in the same six dimensions: this means attention has become concentrated.
2. Reflections
- Does the emergent feature depend only on the parameter count? No — it is also related to perplexity (ppl).
Why do ordinary quantization methods, such as zeropoint and absmax in the int8 family, fail on LLMs? Because as the parameter count grows so does the number of outliers, and ordinary quantization methods suffer severe accuracy loss whenever they encounter outliers.
To summarize the phenomena that accompany the qualitative change:
Outliers appear in all layers
Outliers are distributed regularly across a handful of dimensions
Outlier values rise sharply
The number of outliers also increases
- Can the dimensions other than the outliers be removed? No. The author notes in his blog that removing 95% of the parameters of a vision model does not seriously hurt accuracy, but for NLP transformers that figure drops to 30%, and once emergence has occurred it drops to 5%.
