0. Overview of the Task
The full task list:
Survey existing LLMs and the background needed to quantize LLaMA-based models
Survey the available quantization tooling
Run comparative quantization experiments: quality and performance
Evaluate the experiments: quality (human? automated? …) and performance (speed, GPU memory before quantization -> after quantization)
Summarize and draw conclusions
What this article covers:
Survey existing LLMs and the background needed to quantize LLaMA-based models
Survey the available quantization tooling
1. The Current LLM Landscape
The dominant way of building on LLMs today is foundation model + prompt engineering: the big internet companies open-source the former, while developers tailor the latter to their own business scenarios.
Among the mainstream foundation models is the LLaMA family released by Meta.
The LLaMA series comes in four sizes: 7B, 13B, 33B, and 65B.
This project uses LLaMA 33B as its base model.
2. Background on Quantization
Q1. What is quantization?
Quantization is the process of approximating the continuous values of a signal with a finite set of discrete values. It is a form of model compression.
Q2. Why quantize?
Serving these models consumes a great deal of GPU memory, and under high concurrency the service may crash outright. Quantizing the model saves both memory and bandwidth.
Q3. What kinds of quantization are there?
Binarization uses bitwise operations to achieve parallel computation.
Logarithmic quantization has no acceleration library implemented on any of the three major platforms, presumably because the speedup it delivers is not significant. Only some special-purpose chips make use of it.
Linear quantization uses uniformly distributed cluster centers, so that a simple linear transformation relates the original floating-point data to the quantized fixed-point data. It is the most widely used approach. Song Han’s best-paper work at ICLR 2016, for example, was the first to propose parameter quantization: a clustering algorithm groups nearby values into a single class so that they can all reuse one value.
None of the three approaches above, however, seems applicable to LLMs.
3. Quantization Methods
1. LLM.int8
Proposed in 2022 by Tim Dettmers, a PhD student at the University of Washington.
Paper: https://arxiv.org/pdf/2208.07339.pdf
GitHub: https://github.com/timdettmers/bitsandbytes
(1) Introduction to int8
Before getting to LLM.int8, we need to understand plain int8 quantization. The core idea of int8 quantization is to map FP16 floating-point numbers into the 8-bit integer range, i.e. [-127, 127]. Suppose we want to quantize the vector $\mathbf{x} : [1.2, -0.5, -4.3, 1.2, -3.1, 0.8, 2.4, 5.4]$. We first find its maximum, $max(\mathbf{x}) = 5.4$, and then compute the quantization coefficient $\alpha = 127/5.4 = 23.5$. Multiplying every element of the vector by $\alpha$ gives the quantized vector $\mathbf{x’} : [28, -12, -101, 28, -73, 19, 56, 127]$.
The section above explains how to quantize a single int8 vector, but the method breaks down as soon as the vector contains outliers (emergent features). For instance, take the vector [-0.10, -0.23, 0.08, -0.38, -0.28, -0.29, -2.11, 0.34, -0.53, -67.0]. After int8 quantization and dequantization it becomes [ -0.00, -0.00, 0.00, -0.53, -0.53, -0.53, -2.11, 0.53, -0.53, -67.00] — clearly most of the information has been destroyed. Applying int8 quantization directly to a model therefore degrades its accuracy.
(2) Introduction to LLM.int8
Through experiments, the author observed the following phenomenon (this passage draws on Strong’s article on Zhihu) and proposed a mixed-precision quantization method, LLM.int8.
Outliers appear in virtually every layer, and applying int8 quantization blindly causes a severe accuracy drop. The good news is that these outliers are distributed in a regular pattern. If a 6.7B transformer model has 150,000 outliers per sequence, they will occur in only six feature dimensions (six distinct values of i in X[:, :, i]).
Building on this observation, the author proposed LLM.int8: separate the few dimensions that contain outliers out of the matrix, compute their matrix product in high precision, and quantize everything else. By the author’s account, 99.9% of the dimensions can be handled in int8, while the remaining 0.1% require fp16 multiplication.
The highlighted parts are the outliers; they are left unquantized and computed in fp16. The non-outlier parts are quantized with int8. They are scaled by the row-wise and column-wise absolute maxima $C_x$ and $C_w$, the outputs are quantized to Int8, an 8-bit matrix multiplication is performed, and the result is then dequantized.
(3) Accuracy and Efficiency of LLM.int8
On accuracy:
As shown below, the author experimented with different quantization schemes on models of 125M, 1.3B, 2.7B, 6.7B, and 13B parameters, and evaluated the perplexity of each model before and after quantization. Lower perplexity is better.
Both the symmetric and asymmetric variants of conventional quantization suffer large accuracy drops; compared with the other schemes, models quantized with LLM.int8 lose very little accuracy and are almost on par with fp32.
Note that once the model grows past a critical size, the accuracy of conventional int8 falls off sharply, whereas LLM.int8 barely degrades at all.
On speed:
There are currently two sources of speed measurements: the author’s blog and the paper.
From the blog:
The author compares per-token latency (ms) across model sizes. The experiments show that BLOOM-176B with LLM.int8 is 15% to 23% slower than the fp16 version; see the figure below for details. For the smaller models the slowdown is several times worse.
The author mentions that within a single day he cut the per-token inference latency of T5-3B (bottom row, fourth column from the left) from 312 ms to 173 ms. In other words, the latency in this figure comes mainly from the algorithm, and the team was still optimizing it at the time.
From the paper:
Speed of the 16-bit matrix multiplication in the first hidden layer relative to the baseline, for models of different sizes. Anything below 1.0x is a slowdown.
The two sets of measurements agree on some points and disagree on others. Where they agree:
- Both int8 quantization and LLM.int8 favor larger models in terms of speed
Where they differ:
- Under the per-token measurement, LLM.int8 is slower than fp16. Under the 16-bit hidden-layer matrix multiplication measurement, inference on small models slows down noticeably while inference on large models speeds up.
On GPU memory
(4) Practice
Using the third-party Linear8bitLt class:
import torch
import torch.nn as nn
import bitsandbytes as bnb
from bnb.nn import Linear8bitLt
fp16_model = nn.Sequential(
nn.Linear(64, 64),
nn.Linear(64, 64)
)
[... train the model ...]
torch.save(fp16_model.state_dict(), "model.pt")
int8_model = nn.Sequential(
Linear8bitLt(64, 64, has_fp16_weights=False),
Linear8bitLt(64, 64, has_fp16_weights=False)
)
# The has_fp16_weights flag matters a great deal here. It defaults to True, which enables Int8/FP16 mixed precision during training. For inference, however, we care more about saving memory, so we need has_fp16_weights=False.
int8_model.load_state_dict(torch.load("model.pt")) # not quantized yet at this point
int8_model = int8_model.to(0) # quantization happens here
# quantization only takes place once the model is moved onto the GPU
input_ = torch.randn(64, dtype=torch.float16)
hidden_states = int8_model(input_.to(torch.device('cuda', 0)))
Beyond this basic usage, the author also provides sample code targeting LLaMA models.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# maximum length of the generated text
MAX_NEW_TOKENS = 128
# name of the pretrained model to use
model_name = 'decapoda-research/llama-7b-hf'
# the text to process
text = 'Hamburg is in which country?\n'
# turn the text into the input format the model expects (input_ids is the integer sequence the model consumes)
tokenizer = AutoTokenizer.from_pretrained(model_name)
input_ids = tokenizer(text, return_tensors="pt").input_ids
# GPU-related settings
free_in_GB = int(torch.cuda.mem_get_info()[0]/1024**3)
max_memory = f'{int(torch.cuda.mem_get_info()[0]/1024**3)-2}GB'
n_gpus = torch.cuda.device_count()
max_memory = {i: max_memory for i in range(n_gpus)}
# load_in_8bit=True loads the model at 8-bit precision to reduce memory usage
model = AutoModelForCausalLM.from_pretrained(
model_name,
device_map='auto',
load_in_8bit=True,
max_memory=max_memory
)
generated_ids = model.generate(input_ids, max_length=MAX_NEW_TOKENS)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))
(5) Q&A
- Why does inference get slower after quantization?
Quantization requires splitting the matrix apart, and that extra work slows inference down.
- Why do small models slow down more?
The author has identified the cause: it comes down to the CUDA kernels. In short, int8 cannot saturate the GPU, so it runs at the same speed as fp16, while the extra overhead introduced by quantization drags inference down. In theory a 6B model should see a 20–40% speedup. The author cannot estimate the cost of that overhead, which is more complicated and depends on sequence length and batch size.
Six months ago the author said that speed would improve in later releases.
- What is the relationship between the number of outliers and time/space complexity?
Several of the author’s findings bear on this question, but as noted in question 2 the author himself has not fully worked out the overhead issue.
The emergence of outliers is not sudden but gradual, and it is exponentially related to perplexity rather than to model size.
After the phase shift occurs, outliers start to grow rapidly.
“Phase shift” means that outliers suddenly appear in all layers and begin to coordinate with one another.
(6) Summary
The method is fairly new, easy to use, essentially lossless in accuracy, and effective at reducing GPU memory. Whether its runtime speed falls within an acceptable range, however, still needs to be verified experimentally.
(7) References
Blog post by the author, Tim Dettmers: https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/
Blog post by Younes Belkada: https://huggingface.co/blog/zh/hf-bitsandbytes-integration
Video by Bilibili creator 米粒方糖: https://www.bilibili.com/video/BV1Tx4y1d7sG/?spm_id_from=333.880.my_history.page.click&vd_source=3a72dc49e723efef59bcf133fb8fe42e
The original paper: https://arxiv.org/pdf/2208.07339.pdf
2. GPTQ Quantization (work in progress)
Only 20 GB of GPU memory is needed to run LLaMA 33B.
Proposed by Elias Frantar of the Institute of Science and Technology Austria. The paper was published at ICLR 2023.
Paper: https://arxiv.org/pdf/2210.17323.pdf
GitHub (LLaMA version): https://github.com/qwopqwop200/GPTQ-for-LLaMa
(1) The Method
The lineage runs from OBD to OBS to OBQ and finally to GPTQ.
To be completed.
(2) Performance on LLaMA 33B
Experiments were run on an A100.
(3) Usage
python llama.py LLAMA_HF_FOLDER c4 --wbits 4 --true-sequential --act-order --new-eval
(4) Differences from LLM.int8
GPTQ quantizes an fp16 model directly into 4-bit format. LLM.int8, by contrast, reads fp16 weights and loads them into Linear8bitLt layers, with quantization happening when the model is moved to the GPU.
int8_model.load_state_dict(torch.load("model.pt"))
int8_model = int8_model.to(0) # quantization happens here
5. TODO
- Study the LLM.int8 method in depth
Understand the emergent feature phenomenon https://arxiv.org/pdf/2208.07339.pdf
In the paper “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”, what exactly do int8 absmax and zeropoint mean?
Where did the author’s inspiration for this method come from?
Does the conjecture hold that “outliers” exist so that the model can extract features? If it does, how much accuracy would be lost by keeping only the outliers? Could one design a method that retains a certain fraction of the non-outliers, so as to balance the performance loss against the model’s demand for compute resources?
What is the algorithm behind llama.cpp? https://github.com/ggerganov/llama.cpp
Flesh out the explanation of how GPTQ works by reading the original paper and blog posts. From Zhihu & arXiv
- Learn how to use GPTQ & LLM.int8 from others’ experience. From YouTube
