GPTQ 4位量化技术详解：大语言模型压缩与加速部署指南

gptq by davila7/claude-code-templates

157 周安装量

23,400 GitHub Stars

GitHub

安装命令

npx skills add https://github.com/davila7/claude-code-templates --skill gptq

AI/机器学习性能优化自然语言处理

🇨🇳中文介绍

GPTQ（生成式预训练变换器量化）

一种训练后量化方法，通过分组量化将大语言模型压缩至 4 位，同时保持最小的精度损失。

何时使用 GPTQ

在以下情况使用 GPTQ：

需要在有限的 GPU 内存上运行大型模型（70B+）
希望以 <2% 的精度损失换取 4 倍内存减少
在消费级 GPU（如 RTX 4090, 3090）上部署
需要更快的推理速度（相比 FP16 有 3-4 倍加速）

在以下情况改用 AWQ：

需要稍好一点的精度（损失 <1%）
拥有较新的 GPU（Ampere, Ada 架构）
希望获得 Marlin 内核支持（在某些 GPU 上快 2 倍）

在以下情况改用 bitsandbytes：

需要与 transformers 库简单集成
想要 8 位量化（压缩率较低，质量更好）
不需要预量化模型文件

快速开始

安装

# 安装 AutoGPTQ
pip install auto-gptq

# 安装带 Triton 支持的版本（仅限 Linux，速度更快）
pip install auto-gptq[triton]

# 安装带 CUDA 扩展的版本（速度更快）
pip install auto-gptq --no-build-isolation

# 完整安装
pip install auto-gptq transformers accelerate

加载预量化模型

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM

# 从 HuggingFace 加载量化模型
model_name = "TheBloke/Llama-2-7B-Chat-GPTQ"

model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_triton=False  # 在 Linux 上设置为 True 以获得速度提升
)

tokenizer = AutoTokenizer.from_pretrained(model_name)

# 生成文本
prompt = "Explain quantum computing"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))

广告位招租

在这里展示您的产品或服务

触达数万 AI 开发者，精准高效

联系我们

量化你自己的模型

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from datasets import load_dataset

# 加载模型
model_name = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 量化配置
quantize_config = BaseQuantizeConfig(
    bits=4,              # 4 位量化
    group_size=128,      # 分组大小（推荐：128）
    desc_act=False,      # 激活顺序（False 用于 CUDA 内核）
    damp_percent=0.01    # 阻尼因子
)

# 加载模型以进行量化
model = AutoGPTQForCausalLM.from_pretrained(
    model_name,
    quantize_config=quantize_config
)

# 准备校准数据
dataset = load_dataset("c4", split="train", streaming=True)
calibration_data = [
    tokenizer(example["text"])["input_ids"][:512]
    for example in dataset.take(128)
]

# 执行量化
model.quantize(calibration_data)

# 保存量化模型
model.save_quantized("llama-2-7b-gptq")
tokenizer.save_pretrained("llama-2-7b-gptq")

# 推送到 HuggingFace
model.push_to_hub("username/llama-2-7b-gptq")

GPTQ 的工作原理：

分组权重：将每个权重矩阵划分为多个组（通常每组 128 个元素）
按组量化：每个组拥有自己的缩放因子/零点
最小化误差：使用 Hessian 信息来最小化量化误差
结果：4 位权重，精度接近 FP16

分组大小的权衡：

分组大小	模型大小	精度	速度	推荐场景
-1（按列）	最小	最佳	最慢	仅限研究
32	较小	较好	较慢	需要高精度
128	中等	良好	快速	推荐默认值
256	较大	较低	更快	速度关键
1024	最大	最低	最快	不推荐

Weight matrix: [1024, 4096] = 4.2M elements

Group size = 128:
- Groups: 4.2M / 128 = 32,768 groups
- Each group: own 4-bit scale + zero-point
- Result: Better granularity → better accuracy

标准 4 位（推荐）

from auto_gptq import BaseQuantizeConfig

config = BaseQuantizeConfig(
    bits=4,              # 4 位量化
    group_size=128,      # 标准分组大小
    desc_act=False,      # 更快的 CUDA 内核
    damp_percent=0.01    # 阻尼因子
)

内存：减少 4 倍（70B 模型：140GB → 35GB）
精度：困惑度增加约 1.5%
速度：比 FP16 快 3-4 倍

高精度（3 位，大分组）

config = BaseQuantizeConfig(
    bits=3,              # 3 位（压缩率更高）
    group_size=128,      # 保持标准分组大小
    desc_act=True,       # 精度更好（速度较慢）
    damp_percent=0.01
)

内存：减少 5 倍
精度：困惑度增加约 3%
速度：快 5 倍（但精度较低）

最高精度（4 位，小分组）

config = BaseQuantizeConfig(
    bits=4,
    group_size=32,       # 更小的分组（精度更好）
    desc_act=True,       # 激活重排序
    damp_percent=0.005   # 更低的阻尼因子
)

内存：减少 3.5 倍（稍大）
精度：困惑度增加约 0.8%（最佳）
速度：快 2-3 倍（内核开销）

ExLlamaV2（默认，最快）

model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_exllama=True,      # 使用 ExLlamaV2
    exllama_config={"version": 2}
)

性能：比 Triton 快 1.5-2 倍

Marlin（Ampere+ GPU）

# 使用 Marlin 格式量化
config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False  # Marlin 必需
)

model.quantize(calibration_data, use_marlin=True)

# 使用 Marlin 加载
model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_marlin=True  # 在 A100/H100 上快 2 倍
)

NVIDIA Ampere 或更新架构（A100, H100, RTX 40xx）
计算能力 ≥ 8.0

Triton（仅限 Linux）

model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_triton=True  # 仅限 Linux
)

性能：比 CUDA 后端快 1.2-1.5 倍

与 transformers 集成

直接使用 transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

# 加载量化模型（transformers 会自动检测 GPTQ）
model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Llama-2-13B-Chat-GPTQ",
    device_map="auto",
    trust_remote_code=False
)

tokenizer = AutoTokenizer.from_pretrained("TheBloke/Llama-2-13B-Chat-GPTQ")

# 像使用任何 transformers 模型一样使用
inputs = tokenizer("Hello", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)

QLoRA 微调（GPTQ + LoRA）

from transformers import AutoModelForCausalLM
from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model

# 加载 GPTQ 模型
model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Llama-2-7B-GPTQ",
    device_map="auto"
)

# 为 LoRA 训练做准备
model = prepare_model_for_kbit_training(model)

# LoRA 配置
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# 添加 LoRA 适配器
model = get_peft_model(model, lora_config)

# 微调（内存高效！）
# 70B 模型可在单张 A100 80GB 上训练

模型	FP16	GPTQ 4 位	减少倍数
Llama 2-7B	14 GB	3.5 GB	4×
Llama 2-13B	26 GB	6.5 GB	4×
Llama 2-70B	140 GB	35 GB	4×
Llama 3-405B	810 GB	203 GB	4×

70B 模型可在单张 A100 80GB 上运行（FP16 需要 2 张 A100）
405B 模型可在 3 张 A100 80GB 上运行（FP16 需要 11 张 A100）
13B 模型可在 RTX 4090 24GB 上运行（FP16 会内存溢出）

推理速度（Llama 2-7B, A100）

精度	令牌/秒	对比 FP16
FP16	25 tok/s	1×
GPTQ 4 位（CUDA）	85 tok/s	3.4×
GPTQ 4 位（ExLlama）	105 tok/s	4.2×
GPTQ 4 位（Marlin）	120 tok/s	4.8×

精度（在 WikiText-2 上的困惑度）

模型	FP16	GPTQ 4 位（g=128）	性能下降
Llama 2-7B	5.47	5.55	+1.5%
Llama 2-13B	4.88	4.95	+1.4%
Llama 2-70B	3.32	3.38	+1.8%

出色的质量保持 - 性能下降小于 2%！

# 自动设备映射
model = AutoGPTQForCausalLM.from_quantized(
    "TheBloke/Llama-2-70B-GPTQ",
    device_map="auto",  # 自动跨 GPU 分割
    max_memory={0: "40GB", 1: "40GB"}  # 每个 GPU 的内存限制
)

# 手动设备映射
device_map = {
    "model.embed_tokens": 0,
    "model.layers.0-39": 0,  # 前 40 层在 GPU 0 上
    "model.layers.40-79": 1,  # 后 40 层在 GPU 1 上
    "model.norm": 1,
    "lm_head": 1
}

model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device_map=device_map
)

# 将部分层卸载到 CPU（针对非常大的模型）
model = AutoGPTQForCausalLM.from_quantized(
    "TheBloke/Llama-2-405B-GPTQ",
    device_map="auto",
    max_memory={
        0: "80GB",  # GPU 0
        1: "80GB",  # GPU 1
        2: "80GB",  # GPU 2
        "cpu": "200GB"  # 将溢出部分卸载到 CPU
    }
)

# 高效处理多个提示
prompts = [
    "Explain AI",
    "Explain ML",
    "Explain DL"
]

inputs = tokenizer(prompts, return_tensors="pt", padding=True).to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    pad_token_id=tokenizer.eos_token_id
)

for i, output in enumerate(outputs):
    print(f"Prompt {i}: {tokenizer.decode(output)}")

寻找预量化模型

HuggingFace 上的 TheBloke：

https://huggingface.co/TheBloke
1000+ 个 GPTQ 格式的模型
多种分组大小（32, 128）
同时提供 CUDA 和 Marlin 格式

# 在 HuggingFace 上查找 GPTQ 模型
https://huggingface.co/models?library=gptq

from auto_gptq import AutoGPTQForCausalLM

# 自动从 HuggingFace 下载
model = AutoGPTQForCausalLM.from_quantized(
    "TheBloke/Llama-2-70B-Chat-GPTQ",
    device="cuda:0"
)

LLaMA 系列：Llama 2, Llama 3, Code Llama
Mistral：Mistral 7B, Mixtral 8x7B, 8x22B
Qwen：Qwen, Qwen2, QwQ
DeepSeek：V2, V3
Phi：Phi-2, Phi-3
Yi, Falcon, BLOOM, OPT
HuggingFace 上的 100+ 模型

校准指南 - 数据集选择、量化过程、质量优化
集成指南 - Transformers, PEFT, vLLM, TensorRT-LLM
故障排除 - 常见问题、性能优化

GitHub：https://github.com/AutoGPTQ/AutoGPTQ
论文：GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)
模型：https://huggingface.co/models?library=gptq
Discord：https://discord.gg/autogptq

2026 年 1 月 21 日

🇺🇸English

GPTQ (Generative Pre-trained Transformer Quantization)

Post-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.

When to use GPTQ

Use GPTQ when:

Need to fit large models (70B+) on limited GPU memory
Want 4× memory reduction with <2% accuracy loss
Deploying on consumer GPUs (RTX 4090, 3090)
Need faster inference (3-4× speedup vs FP16)

Use AWQ instead when:

Need slightly better accuracy (<1% loss)
Have newer GPUs (Ampere, Ada)
Want Marlin kernel support (2× faster on some GPUs)

Use bitsandbytes instead when:

Need simple integration with transformers
Want 8-bit quantization (less compression, better quality)
Don't need pre-quantized model files

Quick start

Installation

# Install AutoGPTQ
pip install auto-gptq

# With Triton (Linux only, faster)
pip install auto-gptq[triton]

# With CUDA extensions (faster)
pip install auto-gptq --no-build-isolation

# Full installation
pip install auto-gptq transformers accelerate

Load pre-quantized model

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM

# Load quantized model from HuggingFace
model_name = "TheBloke/Llama-2-7B-Chat-GPTQ"

model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_triton=False  # Set True on Linux for speed
)

tokenizer = AutoTokenizer.from_pretrained(model_name)

# Generate
prompt = "Explain quantum computing"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))

Quantize your own model

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from datasets import load_dataset

# Load model
model_name = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Quantization config
quantize_config = BaseQuantizeConfig(
    bits=4,              # 4-bit quantization
    group_size=128,      # Group size (recommended: 128)
    desc_act=False,      # Activation order (False for CUDA kernel)
    damp_percent=0.01    # Dampening factor
)

# Load model for quantization
model = AutoGPTQForCausalLM.from_pretrained(
    model_name,
    quantize_config=quantize_config
)

# Prepare calibration data
dataset = load_dataset("c4", split="train", streaming=True)
calibration_data = [
    tokenizer(example["text"])["input_ids"][:512]
    for example in dataset.take(128)
]

# Quantize
model.quantize(calibration_data)

# Save quantized model
model.save_quantized("llama-2-7b-gptq")
tokenizer.save_pretrained("llama-2-7b-gptq")

# Push to HuggingFace
model.push_to_hub("username/llama-2-7b-gptq")

Group-wise quantization

How GPTQ works :

Group weights : Divide each weight matrix into groups (typically 128 elements)
Quantize per-group : Each group has its own scale/zero-point
Minimize error : Uses Hessian information to minimize quantization error
Result : 4-bit weights with near-FP16 accuracy

Group size trade-off :

Group Size	Model Size	Accuracy	Speed	Recommendation
-1 (per-column)	Smallest	Best	Slowest	Research only
32	Smaller	Better	Slower	High accuracy needed
128	Medium	Good	Fast	Recommended default
256	Larger	Lower	Faster	Speed critical
1024	Largest	Lowest	Fastest	Not recommended

Example :

Weight matrix: [1024, 4096] = 4.2M elements

Group size = 128:
- Groups: 4.2M / 128 = 32,768 groups
- Each group: own 4-bit scale + zero-point
- Result: Better granularity → better accuracy

Quantization configurations

Standard 4-bit (recommended)

from auto_gptq import BaseQuantizeConfig

config = BaseQuantizeConfig(
    bits=4,              # 4-bit quantization
    group_size=128,      # Standard group size
    desc_act=False,      # Faster CUDA kernel
    damp_percent=0.01    # Dampening factor
)

Performance :

Memory: 4× reduction (70B model: 140GB → 35GB)
Accuracy: ~1.5% perplexity increase
Speed: 3-4× faster than FP16

High accuracy (3-bit with larger groups)

config = BaseQuantizeConfig(
    bits=3,              # 3-bit (more compression)
    group_size=128,      # Keep standard group size
    desc_act=True,       # Better accuracy (slower)
    damp_percent=0.01
)

Trade-off :

Memory: 5× reduction
Accuracy: ~3% perplexity increase
Speed: 5× faster (but less accurate)

Maximum accuracy (4-bit with small groups)

config = BaseQuantizeConfig(
    bits=4,
    group_size=32,       # Smaller groups (better accuracy)
    desc_act=True,       # Activation reordering
    damp_percent=0.005   # Lower dampening
)

Trade-off :

Memory: 3.5× reduction (slightly larger)
Accuracy: ~0.8% perplexity increase (best)
Speed: 2-3× faster (kernel overhead)

Kernel backends

ExLlamaV2 (default, fastest)

model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_exllama=True,      # Use ExLlamaV2
    exllama_config={"version": 2}
)

Performance : 1.5-2× faster than Triton

Marlin (Ampere+ GPUs)

# Quantize with Marlin format
config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False  # Required for Marlin
)

model.quantize(calibration_data, use_marlin=True)

# Load with Marlin
model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_marlin=True  # 2× faster on A100/H100
)

Requirements :

NVIDIA Ampere or newer (A100, H100, RTX 40xx)
Compute capability ≥ 8.0

Triton (Linux only)

model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_triton=True  # Linux only
)

Performance : 1.2-1.5× faster than CUDA backend

Integration with transformers

Direct transformers usage

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load quantized model (transformers auto-detects GPTQ)
model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Llama-2-13B-Chat-GPTQ",
    device_map="auto",
    trust_remote_code=False
)

tokenizer = AutoTokenizer.from_pretrained("TheBloke/Llama-2-13B-Chat-GPTQ")

# Use like any transformers model
inputs = tokenizer("Hello", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)

QLoRA fine-tuning (GPTQ + LoRA)

from transformers import AutoModelForCausalLM
from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model

# Load GPTQ model
model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Llama-2-7B-GPTQ",
    device_map="auto"
)

# Prepare for LoRA training
model = prepare_model_for_kbit_training(model)

# LoRA config
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Add LoRA adapters
model = get_peft_model(model, lora_config)

# Fine-tune (memory efficient!)
# 70B model trainable on single A100 80GB

Performance benchmarks

Memory reduction

Model	FP16	GPTQ 4-bit	Reduction
Llama 2-7B	14 GB	3.5 GB	4×
Llama 2-13B	26 GB	6.5 GB	4×
Llama 2-70B	140 GB	35 GB	4×
Llama 3-405B	810 GB	203 GB	4×

Enables :

70B on single A100 80GB (vs 2× A100 needed for FP16)
405B on 3× A100 80GB (vs 11× A100 needed for FP16)
13B on RTX 4090 24GB (vs OOM with FP16)

Inference speed (Llama 2-7B, A100)

Precision	Tokens/sec	vs FP16
FP16	25 tok/s	1×
GPTQ 4-bit (CUDA)	85 tok/s	3.4×
GPTQ 4-bit (ExLlama)	105 tok/s	4.2×
GPTQ 4-bit (Marlin)	120 tok/s	4.8×

Accuracy (perplexity on WikiText-2)

Model	FP16	GPTQ 4-bit (g=128)	Degradation
Llama 2-7B	5.47	5.55	+1.5%
Llama 2-13B	4.88	4.95	+1.4%
Llama 2-70B	3.32	3.38	+1.8%

Excellent quality preservation - less than 2% degradation!

Common patterns

Multi-GPU deployment

# Automatic device mapping
model = AutoGPTQForCausalLM.from_quantized(
    "TheBloke/Llama-2-70B-GPTQ",
    device_map="auto",  # Automatically split across GPUs
    max_memory={0: "40GB", 1: "40GB"}  # Limit per GPU
)

# Manual device mapping
device_map = {
    "model.embed_tokens": 0,
    "model.layers.0-39": 0,  # First 40 layers on GPU 0
    "model.layers.40-79": 1,  # Last 40 layers on GPU 1
    "model.norm": 1,
    "lm_head": 1
}

model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device_map=device_map
)

CPU offloading

# Offload some layers to CPU (for very large models)
model = AutoGPTQForCausalLM.from_quantized(
    "TheBloke/Llama-2-405B-GPTQ",
    device_map="auto",
    max_memory={
        0: "80GB",  # GPU 0
        1: "80GB",  # GPU 1
        2: "80GB",  # GPU 2
        "cpu": "200GB"  # Offload overflow to CPU
    }
)

Batch inference

# Process multiple prompts efficiently
prompts = [
    "Explain AI",
    "Explain ML",
    "Explain DL"
]

inputs = tokenizer(prompts, return_tensors="pt", padding=True).to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    pad_token_id=tokenizer.eos_token_id
)

for i, output in enumerate(outputs):
    print(f"Prompt {i}: {tokenizer.decode(output)}")

Finding pre-quantized models

TheBloke on HuggingFace :

https://huggingface.co/TheBloke
1000+ models in GPTQ format
Multiple group sizes (32, 128)
Both CUDA and Marlin formats

Search :

# Find GPTQ models on HuggingFace
https://huggingface.co/models?library=gptq

Download :

from auto_gptq import AutoGPTQForCausalLM

# Automatically downloads from HuggingFace
model = AutoGPTQForCausalLM.from_quantized(
    "TheBloke/Llama-2-70B-Chat-GPTQ",
    device="cuda:0"
)

Supported models

LLaMA family : Llama 2, Llama 3, Code Llama
Mistral : Mistral 7B, Mixtral 8x7B, 8x22B
Qwen : Qwen, Qwen2, QwQ
DeepSeek : V2, V3
Phi : Phi-2, Phi-3
Yi, Falcon, BLOOM, OPT
100+ models on HuggingFace

References

Calibration Guide - Dataset selection, quantization process, quality optimization
Integration Guide - Transformers, PEFT, vLLM, TensorRT-LLM
Troubleshooting - Common issues, performance optimization

Resources

GitHub : https://github.com/AutoGPTQ/AutoGPTQ
Paper : GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)
Models : https://huggingface.co/models?library=gptq
Discord : https://discord.gg/autogptq

Weekly Installs

137

Repository

davila7/claude-…emplates

GitHub Stars

22.6K

First Seen

Jan 21, 2026

Security Audits

Gen Agent Trust HubPass SocketPass SnykWarn

Installed on

claude-code112

opencode108

gemini-cli101

cursor99

codex90

antigravity89

AI 代码实施计划编写技能 | 自动化开发任务分解与 TDD 流程规划工具

47,700 周安装