HQQ 半二次量化：无需校准的快速模型量化，支持 8/4/3/2/1 比特精度

hqq-quantization by davila7/claude-code-templates

189 周安装量

24,100 GitHub Stars

GitHub

安装命令

npx skills add https://github.com/davila7/claude-code-templates --skill hqq-quantization

AI/机器学习 PyTorch 性能优化

🇨🇳中文介绍

HQQ - 半二次量化

支持 8/4/3/2/1 比特精度的快速、无需校准的权重量化，提供多种优化后端。

何时使用 HQQ

在以下情况下使用 HQQ：

量化模型时没有校准数据（无需数据集）
需要快速量化（分钟级，而非 GPTQ/AWQ 的小时级）
使用 vLLM 或 HuggingFace Transformers 部署
使用 LoRA/PEFT 微调量化模型
尝试极限量化（2 比特、1 比特）

主要优势：

无需校准：无需样本数据，即可即时量化任何模型
多后端支持：PyTorch、ATEN、TorchAO、Marlin、BitBlas，用于优化推理
灵活精度：可配置分组大小的 8/4/3/2/1 比特量化
框架集成：原生支持 HuggingFace 和 vLLM
兼容 PEFT：可使用 LoRA 微调量化模型

替代方案使用场景：

AWQ：需要基于校准的精度，用于生产环境服务
GPTQ：在有校准数据时追求最高精度
bitsandbytes：简单的 8 比特/4 比特量化，无需自定义后端
llama.cpp/GGUF：CPU 推理，Apple Silicon 部署

快速开始

安装

pip install hqq

# 安装特定后端
pip install hqq[torch]      # PyTorch 后端
pip install hqq[torchao]    # TorchAO int4 后端
pip install hqq[bitblas]    # BitBlas 后端
pip install hqq[marlin]     # Marlin 后端

广告位招租

在这里展示您的产品或服务

触达数万 AI 开发者，精准高效

联系我们

使用 HuggingFace 量化完整模型

from transformers import AutoModelForCausalLM, HqqConfig

# 配置 HQQ
quantization_config = HqqConfig(
    nbits=4,
    group_size=64,
    axis=1
)

# 加载并量化
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=quantization_config,
    device_map="auto"
)

# 模型已量化并准备就绪

HQQ 使用 BaseQuantizeConfig 来定义量化参数：

from hqq.core.quantize import BaseQuantizeConfig

# 标准 4 比特配置
config_4bit = BaseQuantizeConfig(
    nbits=4,           # 每个权重的比特数 (1-8)
    group_size=64,     # 每个量化组的权重数量
    axis=1             # 0=输入维度, 1=输出维度
)

# 激进的 2 比特配置
config_2bit = BaseQuantizeConfig(
    nbits=2,
    group_size=16,     # 低比特量化使用更小的分组
    axis=1
)

# 按层类型混合精度
layer_configs = {
    "self_attn.q_proj": BaseQuantizeConfig(nbits=4, group_size=64),
    "self_attn.k_proj": BaseQuantizeConfig(nbits=4, group_size=64),
    "self_attn.v_proj": BaseQuantizeConfig(nbits=4, group_size=64),
    "mlp.gate_proj": BaseQuantizeConfig(nbits=2, group_size=32),
    "mlp.up_proj": BaseQuantizeConfig(nbits=2, group_size=32),
    "mlp.down_proj": BaseQuantizeConfig(nbits=4, group_size=64),
}

替代 nn.Linear 的核心量化层：

from hqq.core.quantize import HQQLinear
import torch

# 创建量化层
linear = torch.nn.Linear(4096, 4096)
hqq_layer = HQQLinear(linear, config)

# 访问量化权重
W_q = hqq_layer.W_q           # 量化权重
scale = hqq_layer.scale       # 缩放因子
zero = hqq_layer.zero         # 零点

# 反量化以供检查
W_dequant = hqq_layer.dequantize()

HQQ 支持多种推理后端以适应不同硬件：

from hqq.core.quantize import HQQLinear

# 可用后端
backends = [
    "pytorch",          # 纯 PyTorch (默认)
    "pytorch_compile",  # torch.compile 优化
    "aten",            # 自定义 CUDA 内核
    "torchao_int4",    # TorchAO int4 矩阵乘法
    "gemlite",         # GemLite CUDA 内核
    "bitblas",         # BitBlas 优化
    "marlin",          # Marlin 4 比特内核
]

# 全局设置后端
HQQLinear.set_backend("torchao_int4")

# 或按层设置
hqq_layer.set_backend("marlin")

后端选择指南：

后端	最佳适用场景	要求
pytorch	兼容性	任何 GPU
pytorch_compile	中等加速	torch>=2.0
aten	良好平衡	CUDA GPU
torchao_int4	4 比特推理	已安装 torchao
marlin	最大 4 比特速度	Ampere+ GPU
bitblas	灵活比特宽度	已安装 bitblas

加载预量化模型

from transformers import AutoModelForCausalLM, AutoTokenizer

# 从 Hub 加载 HQQ 量化模型
model = AutoModelForCausalLM.from_pretrained(
    "mobiuslabsgmbh/Llama-3.1-8B-HQQ-4bit",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

# 正常使用
inputs = tokenizer("Hello, world!", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)

from transformers import AutoModelForCausalLM, HqqConfig

# 量化
config = HqqConfig(nbits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=config,
    device_map="auto"
)

# 保存量化模型
model.save_pretrained("./llama-8b-hqq-4bit")

# 推送到 Hub
model.push_to_hub("my-org/Llama-3.1-8B-HQQ-4bit")

from transformers import AutoModelForCausalLM, HqqConfig

# 不同层类型使用不同精度
config = HqqConfig(
    nbits=4,
    group_size=64,
    # 注意力层：更高精度
    # MLP 层：更低精度以节省内存
    dynamic_config={
        "attn": {"nbits": 4, "group_size": 64},
        "mlp": {"nbits": 2, "group_size": 32}
    }
)

使用 vLLM 服务 HQQ 模型

from vllm import LLM, SamplingParams

# 加载 HQQ 量化模型
llm = LLM(
    model="mobiuslabsgmbh/Llama-3.1-8B-HQQ-4bit",
    quantization="hqq",
    dtype="float16"
)

# 生成
sampling_params = SamplingParams(temperature=0.7, max_tokens=100)
outputs = llm.generate(["What is machine learning?"], sampling_params)

使用自定义 HQQ 配置的 vLLM

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-8B",
    quantization="hqq",
    quantization_config={
        "nbits": 4,
        "group_size": 64
    }
)

from transformers import AutoModelForCausalLM, HqqConfig
from peft import LoraConfig, get_peft_model

# 加载量化模型
quant_config = HqqConfig(nbits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=quant_config,
    device_map="auto"
)

# 应用 LoRA
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

# 使用 Trainer 或自定义循环正常训练

from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./hqq-lora-output",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    num_train_epochs=3,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch"
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    data_collator=data_collator
)

trainer.train()

工作流程 1：快速模型压缩

from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig

# 1. 配置量化
config = HqqConfig(nbits=4, group_size=64)

# 2. 加载并量化（无需校准！）
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=config,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

# 3. 验证质量
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0]))

# 4. 保存
model.save_pretrained("./llama-8b-hqq")
tokenizer.save_pretrained("./llama-8b-hqq")

工作流程 2：优化推理速度

from hqq.core.quantize import HQQLinear
from transformers import AutoModelForCausalLM, HqqConfig

# 1. 使用最优后端进行量化
config = HqqConfig(nbits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=config,
    device_map="auto"
)

# 2. 设置快速后端
HQQLinear.set_backend("marlin")  # 或 "torchao_int4"

# 3. 编译以获得额外加速
import torch
model = torch.compile(model)

# 4. 基准测试
import time
inputs = tokenizer("Hello", return_tensors="pt").to(model.device)
start = time.time()
for _ in range(10):
    model.generate(**inputs, max_new_tokens=100)
print(f"Avg time: {(time.time() - start) / 10:.2f}s")

从 4 比特开始：对于大多数模型，这是质量与大小的最佳权衡
使用 group_size=64：良好的平衡；极端量化时使用更小的值
明智选择后端：4 比特 Ampere+ 用 Marlin，灵活性用 TorchAO
验证质量：量化后务必测试生成质量
混合精度：保持注意力层更高精度，更多地压缩 MLP 层
PEFT 训练：使用 LoRA r=16-32 以获得良好的微调结果

量化期间内存不足：

# 逐层量化
from hqq.models.hf.base import AutoHQQHFModel

model = AutoHQQHFModel.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=config,
    device_map="sequential"  # 顺序加载层
)

推理速度慢：

# 切换到优化后端
from hqq.core.quantize import HQQLinear
HQQLinear.set_backend("marlin")  # 需要 Ampere+ GPU

# 或编译
model = torch.compile(model, mode="reduce-overhead")

2 比特时质量差：

# 使用更小的分组大小
config = BaseQuantizeConfig(
    nbits=2,
    group_size=16,  # 更小的分组有助于低比特量化
    axis=1
)

高级用法 - 自定义后端、混合精度、优化
故障排除 - 常见问题、调试、基准测试

代码仓库 : https://github.com/mobiusml/hqq
论文 : Half-Quadratic Quantization
HuggingFace 模型 : https://huggingface.co/mobiuslabsgmbh
版本 : 0.2.0+
许可证 : Apache 2.0

2026 年 1 月 21 日

🇺🇸English

HQQ - Half-Quadratic Quantization

Fast, calibration-free weight quantization supporting 8/4/3/2/1-bit precision with multiple optimized backends.

When to use HQQ

Use HQQ when:

Quantizing models without calibration data (no dataset needed)
Need fast quantization (minutes vs hours for GPTQ/AWQ)
Deploying with vLLM or HuggingFace Transformers
Fine-tuning quantized models with LoRA/PEFT
Experimenting with extreme quantization (2-bit, 1-bit)

Key advantages:

No calibration : Quantize any model instantly without sample data
Multiple backends : PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference
Flexible precision : 8/4/3/2/1-bit with configurable group sizes
Framework integration : Native HuggingFace and vLLM support
PEFT compatible : Fine-tune quantized models with LoRA

Use alternatives instead:

AWQ : Need calibration-based accuracy, production serving
GPTQ : Maximum accuracy with calibration data available
bitsandbytes : Simple 8-bit/4-bit without custom backends
llama.cpp/GGUF : CPU inference, Apple Silicon deployment

Quick start

Installation

pip install hqq

# With specific backend
pip install hqq[torch]      # PyTorch backend
pip install hqq[torchao]    # TorchAO int4 backend
pip install hqq[bitblas]    # BitBlas backend
pip install hqq[marlin]     # Marlin backend

Basic quantization

from hqq.core.quantize import BaseQuantizeConfig, HQQLinear
import torch.nn as nn

# Configure quantization
config = BaseQuantizeConfig(
    nbits=4,           # 4-bit quantization
    group_size=64,     # Group size for quantization
    axis=1             # Quantize along output dimension
)

# Quantize a linear layer
linear = nn.Linear(4096, 4096)
hqq_linear = HQQLinear(linear, config)

# Use normally
output = hqq_linear(input_tensor)

Quantize full model with HuggingFace

from transformers import AutoModelForCausalLM, HqqConfig

# Configure HQQ
quantization_config = HqqConfig(
    nbits=4,
    group_size=64,
    axis=1
)

# Load and quantize
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=quantization_config,
    device_map="auto"
)

# Model is quantized and ready to use

Core concepts

Quantization configuration

HQQ uses BaseQuantizeConfig to define quantization parameters:

from hqq.core.quantize import BaseQuantizeConfig

# Standard 4-bit config
config_4bit = BaseQuantizeConfig(
    nbits=4,           # Bits per weight (1-8)
    group_size=64,     # Weights per quantization group
    axis=1             # 0=input dim, 1=output dim
)

# Aggressive 2-bit config
config_2bit = BaseQuantizeConfig(
    nbits=2,
    group_size=16,     # Smaller groups for low-bit
    axis=1
)

# Mixed precision per layer type
layer_configs = {
    "self_attn.q_proj": BaseQuantizeConfig(nbits=4, group_size=64),
    "self_attn.k_proj": BaseQuantizeConfig(nbits=4, group_size=64),
    "self_attn.v_proj": BaseQuantizeConfig(nbits=4, group_size=64),
    "mlp.gate_proj": BaseQuantizeConfig(nbits=2, group_size=32),
    "mlp.up_proj": BaseQuantizeConfig(nbits=2, group_size=32),
    "mlp.down_proj": BaseQuantizeConfig(nbits=4, group_size=64),
}

HQQLinear layer

The core quantized layer that replaces nn.Linear:

from hqq.core.quantize import HQQLinear
import torch

# Create quantized layer
linear = torch.nn.Linear(4096, 4096)
hqq_layer = HQQLinear(linear, config)

# Access quantized weights
W_q = hqq_layer.W_q           # Quantized weights
scale = hqq_layer.scale       # Scale factors
zero = hqq_layer.zero         # Zero points

# Dequantize for inspection
W_dequant = hqq_layer.dequantize()

Backends

HQQ supports multiple inference backends for different hardware:

from hqq.core.quantize import HQQLinear

# Available backends
backends = [
    "pytorch",          # Pure PyTorch (default)
    "pytorch_compile",  # torch.compile optimized
    "aten",            # Custom CUDA kernels
    "torchao_int4",    # TorchAO int4 matmul
    "gemlite",         # GemLite CUDA kernels
    "bitblas",         # BitBlas optimized
    "marlin",          # Marlin 4-bit kernels
]

# Set backend globally
HQQLinear.set_backend("torchao_int4")

# Or per layer
hqq_layer.set_backend("marlin")

Backend selection guide:

Backend	Best For	Requirements
pytorch	Compatibility	Any GPU
pytorch_compile	Moderate speedup	torch>=2.0
aten	Good balance	CUDA GPU
torchao_int4	4-bit inference	torchao installed
marlin	Maximum 4-bit speed	Ampere+ GPU
bitblas	Flexible bit-widths	bitblas installed

HuggingFace integration

Load pre-quantized models

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load HQQ-quantized model from Hub
model = AutoModelForCausalLM.from_pretrained(
    "mobiuslabsgmbh/Llama-3.1-8B-HQQ-4bit",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

# Use normally
inputs = tokenizer("Hello, world!", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)

Quantize and save

from transformers import AutoModelForCausalLM, HqqConfig

# Quantize
config = HqqConfig(nbits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=config,
    device_map="auto"
)

# Save quantized model
model.save_pretrained("./llama-8b-hqq-4bit")

# Push to Hub
model.push_to_hub("my-org/Llama-3.1-8B-HQQ-4bit")

Mixed precision quantization

from transformers import AutoModelForCausalLM, HqqConfig

# Different precision per layer type
config = HqqConfig(
    nbits=4,
    group_size=64,
    # Attention layers: higher precision
    # MLP layers: lower precision for memory savings
    dynamic_config={
        "attn": {"nbits": 4, "group_size": 64},
        "mlp": {"nbits": 2, "group_size": 32}
    }
)

vLLM integration

Serve HQQ models with vLLM

from vllm import LLM, SamplingParams

# Load HQQ-quantized model
llm = LLM(
    model="mobiuslabsgmbh/Llama-3.1-8B-HQQ-4bit",
    quantization="hqq",
    dtype="float16"
)

# Generate
sampling_params = SamplingParams(temperature=0.7, max_tokens=100)
outputs = llm.generate(["What is machine learning?"], sampling_params)

vLLM with custom HQQ config

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-8B",
    quantization="hqq",
    quantization_config={
        "nbits": 4,
        "group_size": 64
    }
)

PEFT/LoRA fine-tuning

Fine-tune quantized models

from transformers import AutoModelForCausalLM, HqqConfig
from peft import LoraConfig, get_peft_model

# Load quantized model
quant_config = HqqConfig(nbits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=quant_config,
    device_map="auto"
)

# Apply LoRA
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

# Train normally with Trainer or custom loop

QLoRA-style training

from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./hqq-lora-output",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    num_train_epochs=3,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch"
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    data_collator=data_collator
)

trainer.train()

Quantization workflows

Workflow 1: Quick model compression

from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig

# 1. Configure quantization
config = HqqConfig(nbits=4, group_size=64)

# 2. Load and quantize (no calibration needed!)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=config,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

# 3. Verify quality
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0]))

# 4. Save
model.save_pretrained("./llama-8b-hqq")
tokenizer.save_pretrained("./llama-8b-hqq")

Workflow 2: Optimize for inference speed

from hqq.core.quantize import HQQLinear
from transformers import AutoModelForCausalLM, HqqConfig

# 1. Quantize with optimal backend
config = HqqConfig(nbits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=config,
    device_map="auto"
)

# 2. Set fast backend
HQQLinear.set_backend("marlin")  # or "torchao_int4"

# 3. Compile for additional speedup
import torch
model = torch.compile(model)

# 4. Benchmark
import time
inputs = tokenizer("Hello", return_tensors="pt").to(model.device)
start = time.time()
for _ in range(10):
    model.generate(**inputs, max_new_tokens=100)
print(f"Avg time: {(time.time() - start) / 10:.2f}s")

Best practices

Start with 4-bit : Best quality/size tradeoff for most models
Use group_size=64 : Good balance; smaller for extreme quantization
Choose backend wisely : Marlin for 4-bit Ampere+, TorchAO for flexibility
Verify quality : Always test generation quality after quantization
Mixed precision : Keep attention at higher precision, compress MLP more
PEFT training : Use LoRA r=16-32 for good fine-tuning results

Common issues

Out of memory during quantization:

# Quantize layer-by-layer
from hqq.models.hf.base import AutoHQQHFModel

model = AutoHQQHFModel.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=config,
    device_map="sequential"  # Load layers sequentially
)

Slow inference:

# Switch to optimized backend
from hqq.core.quantize import HQQLinear
HQQLinear.set_backend("marlin")  # Requires Ampere+ GPU

# Or compile
model = torch.compile(model, mode="reduce-overhead")

Poor quality at 2-bit:

# Use smaller group size
config = BaseQuantizeConfig(
    nbits=2,
    group_size=16,  # Smaller groups help at low bits
    axis=1
)

References

Advanced Usage - Custom backends, mixed precision, optimization
Troubleshooting - Common issues, debugging, benchmarks

Resources

Repository : https://github.com/mobiusml/hqq
Paper : Half-Quadratic Quantization
HuggingFace Models : https://huggingface.co/mobiuslabsgmbh
Version : 0.2.0+
License : Apache 2.0

Weekly Installs

151

Repository

davila7/claude-…emplates

GitHub Stars

23.4K

First Seen

Jan 21, 2026

Security Audits

Gen Agent Trust HubPass SocketPass SnykWarn

Installed on

claude-code127

opencode123

gemini-cli117

cursor116

antigravity105

codex104

AI 代码实施计划编写技能 | 自动化开发任务分解与 TDD 流程规划工具

48,300 周安装

HQQ 半二次量化：无需校准的快速模型量化，支持 8/4/3/2/1 比特精度

🇨🇳中文介绍

HQQ - 半二次量化

何时使用 HQQ

快速开始

安装

相关 Skills

基础量化

使用 HuggingFace 量化完整模型

核心概念

量化配置

HQQLinear 层

后端

HuggingFace 集成