TRL 强化学习微调指南：使用 SFT、DPO、PPO 对齐语言模型与人类偏好

fine-tuning-with-trl by davila7/claude-code-templates

184 周安装量

23,400 GitHub Stars

GitHub

安装命令

npx skills add https://github.com/davila7/claude-code-templates --skill fine-tuning-with-trl

AI/机器学习提示工程自然语言处理

🇨🇳中文介绍

TRL - Transformer 强化学习

快速开始

TRL 提供了用于将语言模型与人类偏好对齐的后训练方法。

安装：

pip install trl transformers datasets peft accelerate

监督式微调（指令调优）：

from trl import SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,  # 提示-完成对
)
trainer.train()

DPO（与偏好对齐）：

from trl import DPOTrainer, DPOConfig

config = DPOConfig(output_dir="model-dpo", beta=0.1)
trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=preference_dataset,  # 选中/拒绝对
    processing_class=tokenizer
)
trainer.train()

常见工作流程

工作流程 1：完整的 RLHF 流程（SFT → 奖励模型 → PPO）

从基础模型到与人类对齐的模型的完整流程。

复制此清单：

RLHF 训练：
- [ ] 步骤 1：监督式微调（SFT）
- [ ] 步骤 2：训练奖励模型
- [ ] 步骤 3：PPO 强化学习
- [ ] 步骤 4：评估对齐模型

广告位招租

在这里展示您的产品或服务

触达数万 AI 开发者，精准高效

联系我们

工作流程 2：使用 DPO 进行简单的偏好对齐

无需奖励模型，将模型与偏好对齐。

DPO 训练：
- [ ] 步骤 1：准备偏好数据集
- [ ] 步骤 2：配置 DPO
- [ ] 步骤 3：使用 DPOTrainer 训练
- [ ] 步骤 4：评估对齐效果

步骤 1：准备偏好数据集

{
  "prompt": "What is the capital of France?",
  "chosen": "The capital of France is Paris.",
  "rejected": "I don't know."
}

from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
# 或加载你自己的
# dataset = load_dataset("json", data_files="preferences.json")

步骤 2：配置 DPO

from trl import DPOConfig

config = DPOConfig(
    output_dir="Qwen2.5-0.5B-DPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=5e-7,
    beta=0.1,  # KL 惩罚强度
    max_prompt_length=512,
    max_length=1024,
    logging_steps=10
)

步骤 3：使用 DPOTrainer 训练

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    processing_class=tokenizer
)

trainer.train()
trainer.save_model()

CLI 替代方案：

trl dpo \
    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name argilla/Capybara-Preferences \
    --output_dir Qwen2.5-0.5B-DPO \
    --per_device_train_batch_size 4 \
    --learning_rate 5e-7 \
    --beta 0.1

工作流程 3：使用 GRPO 进行内存高效的在线 RL

使用最少内存进行强化学习训练。

GRPO 训练：
- [ ] 步骤 1：定义奖励函数
- [ ] 步骤 2：配置 GRPO
- [ ] 步骤 3：使用 GRPOTrainer 训练

步骤 1：定义奖励函数

def reward_function(completions, **kwargs):
    """
    计算完成文本的奖励。

    参数：
        completions: 生成的文本列表

    返回：
        奖励分数列表（浮点数）
    """
    rewards = []
    for completion in completions:
        # 示例：基于长度和独特单词的奖励
        score = len(completion.split())  # 偏好更长的回复
        score += len(set(completion.lower().split()))  # 奖励独特单词
        rewards.append(score)
    return rewards

或使用奖励模型：

from transformers import pipeline

reward_model = pipeline("text-classification", model="reward-model-path")

def reward_from_model(completions, prompts, **kwargs):
    # 组合提示 + 完成文本
    full_texts = [p + c for p, c in zip(prompts, completions)]
    # 获取奖励分数
    results = reward_model(full_texts)
    return [r["score"] for r in results]

步骤 2：配置 GRPO

from trl import GRPOConfig

config = GRPOConfig(
    output_dir="Qwen2-GRPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=1e-5,
    num_generations=4,  # 每个提示生成 4 个完成文本
    max_new_tokens=128
)

步骤 3：使用 GRPOTrainer 训练

from datasets import load_dataset
from trl import GRPOTrainer

# 加载仅包含提示的数据集
dataset = load_dataset("trl-lib/tldr", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2-0.5B-Instruct",
    reward_funcs=reward_function,  # 你的奖励函数
    args=config,
    train_dataset=dataset
)

trainer.train()

trl grpo \
    --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
    --dataset_name trl-lib/tldr \
    --output_dir Qwen2-GRPO \
    --num_generations 4

何时使用与替代方案

在以下情况下使用 TRL：

需要将模型与人类偏好对齐
拥有偏好数据（选中/拒绝对）
希望使用强化学习（PPO、GRPO）
需要奖励模型训练
进行 RLHF（完整流程）

SFT：拥有提示-完成对，希望基本的指令遵循
DPO：拥有偏好数据，希望简单的对齐（无需奖励模型）
PPO：拥有奖励模型，需要对 RL 进行最大程度的控制
GRPO：内存受限，希望进行在线 RL
奖励模型：构建 RLHF 流程，需要对生成内容进行评分

改用替代方案：

HuggingFace Trainer：无需 RL 的基础微调
Axolotl：基于 YAML 的训练配置
LitGPT：教育用途，最小化微调
Unsloth：快速的 LoRA 训练

问题：DPO 训练期间内存溢出（OOM）

减少批次大小和序列长度：

config = DPOConfig(
    per_device_train_batch_size=1,  # 从 4 减少
    max_length=512,  # 从 1024 减少
    gradient_accumulation_steps=8  # 保持有效批次大小
)

或使用梯度检查点：

model.gradient_checkpointing_enable()

问题：对齐质量差

调整 beta 参数：

# 更高的 beta = 更保守（更接近参考模型）
config = DPOConfig(beta=0.5)  # 默认 0.1

# 更低的 beta = 更激进的对齐
config = DPOConfig(beta=0.01)

问题：奖励模型未学习

检查损失类型和学习率：

config = RewardConfig(
    learning_rate=1e-5,  # 尝试不同的学习率
    num_train_epochs=3  # 训练更长时间
)

确保偏好数据集有明确的优胜者：

# 验证数据集
print(dataset[0])
# 应有明确的 chosen > rejected

问题：PPO 训练不稳定

config = PPOConfig(
    kl_coef=0.1,  # 从 0.05 增加
    cliprange=0.1  # 从 0.2 减少
)

SFT 训练指南：有关数据集格式、聊天模板、打包策略和多 GPU 训练，请参阅 references/sft-training.md。

DPO 变体：有关 IPO、cDPO、RPO 和其他 DPO 损失函数及推荐超参数，请参阅 references/dpo-variants.md。

奖励建模：有关结果奖励与过程奖励、Bradley-Terry 损失和奖励模型评估，请参阅 references/reward-modeling.md。

在线 RL 方法：有关 PPO、GRPO、RLOO 和 OnlineDPO 的详细配置，请参阅 references/online-rl.md。

GPU：NVIDIA（需要 CUDA）
VRAM：取决于模型和方法
- SFT 7B：16GB（使用 LoRA）
- DPO 7B：24GB（存储参考模型）
- PPO 7B：40GB（策略 + 奖励模型）
- GRPO 7B：24GB（内存效率更高）
多 GPU：通过 accelerate 支持
混合精度：推荐 BF16（A100/H100）

对所有方法使用 LoRA/QLoRA
启用梯度检查点
使用较小的批次大小并配合梯度累积

文档：https://huggingface.co/docs/trl/
GitHub：https://github.com/huggingface/trl
论文：
- "Training language models to follow instructions with human feedback" (InstructGPT, 2022)
- "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO, 2023)
- "Group Relative Policy Optimization" (GRPO, 2024)
示例：https://github.com/huggingface/trl/tree/main/examples/scripts

2026 年 1 月 21 日

🇺🇸English

TRL - Transformer Reinforcement Learning

Quick start

TRL provides post-training methods for aligning language models with human preferences.

Installation :

pip install trl transformers datasets peft accelerate

Supervised Fine-Tuning (instruction tuning):

from trl import SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,  # Prompt-completion pairs
)
trainer.train()

DPO (align with preferences):

from trl import DPOTrainer, DPOConfig

config = DPOConfig(output_dir="model-dpo", beta=0.1)
trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=preference_dataset,  # chosen/rejected pairs
    processing_class=tokenizer
)
trainer.train()

Common workflows

Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)

Complete pipeline from base model to human-aligned model.

Copy this checklist:

RLHF Training:
- [ ] Step 1: Supervised fine-tuning (SFT)
- [ ] Step 2: Train reward model
- [ ] Step 3: PPO reinforcement learning
- [ ] Step 4: Evaluate aligned model

Step 1: Supervised fine-tuning

Train base model on instruction-following data:

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

# Load model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

# Load instruction dataset
dataset = load_dataset("trl-lib/Capybara", split="train")

# Configure training
training_args = SFTConfig(
    output_dir="Qwen2.5-0.5B-SFT",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=2e-5,
    logging_steps=10,
    save_strategy="epoch"
)

# Train
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer
)
trainer.train()
trainer.save_model()

Step 2: Train reward model

Train model to predict human preferences:

from transformers import AutoModelForSequenceClassification
from trl import RewardTrainer, RewardConfig

# Load SFT model as base
model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen2.5-0.5B-SFT",
    num_labels=1  # Single reward score
)
tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT")

# Load preference data (chosen/rejected pairs)
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

# Configure training
training_args = RewardConfig(
    output_dir="Qwen2.5-0.5B-Reward",
    per_device_train_batch_size=2,
    num_train_epochs=1,
    learning_rate=1e-5
)

# Train reward model
trainer = RewardTrainer(
    model=model,
    args=training_args,
    processing_class=tokenizer,
    train_dataset=dataset
)
trainer.train()
trainer.save_model()

Step 3: PPO reinforcement learning

Optimize policy using reward model:

python -m trl.scripts.ppo \
    --model_name_or_path Qwen2.5-0.5B-SFT \
    --reward_model_path Qwen2.5-0.5B-Reward \
    --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \
    --output_dir Qwen2.5-0.5B-PPO \
    --learning_rate 3e-6 \
    --per_device_train_batch_size 64 \
    --total_episodes 10000

Step 4: Evaluate

from transformers import pipeline

# Load aligned model
generator = pipeline("text-generation", model="Qwen2.5-0.5B-PPO")

# Test
prompt = "Explain quantum computing to a 10-year-old"
output = generator(prompt, max_length=200)[0]["generated_text"]
print(output)

Workflow 2: Simple preference alignment with DPO

Align model with preferences without reward model.

Copy this checklist:

DPO Training:
- [ ] Step 1: Prepare preference dataset
- [ ] Step 2: Configure DPO
- [ ] Step 3: Train with DPOTrainer
- [ ] Step 4: Evaluate alignment

Step 1: Prepare preference dataset

Dataset format:

{
  "prompt": "What is the capital of France?",
  "chosen": "The capital of France is Paris.",
  "rejected": "I don't know."
}

Load dataset:

from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
# Or load your own
# dataset = load_dataset("json", data_files="preferences.json")

Step 2: Configure DPO

from trl import DPOConfig

config = DPOConfig(
    output_dir="Qwen2.5-0.5B-DPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=5e-7,
    beta=0.1,  # KL penalty strength
    max_prompt_length=512,
    max_length=1024,
    logging_steps=10
)

Step 3: Train with DPOTrainer

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    processing_class=tokenizer
)

trainer.train()
trainer.save_model()

CLI alternative :

trl dpo \
    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name argilla/Capybara-Preferences \
    --output_dir Qwen2.5-0.5B-DPO \
    --per_device_train_batch_size 4 \
    --learning_rate 5e-7 \
    --beta 0.1

Workflow 3: Memory-efficient online RL with GRPO

Train with reinforcement learning using minimal memory.

Copy this checklist:

GRPO Training:
- [ ] Step 1: Define reward function
- [ ] Step 2: Configure GRPO
- [ ] Step 3: Train with GRPOTrainer

Step 1: Define reward function

def reward_function(completions, **kwargs):
    """
    Compute rewards for completions.

    Args:
        completions: List of generated texts

    Returns:
        List of reward scores (floats)
    """
    rewards = []
    for completion in completions:
        # Example: reward based on length and unique words
        score = len(completion.split())  # Favor longer responses
        score += len(set(completion.lower().split()))  # Reward unique words
        rewards.append(score)
    return rewards

Or use a reward model:

from transformers import pipeline

reward_model = pipeline("text-classification", model="reward-model-path")

def reward_from_model(completions, prompts, **kwargs):
    # Combine prompt + completion
    full_texts = [p + c for p, c in zip(prompts, completions)]
    # Get reward scores
    results = reward_model(full_texts)
    return [r["score"] for r in results]

Step 2: Configure GRPO

from trl import GRPOConfig

config = GRPOConfig(
    output_dir="Qwen2-GRPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=1e-5,
    num_generations=4,  # Generate 4 completions per prompt
    max_new_tokens=128
)

Step 3: Train with GRPOTrainer

from datasets import load_dataset
from trl import GRPOTrainer

# Load prompt-only dataset
dataset = load_dataset("trl-lib/tldr", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2-0.5B-Instruct",
    reward_funcs=reward_function,  # Your reward function
    args=config,
    train_dataset=dataset
)

trainer.train()

CLI :

trl grpo \
    --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
    --dataset_name trl-lib/tldr \
    --output_dir Qwen2-GRPO \
    --num_generations 4

When to use vs alternatives

Use TRL when:

Need to align model with human preferences
Have preference data (chosen/rejected pairs)
Want to use reinforcement learning (PPO, GRPO)
Need reward model training
Doing RLHF (full pipeline)

Method selection :

SFT : Have prompt-completion pairs, want basic instruction following
DPO : Have preferences, want simple alignment (no reward model needed)
PPO : Have reward model, need maximum control over RL
GRPO : Memory-constrained, want online RL
Reward Model : Building RLHF pipeline, need to score generations

Use alternatives instead:

HuggingFace Trainer : Basic fine-tuning without RL
Axolotl : YAML-based training configuration
LitGPT : Educational, minimal fine-tuning
Unsloth : Fast LoRA training

Common issues

Issue: OOM during DPO training

Reduce batch size and sequence length:

config = DPOConfig(
    per_device_train_batch_size=1,  # Reduce from 4
    max_length=512,  # Reduce from 1024
    gradient_accumulation_steps=8  # Maintain effective batch
)

Or use gradient checkpointing:

model.gradient_checkpointing_enable()

Issue: Poor alignment quality

Tune beta parameter:

# Higher beta = more conservative (stays closer to reference)
config = DPOConfig(beta=0.5)  # Default 0.1

# Lower beta = more aggressive alignment
config = DPOConfig(beta=0.01)

Issue: Reward model not learning

Check loss type and learning rate:

config = RewardConfig(
    learning_rate=1e-5,  # Try different LR
    num_train_epochs=3  # Train longer
)

Ensure preference dataset has clear winners:

# Verify dataset
print(dataset[0])
# Should have clear chosen > rejected

Issue: PPO training unstable

Adjust KL coefficient:

config = PPOConfig(
    kl_coef=0.1,  # Increase from 0.05
    cliprange=0.1  # Reduce from 0.2
)

Advanced topics

SFT training guide : See references/sft-training.md for dataset formats, chat templates, packing strategies, and multi-GPU training.

DPO variants : See references/dpo-variants.md for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.

Reward modeling : See references/reward-modeling.md for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.

Online RL methods : See references/online-rl.md for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.

Hardware requirements

GPU : NVIDIA (CUDA required)
VRAM : Depends on model and method
- SFT 7B: 16GB (with LoRA)
- DPO 7B: 24GB (stores reference model)
- PPO 7B: 40GB (policy + reward model)
- GRPO 7B: 24GB (more memory efficient)
Multi-GPU : Supported via accelerate
Mixed precision : BF16 recommended (A100/H100)

Memory optimization :

Use LoRA/QLoRA for all methods
Enable gradient checkpointing
Use smaller batch sizes with gradient accumulation

Resources

Docs: https://huggingface.co/docs/trl/
GitHub: https://github.com/huggingface/trl
Papers:
- "Training language models to follow instructions with human feedback" (InstructGPT, 2022)
- "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO, 2023)
- "Group Relative Policy Optimization" (GRPO, 2024)
Examples: https://github.com/huggingface/trl/tree/main/examples/scripts

Weekly Installs

159

Repository

davila7/claude-…emplates

GitHub Stars

22.6K

First Seen

Jan 21, 2026

Security Audits

Gen Agent Trust HubPass SocketPass SnykWarn

Installed on

claude-code129

opencode127

gemini-cli117

cursor116

codex106

antigravity104

超能力技能使用指南：AI助手技能调用优先级与工作流程详解

46,500 周安装

TRL 强化学习微调指南：使用 SFT、DPO、PPO 对齐语言模型与人类偏好

🇨🇳中文介绍

TRL - Transformer 强化学习

快速开始

常见工作流程

工作流程 1：完整的 RLHF 流程（SFT → 奖励模型 → PPO）

相关 Skills

工作流程 2：使用 DPO 进行简单的偏好对齐

工作流程 3：使用 GRPO 进行内存高效的在线 RL

何时使用与替代方案

常见问题

高级主题

硬件要求

资源

🇺🇸English

TRL - Transformer Reinforcement Learning

Quick start

Common workflows

Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)

Workflow 2: Simple preference alignment with DPO

Workflow 3: Memory-efficient online RL with GRPO

When to use vs alternatives

Common issues

Advanced topics

Hardware requirements

Resources

最新 Skills