NeMo Curator - NVIDIA GPU加速的大语言模型数据整理工具包，16倍去重速度

nemo-curator by davila7/claude-code-templates

183 周安装量

23,400 GitHub Stars

GitHub

安装命令

npx skills add https://github.com/davila7/claude-code-templates --skill nemo-curator

AI/机器学习数据处理自然语言处理

🇨🇳中文介绍

NeMo Curator - GPU 加速的数据整理工具

NVIDIA 用于为大语言模型准备高质量训练数据的工具包。

何时使用 NeMo Curator

在以下情况下使用 NeMo Curator：

从网络抓取数据（如 Common Crawl）准备 LLM 训练数据时
需要快速去重（比 CPU 快 16 倍）
整理多模态数据集（文本、图像、视频、音频）
过滤低质量或有毒内容
跨 GPU 集群扩展数据处理能力

性能表现：

模糊去重速度快 16 倍（处理 8TB RedPajama v2 数据集）
与 CPU 方案相比，总拥有成本降低 40%
跨 GPU 节点实现接近线性的扩展

替代方案：

datatrove：基于 CPU 的开源数据处理工具
dolma：Allen AI 的数据工具包
Ray Data：通用的机器学习数据处理工具（不专注于数据整理）

快速开始

安装

# 文本整理（CUDA 12）
uv pip install "nemo-curator[text_cuda12]"

# 所有模态
uv pip install "nemo-curator[all_cuda12]"

# 仅 CPU（速度较慢）
uv pip install "nemo-curator[cpu]"

基础文本整理流程

from nemo_curator import ScoreFilter, Modify
from nemo_curator.datasets import DocumentDataset
import pandas as pd

# 加载数据
df = pd.DataFrame({"text": ["Good document", "Bad doc", "Excellent text"]})
dataset = DocumentDataset(df)

# 质量过滤
def quality_score(doc):
    return len(doc["text"].split()) > 5  # 过滤短文档

filtered = ScoreFilter(quality_score)(dataset)

# 去重
from nemo_curator.modules import ExactDuplicates
deduped = ExactDuplicates()(filtered)

# 保存
deduped.to_parquet("curated_data/")

广告位招租

在这里展示您的产品或服务

触达数万 AI 开发者，精准高效

联系我们

阶段 1：质量过滤

from nemo_curator.filters import (
    WordCountFilter,
    RepeatedLinesFilter,
    UrlRatioFilter,
    NonAlphaNumericFilter
)

# 应用 30 多种启发式过滤器
from nemo_curator import ScoreFilter

# 词数过滤器
dataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))

# 移除重复内容
dataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))

# URL 比例过滤器
dataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))

from nemo_curator.modules import ExactDuplicates

# 移除完全重复项
deduped = ExactDuplicates(id_field="id", text_field="text")(dataset)

模糊去重（在 GPU 上快 16 倍）：

from nemo_curator.modules import FuzzyDuplicates

# MinHash + LSH 去重
fuzzy_dedup = FuzzyDuplicates(
    id_field="id",
    text_field="text",
    num_hashes=260,      # MinHash 参数
    num_buckets=20,
    hash_method="md5"
)

deduped = fuzzy_dedup(dataset)

from nemo_curator.modules import SemanticDuplicates

# 基于嵌入的去重
semantic_dedup = SemanticDuplicates(
    id_field="id",
    text_field="text",
    embedding_model="sentence-transformers/all-MiniLM-L6-v2",
    threshold=0.8  # 余弦相似度阈值
)

deduped = semantic_dedup(dataset)

阶段 3：PII 信息脱敏

from nemo_curator.modules import Modify
from nemo_curator.modifiers import PIIRedactor

# 脱敏个人可识别信息
pii_redactor = PIIRedactor(
    supported_entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "PERSON", "LOCATION"],
    anonymize_action="replace"  # 或 "redact"
)

redacted = Modify(pii_redactor)(dataset)

阶段 4：分类器过滤

from nemo_curator.classifiers import QualityClassifier

# 质量分类
quality_clf = QualityClassifier(
    model_path="nvidia/quality-classifier-deberta",
    batch_size=256,
    device="cuda"
)

# 过滤低质量文档
high_quality = dataset.filter(lambda doc: quality_clf(doc["text"]) > 0.5)

GPU 与 CPU 性能对比

操作	CPU (16 核)	GPU (A100)	加速比
模糊去重 (8TB)	120 小时	7.5 小时	16×
精确去重 (1TB)	8 小时	0.5 小时	16×
质量过滤	2 小时	0.2 小时	10×

from nemo_curator import get_client
import dask_cuda

# 初始化 GPU 集群
client = get_client(cluster_type="gpu", n_workers=8)

# 使用 8 个 GPU 处理
deduped = FuzzyDuplicates(...)(dataset)

多模态数据整理

from nemo_curator.image import (
    AestheticFilter,
    NSFWFilter,
    CLIPEmbedder
)

# 美学评分
aesthetic_filter = AestheticFilter(threshold=5.0)
filtered_images = aesthetic_filter(image_dataset)

# NSFW 检测
nsfw_filter = NSFWFilter(threshold=0.9)
safe_images = nsfw_filter(filtered_images)

# 生成 CLIP 嵌入
clip_embedder = CLIPEmbedder(model="openai/clip-vit-base-patch32")
image_embeddings = clip_embedder(safe_images)

from nemo_curator.video import (
    SceneDetector,
    ClipExtractor,
    InternVideo2Embedder
)

# 检测场景
scene_detector = SceneDetector(threshold=27.0)
scenes = scene_detector(video_dataset)

# 提取片段
clip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)
clips = clip_extractor(scenes)

# 生成嵌入
video_embedder = InternVideo2Embedder()
video_embeddings = video_embedder(clips)

from nemo_curator.audio import (
    ASRInference,
    WERFilter,
    DurationFilter
)

# ASR 转录
asr = ASRInference(model="nvidia/stt_en_fastconformer_hybrid_large_pc")
transcribed = asr(audio_dataset)

# 根据 WER（词错误率）过滤
wer_filter = WERFilter(max_wer=0.3)
high_quality_audio = wer_filter(transcribed)

# 时长过滤
duration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)
filtered_audio = duration_filter(high_quality_audio)

网络抓取数据整理（Common Crawl）

from nemo_curator import ScoreFilter, Modify
from nemo_curator.filters import *
from nemo_curator.modules import *
from nemo_curator.datasets import DocumentDataset

# 加载 Common Crawl 数据
dataset = DocumentDataset.read_parquet("common_crawl/*.parquet")

# 流程
pipeline = [
    # 1. 质量过滤
    WordCountFilter(min_words=100, max_words=50000),
    RepeatedLinesFilter(max_repeated_line_fraction=0.2),
    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),
    UrlRatioFilter(max_url_ratio=0.3),

    # 2. 语言过滤
    LanguageIdentificationFilter(target_languages=["en"]),

    # 3. 去重
    ExactDuplicates(id_field="id", text_field="text"),
    FuzzyDuplicates(id_field="id", text_field="text", num_hashes=260),

    # 4. PII 脱敏
    PIIRedactor(),

    # 5. NSFW 过滤
    NSFWClassifier(threshold=0.8)
]

# 执行
for stage in pipeline:
    dataset = stage(dataset)

# 保存
dataset.to_parquet("curated_common_crawl/")

from nemo_curator import get_client
from dask_cuda import LocalCUDACluster

# 多 GPU 集群
cluster = LocalCUDACluster(n_workers=8)
client = get_client(cluster=cluster)

# 处理大型数据集
dataset = DocumentDataset.read_parquet("s3://large_dataset/*.parquet")
deduped = FuzzyDuplicates(...)(dataset)

# 清理
client.close()
cluster.close()

模糊去重（8TB RedPajama v2）

CPU (256 核)：120 小时
GPU (8× A100)：7.5 小时
加速比：16×

精确去重（1TB）

CPU (64 核)：8 小时
GPU (4× A100)：0.5 小时
加速比：16×

质量过滤（100GB）

CPU (32 核)：2 小时
GPU (2× A100)：0.2 小时
加速比：10×

基于 CPU 的整理（AWS c5.18xlarge × 10）：

成本：$3.60/小时 × 10 = $36/小时
处理 8TB 所需时间：120 小时
总计：$4,320

基于 GPU 的整理（AWS p4d.24xlarge × 2）：

成本：$32.77/小时 × 2 = $65.54/小时
处理 8TB 所需时间：7.5 小时
总计：$491.55

节省：成本降低 89%（节省 $3,828）

支持的数据格式

输入：Parquet、JSONL、CSV
输出：Parquet（推荐）、JSONL
WebDataset：用于多模态数据的 TAR 归档文件

NVIDIA 使用 NeMo Curator 准备 Nemotron-4 训练数据
整理的开源数据集：RedPajama v2、The Pile

过滤指南 - 30 多种质量过滤器和启发式方法
去重指南 - 精确、模糊、语义去重方法

GitHub：https://github.com/NVIDIA/NeMo-Curator ⭐ 500+
文档：https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/
版本：0.4.0+
许可证：Apache 2.0

2026 年 1 月 21 日

🇺🇸English

NeMo Curator - GPU-Accelerated Data Curation

NVIDIA's toolkit for preparing high-quality training data for LLMs.

When to use NeMo Curator

Use NeMo Curator when:

Preparing LLM training data from web scrapes (Common Crawl)
Need fast deduplication (16× faster than CPU)
Curating multi-modal datasets (text, images, video, audio)
Filtering low-quality or toxic content
Scaling data processing across GPU cluster

Performance :

16× faster fuzzy deduplication (8TB RedPajama v2)
40% lower TCO vs CPU alternatives
Near-linear scaling across GPU nodes

Use alternatives instead :

datatrove : CPU-based, open-source data processing
dolma : Allen AI's data toolkit
Ray Data : General ML data processing (no curation focus)

Quick start

Installation

# Text curation (CUDA 12)
uv pip install "nemo-curator[text_cuda12]"

# All modalities
uv pip install "nemo-curator[all_cuda12]"

# CPU-only (slower)
uv pip install "nemo-curator[cpu]"

Basic text curation pipeline

from nemo_curator import ScoreFilter, Modify
from nemo_curator.datasets import DocumentDataset
import pandas as pd

# Load data
df = pd.DataFrame({"text": ["Good document", "Bad doc", "Excellent text"]})
dataset = DocumentDataset(df)

# Quality filtering
def quality_score(doc):
    return len(doc["text"].split()) > 5  # Filter short docs

filtered = ScoreFilter(quality_score)(dataset)

# Deduplication
from nemo_curator.modules import ExactDuplicates
deduped = ExactDuplicates()(filtered)

# Save
deduped.to_parquet("curated_data/")

Data curation pipeline

Stage 1: Quality filtering

from nemo_curator.filters import (
    WordCountFilter,
    RepeatedLinesFilter,
    UrlRatioFilter,
    NonAlphaNumericFilter
)

# Apply 30+ heuristic filters
from nemo_curator import ScoreFilter

# Word count filter
dataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))

# Remove repetitive content
dataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))

# URL ratio filter
dataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))

Stage 2: Deduplication

Exact deduplication :

from nemo_curator.modules import ExactDuplicates

# Remove exact duplicates
deduped = ExactDuplicates(id_field="id", text_field="text")(dataset)

Fuzzy deduplication (16× faster on GPU):

from nemo_curator.modules import FuzzyDuplicates

# MinHash + LSH deduplication
fuzzy_dedup = FuzzyDuplicates(
    id_field="id",
    text_field="text",
    num_hashes=260,      # MinHash parameters
    num_buckets=20,
    hash_method="md5"
)

deduped = fuzzy_dedup(dataset)

Semantic deduplication :

from nemo_curator.modules import SemanticDuplicates

# Embedding-based deduplication
semantic_dedup = SemanticDuplicates(
    id_field="id",
    text_field="text",
    embedding_model="sentence-transformers/all-MiniLM-L6-v2",
    threshold=0.8  # Cosine similarity threshold
)

deduped = semantic_dedup(dataset)

Stage 3: PII redaction

from nemo_curator.modules import Modify
from nemo_curator.modifiers import PIIRedactor

# Redact personally identifiable information
pii_redactor = PIIRedactor(
    supported_entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "PERSON", "LOCATION"],
    anonymize_action="replace"  # or "redact"
)

redacted = Modify(pii_redactor)(dataset)

Stage 4: Classifier filtering

from nemo_curator.classifiers import QualityClassifier

# Quality classification
quality_clf = QualityClassifier(
    model_path="nvidia/quality-classifier-deberta",
    batch_size=256,
    device="cuda"
)

# Filter low-quality documents
high_quality = dataset.filter(lambda doc: quality_clf(doc["text"]) > 0.5)

GPU acceleration

GPU vs CPU performance

Operation	CPU (16 cores)	GPU (A100)	Speedup
Fuzzy dedup (8TB)	120 hours	7.5 hours	16×
Exact dedup (1TB)	8 hours	0.5 hours	16×
Quality filtering	2 hours	0.2 hours	10×

Multi-GPU scaling

from nemo_curator import get_client
import dask_cuda

# Initialize GPU cluster
client = get_client(cluster_type="gpu", n_workers=8)

# Process with 8 GPUs
deduped = FuzzyDuplicates(...)(dataset)

Multi-modal curation

Image curation

from nemo_curator.image import (
    AestheticFilter,
    NSFWFilter,
    CLIPEmbedder
)

# Aesthetic scoring
aesthetic_filter = AestheticFilter(threshold=5.0)
filtered_images = aesthetic_filter(image_dataset)

# NSFW detection
nsfw_filter = NSFWFilter(threshold=0.9)
safe_images = nsfw_filter(filtered_images)

# Generate CLIP embeddings
clip_embedder = CLIPEmbedder(model="openai/clip-vit-base-patch32")
image_embeddings = clip_embedder(safe_images)

Video curation

from nemo_curator.video import (
    SceneDetector,
    ClipExtractor,
    InternVideo2Embedder
)

# Detect scenes
scene_detector = SceneDetector(threshold=27.0)
scenes = scene_detector(video_dataset)

# Extract clips
clip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)
clips = clip_extractor(scenes)

# Generate embeddings
video_embedder = InternVideo2Embedder()
video_embeddings = video_embedder(clips)

Audio curation

from nemo_curator.audio import (
    ASRInference,
    WERFilter,
    DurationFilter
)

# ASR transcription
asr = ASRInference(model="nvidia/stt_en_fastconformer_hybrid_large_pc")
transcribed = asr(audio_dataset)

# Filter by WER (word error rate)
wer_filter = WERFilter(max_wer=0.3)
high_quality_audio = wer_filter(transcribed)

# Duration filtering
duration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)
filtered_audio = duration_filter(high_quality_audio)

Common patterns

Web scrape curation (Common Crawl)

from nemo_curator import ScoreFilter, Modify
from nemo_curator.filters import *
from nemo_curator.modules import *
from nemo_curator.datasets import DocumentDataset

# Load Common Crawl data
dataset = DocumentDataset.read_parquet("common_crawl/*.parquet")

# Pipeline
pipeline = [
    # 1. Quality filtering
    WordCountFilter(min_words=100, max_words=50000),
    RepeatedLinesFilter(max_repeated_line_fraction=0.2),
    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),
    UrlRatioFilter(max_url_ratio=0.3),

    # 2. Language filtering
    LanguageIdentificationFilter(target_languages=["en"]),

    # 3. Deduplication
    ExactDuplicates(id_field="id", text_field="text"),
    FuzzyDuplicates(id_field="id", text_field="text", num_hashes=260),

    # 4. PII redaction
    PIIRedactor(),

    # 5. NSFW filtering
    NSFWClassifier(threshold=0.8)
]

# Execute
for stage in pipeline:
    dataset = stage(dataset)

# Save
dataset.to_parquet("curated_common_crawl/")

Distributed processing

from nemo_curator import get_client
from dask_cuda import LocalCUDACluster

# Multi-GPU cluster
cluster = LocalCUDACluster(n_workers=8)
client = get_client(cluster=cluster)

# Process large dataset
dataset = DocumentDataset.read_parquet("s3://large_dataset/*.parquet")
deduped = FuzzyDuplicates(...)(dataset)

# Cleanup
client.close()
cluster.close()

Performance benchmarks

Fuzzy deduplication (8TB RedPajama v2)

CPU (256 cores) : 120 hours
GPU (8× A100) : 7.5 hours
Speedup : 16×

Exact deduplication (1TB)

CPU (64 cores) : 8 hours
GPU (4× A100) : 0.5 hours
Speedup : 16×

Quality filtering (100GB)

CPU (32 cores) : 2 hours
GPU (2× A100) : 0.2 hours
Speedup : 10×

Cost comparison

CPU-based curation (AWS c5.18xlarge × 10):

Cost: $3.60/hour × 10 = $36/hour
Time for 8TB: 120 hours
Total : $4,320

GPU-based curation (AWS p4d.24xlarge × 2):

Cost: $32.77/hour × 2 = $65.54/hour
Time for 8TB: 7.5 hours
Total : $491.55

Savings : 89% reduction ($3,828 saved)

Supported data formats

Input : Parquet, JSONL, CSV
Output : Parquet (recommended), JSONL
WebDataset : TAR archives for multi-modal

Use cases

Production deployments :

NVIDIA used NeMo Curator to prepare Nemotron-4 training data
Open-source datasets curated: RedPajama v2, The Pile

References

Filtering Guide - 30+ quality filters, heuristics
Deduplication Guide - Exact, fuzzy, semantic methods

Resources

GitHub : https://github.com/NVIDIA/NeMo-Curator ⭐ 500+
Docs : https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/
Version : 0.4.0+
License : Apache 2.0

Weekly Installs

183

Repository

davila7/claude-…emplates

GitHub Stars

23.4K

First Seen

Jan 21, 2026

Security Audits

Gen Agent Trust HubPass SocketPass SnykWarn

Installed on

opencode148

claude-code147

gemini-cli141

cursor133

codex128

antigravity118

超能力技能使用指南：AI助手技能调用优先级与工作流程详解

46,500 周安装