Agent skill · data analytics · davila7
nemo-curator
GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.
Why this skill is useful
Adds executable scripts for GPU-accelerated data curation and deduplication processes that enhance LLM training data preparation.
What it needs
Requires cudf, dask, nemo-curator, rapids installed locally. About 4k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
NeMo Curator - GPU-Accelerated Data Curation NVIDIA's toolkit for preparing high-quality training data for LLMs. When to use NeMo Curator Use NeMo Curator when: Preparing LLM training data from web scrapes (Common Crawl) Need fast deduplication (16× faster than CPU) Curating multi-modal datasets (text, images, video, audio) Filtering low-quality or toxic content Scaling data processing across GPU cluster Performance: 16× faster fuzzy deduplication (8TB RedPajama v2) 40% lower TCO vs CPU alternatives Near-linear scaling across GPU nodes Use alternatives instead: datatrove: CPU-based, open-source data processing dolma: Allen AI's data toolkit Ray Data: General ML data processing (no curation focus) Quick start Installation Basic text curation pipeline Data curation pipeline Stage 1: Quality filtering Stage 2: Deduplication Exact deduplication: Fuzzy deduplication (16× faster on GPU): Semantic deduplication: Stage 3: PII redaction Stage 4: Classifier filtering GPU acceleration GPU vs CPU performance Operation CPU (16 cores) GPU (A100) Speedup ----------- ---------------- ------------ --------- Fuzzy dedup (8TB) 120 hours 7.5 hours 16× Exact dedup (1TB) 8 hours 0.5 hours 16× Quality filtering 2 hours 0.2 hours 10× Multi-GPU scaling Multi-modal curation Image curation Video curation Audio curation Common patterns Web scrape curation (Common Crawl) Distributed processing Performance benchmarks Fuzzy deduplication (8TB RedPajama v2) CPU (256 cores): 120 hours GPU (8× A100): 7.5 hours Speedup: 16× Exact deduplication (1TB) CPU (64 cores): 8 hours GPU (4× A100): 0.5 hours Speedup: 16× Quality filtering (100GB) CPU (32 cores): 2 hours GPU (2× A100): 0.2 hours Speedup: 10× Cost comparison CPU-based curation (AWS c5.18xlarge × 10): Cost: $3.60/hour × 10 = $36/hour Time for 8TB: 120 hours Total: $4,320 GPU-based curation (AWS p4d.24xlarge × 2): Cost: $32.77/hour × 2 = $65.54/hour Time for 8TB: 7.5 hours Total: $491.55 Savings: 89% reduction ($3,828 saved) Supported data formats …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/data-processing-nemo-curator