Agent skill · data analytics · nousresearch
nemo-curator
Curate LLM training data: dedupe, filter, PII redaction.
Why this skill is useful
Adds executable scripts for GPU-accelerated data curation pipelines that significantly enhance LLM training data preparation.
What it needs
Requires cudf, dask, nemo-curator, rapids installed locally. About 5k tokens when loaded. Last updated 2026-08-07. 226,679 stars on the source repository.
What this skill does
NeMo Curator - GPU-Accelerated Data Curation NVIDIA's toolkit for preparing high-quality training data for LLMs. When to use NeMo Curator Use NeMo Curator when: Preparing LLM training data from web scrapes (Common Crawl) Need fast deduplication (16× faster than CPU) Curating multi-modal datasets (text, images, video, audio) Filtering low-quality or toxic content Scaling data processing across GPU cluster Performance: 16× faster fuzzy deduplication (8TB RedPajama v2) 40% lower TCO vs CPU alternatives Near-linear scaling across GPU nodes Use alternatives instead: datatrove: CPU-based, open-source data processing dolma: Allen AI's data toolkit Ray Data: General ML data processing (no curation focus) Quick start Installation Basic text curation pipeline Major version rewrite (1.x): NeMo Curator was rewritten around a Ray-based pipeline/stage architecture. The old DocumentDataset + nemocurator.modules. / ScoreFilter / Modify call-the-object-on-a-dataset API from 0.x is gone. In 1.x you compose ProcessingStages into a Pipeline and run it with an executor. The exact stage/import surface differs per modality — treat the examples in this skill below as conceptual (0.x-style) and follow the current quickstart and text guide for the exact 1.x APIs rather than copying imports verbatim. Shape of a 1.x pipeline (from the upstream quickstart): The 0.x-style snippets in the sections that follow illustrate the concepts (quality filtering, exact/fuzzy/semantic dedup, PII redaction, classifier filtering). For runnable 1.x code, map each concept onto the corresponding stage from the modality guide. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nousresearch/nemo-curator