Agent skill · vasilyu1983
ai-data-curation-pretraining
Builds and audits LLM pretraining corpora: extraction, filtering, dedup, decontamination, data mixing, synthetic data. Use when curating or ablating a pretraining data pipeline.
What it needs
About 11k tokens when loaded.
What this skill does
Pretraining Data Curation — Functional Reference Skill Domain: Building web-scale and synthetic pretraining corpora, running controlled data ablations. Distinct from applications-layer retrieval (RAG) and general data engineering. No theory. No generic pipeline intros. Focus on stage-by-stage decisions, heuristic thresholds, tooling choices, and ablation protocol. ASCII Flow When to Use This Skill Activate when the task involves: Finding existing high-quality datasets for pretraining or fine-tuning before building from scratch (see Dataset Discovery reference) Sourcing and filtering CommonCrawl WARCs or other web-scale corpora Implementing or debugging any stage of the curation pipeline above Designing quality filters (heuristic or classifier-based) Running MinHash / LSH deduplication or exact-substring dedup Decontaminating a dataset against evaluation benchmarks Generating synthetic pretraining data (Cosmopedia, Self-Instruct, Evol-Instruct, Nemotron) Designing and executing controlled data ablations Writing datasheets (Gebru et al.) for a curated dataset Understanding open recipe datasets: FineWeb, Dolma, The Pile, RedPajama, SlimPajama, C4, RefinedWeb, OLMo Scope Boundaries This skill covers the corpus side of pretraining — from raw crawl to tokenized shards and ablation measurement. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills vasilyu1983/ai-data-curation-pretraining