Agent skill · vasilyu1983
ai-distributed-training
Guides multi-GPU pre-training: DDP, FSDP2, ZeRO, tensor/pipeline/expert parallelism, fp8/Muon. Use when scaling a run, training MoE, or reproducing GPT-2 on rented GPUs.
What it needs
About 11k tokens when loaded.
What this skill does
Distributed Training - Systems Performance Skill Operational focus: picking and implementing the right parallelism strategy, not the theory. Covers data parallelism through FSDP/ZeRO/tensor+pipeline parallelism, memory-efficient attention, mixed precision at scale, activation checkpointing, rented-GPU cost discipline, and reproducing GPT-2 124M as the canonical sanity check. Profile before you scale. Debug on the smallest GPU that fits. Stop the instance when done. ASCII Flow When to Use This Skill Activate when the user asks about: Choosing between DDP, FSDP2, DeepSpeed ZeRO stages 1/2/3, or Megatron-LM Training Mixture-of-Experts (MoE) models: expert parallelism, all-to-all, load balancing OOM errors on multi-GPU training runs Memory-efficient attention (FlashAttention-2/3, xformers) Mixed precision (bf16, fp8, nvfp4) trade-offs at pre-training scale Optimizer choice at scale (AdamW vs Muon/MuonClip) Targeting current-gen hardware (H100, Blackwell B200/GB200 NVL72, early Rubin NVL72 access) Gradient checkpointing vs activation checkpointing cost Pre-training frameworks: litgpt, torchtitan, nanotron, levanter Reproducing GPT-2 (modded-nanoGPT or nanochat as the active reference; llm.c as the educational one) Rented GPU cost management (RunPod, Lambda, Vast.ai, Modal) Spot / interruptible instance checkpoint strategies Profiling a training run before deciding to scale Scope Boundaries (Use These Skills for Depth) Data mix, filtering, dedup, decontamination -> ai-data-curation-pretraining. Before spending on N GPUs, the data mix matters more than the parallelism — a better corpus beats a better topology at the same budget, and it is far cheaper to change. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills vasilyu1983/ai-distributed-training