Agent skill · software engineering · nousresearch
torchtitan
Pretrain LLMs at scale with PyTorch 4D parallelism.
Why this skill is useful
Adds executable scripts and detailed workflows for large-scale LLM pretraining using PyTorch and TorchTitan, which are not commonly found in public documentation.
What it needs
Requires torch, torchao, torchtitan installed locally. Requires huggingface account access. About 5k tokens when loaded. Last updated 2026-08-07. 226,679 stars on the source repository.
What this skill does
TorchTitan - PyTorch Native Distributed LLM Pretraining Quick start TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs. Installation: Download tokenizer: Start training on 8 GPUs: Common workflows Workflow 1: Pretrain Llama 3.1 8B on single node Copy this checklist: Step 1: Download tokenizer Step 2: Configure training In torchtitan's current layout, run configs are defined in a Python config registry (torchtitan/models/llama3/configregistry.py) and selected by name via CONFIG=<name> (or --config <name>). To customize, register your own config in the registry, or override individual fields on the command line (e.g. --optimizer.lr 3e-4 --training.steps 1000). The equivalent settings for an 8B run look like this (shown as fields; set them in the registry entry or as --section.key value overrides): Step 3: Launch training Step 4: Monitor and checkpoint TensorBoard logs are saved to ./outputs/tb/: Workflow 2: Multi-node training with SLURM Step 1: Configure parallelism for scale For 70B model on 256 GPUs (32 nodes): Step 2: Set up SLURM script Step 3: Submit job Step 4: Resume from checkpoint Training auto-resumes if checkpoint exists in configured folder. Workflow 3: Enable Float8 training for H100s Float8 provides 30-50% speedup on H100 GPUs. Step 1: Install torchao Step 2: Configure Float8 In the current torchtitan, Float8 is applied at config time via the quantization parameter in your modelregistry() call inside the config registry (not via a [quantize.linear.float8] TOML section). …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nousresearch/torchtitan