Agent skill · software engineering · davila7
distributed-llm-pretraining-torchtitan
Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.
Why this skill is useful
Adds executable scripts and configurations for distributed LLM pretraining that significantly enhance the AI's capabilities in large-scale model training.
What it needs
Requires torch, torchao, torchtitan installed locally. About 5k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
TorchTitan - PyTorch Native Distributed LLM Pretraining Quick start TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs. Installation: Download tokenizer: Start training on 8 GPUs: Common workflows Workflow 1: Pretrain Llama 3.1 8B on single node Copy this checklist: Step 1: Download tokenizer Step 2: Configure training Edit or create a TOML config file: Step 3: Launch training Step 4: Monitor and checkpoint TensorBoard logs are saved to ./outputs/tb/: Workflow 2: Multi-node training with SLURM Step 1: Configure parallelism for scale For 70B model on 256 GPUs (32 nodes): Step 2: Set up SLURM script Step 3: Submit job Step 4: Resume from checkpoint Training auto-resumes if checkpoint exists in configured folder. Workflow 3: Enable Float8 training for H100s Float8 provides 30-50% speedup on H100 GPUs. Step 1: Install torchao Step 2: Configure Float8 Add to your TOML config: Step 3: Launch with compile Workflow 4: 4D parallelism for 405B models Step 1: Create seed checkpoint Required for consistent initialization across PP stages: Step 2: Configure 4D parallelism Step 3: Launch on 512 GPUs When to use vs alternatives Use TorchTitan when: Pretraining LLMs from scratch (8B to 405B+) Need PyTorch-native solution without third-party dependencies Require composable 4D parallelism (FSDP2, TP, PP, CP) Training on H100s with Float8 support Want interoperable checkpoints with torchtune/HuggingFace Use alternatives instead: Megatron-LM: Maximum performance for NVIDIA-only deployments DeepSpeed: Broader ZeRO optimization ecosystem, inference support Axolotl/TRL: Fine-tuning rather than pretraining LitGPT: Educational, smaller-scale training Common issues Issue: Out of memory on large models Enable activation checkpointing and reduce batch size: Or use gradient accumulation: Issue: TP causes high memory with async collectives Set environment variable: Issue …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/model-architecture-torchtitan