Agent skill · software engineering · davila7
training-llms-megatron
Trains large language models (2B-462B parameters) using NVIDIA Megatron-Core with advanced parallelism strategies. Use when training models >1B parameters, need maximum GPU efficiency (47% MFU on H100), or require tensor/pipeline/sequence/context/expert parallelism. Production-ready framework used for Nemotron, LLaMA, DeepSeek.
Why this skill is useful
Adds executable scripts and detailed workflows for training large language models with advanced parallelism techniques that are not commonly found in public documentation.
What it needs
Requires apex, docker, megatron-core, torch, transformer-engine installed locally. About 5k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
Megatron-Core - Large-Scale LLM Training Quick start Megatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies. Installation: Simple distributed training: Common workflows Workflow 1: Train LLaMA-style model with 3D parallelism Copy this checklist: Step 1: Choose parallelism configuration Model size determines parallelism strategy: Model Size GPUs Tensor Parallel Pipeline Parallel Data Parallel Context Parallel ------------ ------ ----------------- ------------------- --------------- ------------------ 7B 8 1 1 8 1 13B 8 2 1 4 1 70B 64 4 4 4 1 405B 128 8 8 2 2 Step 2: Configure training hyperparameters Step 3: Launch distributed training Step 4: Monitor performance metrics Key metrics to track: Workflow 2: Configure Mixture of Experts (MoE) training For sparse MoE models like Mixtral. Step 1: Configure expert parallelism Step 2: Set MoE hyperparameters Step 3: Launch training with EP Expert parallelism distributes different experts across GPUs, reducing memory while maintaining capacity. Workflow 3: Optimize for maximum throughput Achieve 47% MFU on H100. Step 1: Enable optimizations Step 2: Use FP8 precision (H100 only) Result: 1.5-2x speedup on H100 vs BF16. Step 3: Optimize micro-batch size Find largest micro-batch that fits in memory: Typical values: 7B model: 4-8 70B model: 1-2 405B model: 1 Step 4: Tune parallelism degrees Rules of thumb: Example 405B on 128 H100s: When to use vs alternatives Use Megatron-Core when: Training models >10B parameters Need maximum efficiency (target >40% MFU) Using NVIDIA GPUs (A100, H100) Production training at scale Want fine-grained parallelism control Use alternatives instead: PyTorch FSDP: Models <70B, simpler API, PyTorch native DeepSpeed: Easier setup, good for <100B models HuggingFace Accelerate: Prototyping, simpler workflows LitGPT: Educational, single-file implementations Common issues Issue: Low GPU utilization (<30% MFU) Causes: 1. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/distributed-training-megatron-core