Agent skill · software engineering · nvidia
nemo-mbridge-mlm-bridge-training
Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data. Covers correlation testing, available recipes, and multi-GPU examples.
Why this skill is useful
Includes executable scripts for running Megatron-LM and Megatron Bridge training with specific configurations and multi-GPU setups.
What it needs
Requires torch installed locally. About 3k tokens when loaded. Last updated 2026-08-07. 2,816 stars on the source repository.
What this skill does
MLM vs Bridge Training For how they differ, the arg mapping tables, gotchas, and translation script, see: @docs/megatron-lm-to-megatron-bridge.md First Answer Checklist For MLM-vs-Bridge correlation questions, always name these items up front: 1. Bridge recipe: vanillagptpretrainconfig. 2. Bridge entry point: scripts/training/runrecipe.py. 3. MLM entry point: 3rdparty/Megatron-LM/pretraingpt.py. 4. Launch wrapper for both: uv run python -m torch.distributed.run. 5. Fresh-run cleanup: rm -rf nemoexperiments before the Bridge run. Also state that MLM needs PYTHONPATH=3rdparty/Megatron-LM:$PYTHONPATH, matched Bridge and MLM losses should agree within BF16 rounding, and files under 3rdparty/Megatron-LM/ should not be modified from this repo. Correlation Testing Use vanillagptpretrainconfig for loss-correlation testing. This recipe uses bare GPTModelProvider defaults (LayerNorm, GeLU, learnedabsolute position embeddings, vocabsize inherited from tokenizer) — matching MLM pretraingpt.py defaults with no args. MLM Correlation Run (2L/256H, 1 GPU) Bridge Correlation Run (same config, 1 GPU) Verification With matched parameters the LM losses should be nearly identical at each iteration. Compare lm loss values from both logs — they should agree to within BF16 rounding. Multi-GPU Examples MLM 2-GPU with TP=2 Bridge 2-GPU with TP=2 Available Recipes Common recipes (use with --recipe): vanillagptpretrainconfig — Minimal GPT (bare GPTModelProvider defaults, ideal for correlation testing and custom configs) llama321bpretrainconfig — Llama 3.2 1B (16L, 2048H, GBS=512, seq=8192) llama38bpretrainconfig — Llama 3 8B qwen38bpretrainconfig — Qwen3 8B deepseekv2litepretrainconfig — DeepSeek-V2-Lite 16B MoE SFT/PEFT variants use sftconfig / peftconfig suffix. Megatron-Core Submodule For what the submodule is and why two versions exist, see @docs/megatron-lm-to-megatron-bridge.md. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nvidia/nemo-mbridge-mlm-bridge-training