Agent skill · NVIDIA
nemo-mbridge-perf-parallelism-strategies
Operational guide for choosing and combining parallelism strategies in Megatron Bridge, including sizing rules, hardware topology mapping, and combined parallelism configuration.
What it needs
About 6k tokens when loaded.
What this skill does
Parallelism Strategy Selection Skill For stable background on each parallelism type, see: @docs/parallelisms.md @skills/nemo-mbridge-perf-parallelism-strategies/card.yaml Decision by Model Size Dense models Model size GPUs Recommended starting point --- --- --- < 1B 1-8 DP only 1-10B 8-16 TP=2-4 + DP 10-70B 16-64 TP=4-8 + PP=2-4 + DP 70-175B 64-256 TP=8 + PP=4-8 + DP 175-500B 256-1024 TP=8 + PP=8-16 + CP=2 + DP MoE models MoE parallelism differs from dense models. Because only a fraction of parameters are active per token, TP can often stay at 1 or 2 — the active parameter shard already fits on a single GPU. EP is the primary scaling dimension, with PP handling cross-node layer distribution. Model (total / active) TP PP EP Notes --- --- --- --- --- OLMoE 7B / 1B 1 1 8 EP only, fits single node Moonlight 16B / 3B 2 1 8 small TP for shared layers DeepSeek-V2 236B / 21B 1 4 32 no TP at all GLM-4.5 Air 106B / 12B 1 4 8 no TP at all Qwen3 30B-A3B 4 2 4 GLM-4.5 355B / 32B 2 8 16 Qwen3 235B-A22B 4 16 8 CP=2 for pretrain DeepSeek-V3 671B / 37B 2 16 64 TP=2, not 8 Kimi-K2 1T 2 16 32 Key patterns: TP is sized by active params, not total params. A 671B MoE with 37B active needs far less TP than a 70B dense model. EP scales with expert count. Common: EP = numexperts or numexperts / expertspergpu. PP handles depth. Large MoE models use PP=8-16 across nodes. ETP (expert tensor parallelism) is rarely used. Llama 4 is an exception (ETP=4). These are starting points, not hard rules. Always profile the first iteration to verify memory and communication. Decision by Hardware Topology Single node with NVLink: Multiple nodes with InfiniBand: Limited network (Ethernet): The stable rule is: keep TP within a single NVLink domain. Use PP or DP for cross-node scaling. TP across nodes is almost always a performance loss. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/nemo-mbridge-perf-parallelism-strategies