Agent skill · NVIDIA
nemo-mbridge-perf-hierarchical-context-parallel
Operational guide for enabling hierarchical context parallelism in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification.
What it needs
About 2k tokens when loaded.
What this skill does
Hierarchical Context Parallel Skill This skill covers hierarchical context parallelism: nested context-parallel process groups used by cpcommtype="a2a+p2p" and configured with hierarchicalcontextparallelsizes. For what hierarchical CP is, when to use it, and the decision tree (a2a+p2p vs pure a2a vs p2p), see: @docs/training/hierarchical-context-parallel.md @skills/nemo-mbridge-perf-hierarchical-context-parallel/card.yaml Enablement Minimal Bridge override: Required constraints: prod(hierarchicalcontextparallelsizes) == contextparallelsize seqlength % (2 contextparallelsize) == 0 Transformer Engine >= 1.12.0 Code Anchors Upstream config and validation: Bridge MPU path: Bridge decentralized-PG path: Implementation Map The code anchors above show the config declarations and argument validation. Validation (MCore) TransformerConfig.postinit enforces that a2a+p2p requires HCP sizes and the product matches CP. Process group creation parallelstate.initializemodelparallel creates hierarchical CP sub-groups when HCP sizes are provided via createhierarchicalgroups. Bridge currently gets those groups through the MPU-backed ProcessGroupCollection. TE integration TEDotProductAttention passes the hierarchical groups to Transformer Engine when a2a+p2p is used. Requires Transformer Engine >= 1.12.0. Pitfalls 1. Bridge HCP is MPU-only today: If usedecentralizedpg=True, Bridge initializes flat CP groups and leaves HCP unset. 2. No checked-in Bridge recipe currently exercises HCP directly. 3. Single-GPU load helpers clear hierarchicalcontextparallelsizes. 4. Silent broken training on old stacks: If you use a2a+p2p without setting hierarchicalcontextparallelsizes, MCore now asserts. Older versions would silently disable CP communication, so each rank attended only to its local chunk and produced artificially high throughput with broken gradients. 5. Product must match: prod(hierarchicalcontextparallelsizes) must exactly equal contextparallelsize. A mismatch triggers an assertion. 6. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/nemo-mbridge-perf-hierarchical-context-parallel