Agent skill · software engineering · nvidia
nemo-mbridge-perf-cuda-graphs
Validate and use CUDA graph capture in Megatron Bridge, including local full-iteration graphs and Transformer Engine scoped graphs for attention, MLP, and MoE modules.
Why this skill is useful
Adds specific CUDA graph configurations and performance harness scripts that enhance GPU operation efficiency, which are not commonly known.
What it needs
Requires python installed locally. About 6k tokens when loaded. Last updated 2026-08-07. 2,816 stars on the source repository.
What this skill does
CUDA Graphs Stable documentation: @docs/training/cuda-graphs.md Card: @skills/nemo-mbridge-perf-cuda-graphs/card.yaml <!-- NVSkills CI refresh: 2026-06-15. No instruction changes. --> What It Is CUDA graphs capture GPU operations once and replay them with minimal host-driver overhead. Bridge supports two implementations: cudagraphimpl Mechanism Scope support --- --- --- "local" MCore FullCudaGraphWrapper wrapping entire fwd+bwd fulliteration "transformerengine" TE makegraphedcallables() per layer attn, mlp, moe, moerouter, moepreprocess, mamba Quick Decision Start with TE-scoped graphs for most training workloads, then verify replay timing against eager on the same dispatcher, layout, and container: dense models: attn, then optionally mlp dropless MoE: attn moerouter moepreprocess VLMs: the same dropless-MoE scope, but only after the real-data path is stable Use local + fulliteration only when you specifically want full-iteration capture and can satisfy the tighter constraints. For recompute-heavy workloads: TE-scoped graphs pair naturally with selective recompute full recompute usually pushes you toward local full-iteration graphs or away from graphs entirely Related docs: @docs/training/cuda-graphs.md @docs/training/activation-recomputation.md Enablement Local full-iteration graph TE scoped graph (dense model) TE scoped graph (MoE model) Performance harness CLI Valid CLI values live in scripts/performance/argumentparser.py: VALIDCUDAGRAPHIMPLS: ["none", "local", "transformerengine"] VALIDCUDAGRAPHSCOPES: ["fulliteration", "attn", "mlp", "moe", "moerouter", "moepreprocess", "mamba"] The performance harness uses a comma-separated --cudagraphscope value and auto-enables model.useterngtracker plus rng.terngtracker when --cudagraphimpl is not none. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nvidia/nemo-mbridge-perf-cuda-graphs