Agent skill · NVIDIA
nemo-mbridge-perf-cpu-offloading
Validate and use CPU offloading in Megatron Bridge, including layer-level activation offloading and fractional optimizer state offloading with HybridDeviceOptimizer.
What it needs
About 4k tokens when loaded.
What this skill does
CPU Offloading References Stable docs: @docs/training/cpu-offloading.md Structured metadata: @skills/nemo-mbridge-perf-cpu-offloading/card.yaml What It Is Two independent mechanisms to move data from GPU to CPU memory: Mechanism Config namespace What gets offloaded PP restriction --- --- --- --- Activation offloading model.cpuoffloading Activations (and optionally weights) per transformer layer PP must be 1 Optimizer offloading optimizer.optimizercpuoffload Adam optimizer states (momentum + variance) via HybridDeviceOptimizer None Quick Decision Situation Recommendation --- --- Large MoE model (30B+), needs PP > 1 Optimizer offloading — activation offloading is blocked by PP=1 Small/medium model, PP=1 fits, activation memory dominates Activation offloading Want tunable memory-speed tradeoff Optimizer offloading with fractional optimizeroffloadfraction Throughput is top priority Don't enable — offloading always adds overhead CUDA graphs are needed Only optimizer offloading — activation offloading is incompatible Memory pressure is moderate Optimizer offload at 25–50% fraction for best efficiency Enablement Optimizer CPU offloading (recommended for large models) CLI overrides: Activation CPU offloading (small/medium models only) Config Parameter Reference Optimizer offloading Parameter Default Description ----------- --------- ------------- optimizercpuoffload False Master switch optimizeroffloadfraction 0.0 Fraction of optimizer states on CPU (0.0–1.0) overlapcpuoptimizerd2hh2d False Overlap GPU↔CPU transfers with compute usetorchoptimizerforcpuoffload False Use torch.optim instead of fused optimizer for CPU portion Activation offloading Parameter Default Description ----------- --------- ------------- cpuoffloading False Master switch cpuoffloadingnumlayers 0 Number of transformer layers to offload (0 to numlayers-1) cpuoffloadingactivations True Offload activations cpuoffloadingweights False Offload weights cpuoffloadingdoublebuffering False Double-buffer across la …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/nemo-mbridge-perf-cpu-offloading