Agent skill · NVIDIA
nemo-mbridge-resiliency
Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.
What it needs
About 5k tokens when loaded.
What this skill does
Resiliency Stable docs: @docs/training/resiliency.md, @docs/training/checkpointing.md Card: @skills/nemo-mbridge-resiliency/card.yaml Enablement Fault tolerance (Slurm only) Option 1: NeMo Run plugin (recommended) Plugin parameter Default Description --- --- --- numinjobrestarts 3 Max restarts within same job numjobretriesonfailure 2 Max new job launches on failure initialrankheartbeattimeout 1800 First heartbeat timeout (seconds) rankheartbeattimeout 300 Subsequent heartbeat timeout (seconds) Option 2: Direct config + ftlauncher Launch with ftlauncher (not torchrun): Config parameter Default Description --- --- --- enableftpackage False Enable fault tolerance calcfttimeouts False Auto-compute optimal timeouts simulatefault False Enable fault simulation for testing simulatedfaulttype "random" "rankhung", "rankkilled", or "random" simulatedfaultrank None Specific rank to fault (random if None) simulatedfaultbasedelay 0 Base delay before simulating fault Section-based timeout monitoring covers setup, training steps, checkpointing, and out-of-section time independently. Timeouts are saved to ftstate.json for subsequent runs when calcfttimeouts=True. NVRx straggler detection Parameter Default Description --- --- --- enabled False Enable straggler detection reporttimeinterval 300.0 Seconds between straggler checks calcrelativegpuperf True Compare ranks against each other calcindividualgpuperf True Track per-rank degradation over time gpurelativeperfthreshold 0.7 Threshold for relative performance (0-1) gpuindividualperfthreshold 0.7 Threshold for individual performance (0-1) stopifdetected False Terminate training on straggler numgpuperfscorestoprint 5 Number of best/worst scores to print profilinginterval 1 Profiling interval for detector Preemption Plugin (Slurm) Plugin parameter Default Description --- --- --- preempttime 60 Seconds before job limit to send signal enableexithandler True Enable signal handler in training enableexithandlerfordataloader False Enable for …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/nemo-mbridge-resiliency