Agent skill · software engineering · nvidia
nemo-mbridge-resiliency
Resiliency features in Megatron Bridge including fault tolerance, straggler detection, in-process restart, preemption, and re-run state machine.
Why this skill is useful
Adds specific configurations and scripts for fault tolerance and straggler detection in Megatron Bridge that the AI wouldn't reliably generate on its own.
What it needs
Requires nvidia-resiliency-ext installed locally. About 5k tokens when loaded. Last updated 2026-08-07. 2,816 stars on the source repository.
What this skill does
Resiliency Stable docs: @docs/training/resiliency.md, @docs/training/checkpointing.md Card: @skills/nemo-mbridge-resiliency/card.yaml Enablement Fault tolerance (Slurm only) Option 1: NeMo Run plugin (recommended) Plugin parameter Default Description --- --- --- numinjobrestarts 3 Max restarts within same job numjobretriesonfailure 2 Max new job launches on failure initialrankheartbeattimeout 1800 First heartbeat timeout (seconds) rankheartbeattimeout 300 Subsequent heartbeat timeout (seconds) Option 2: Direct config + ftlauncher Launch with ftlauncher (not torchrun): Config parameter Default Description --- --- --- enableftpackage False Enable fault tolerance calcfttimeouts False Auto-compute optimal timeouts simulatefault False Enable fault simulation for testing simulatedfaulttype "random" "rankhung", "rankkilled", or "random" simulatedfaultrank None Specific rank to fault (random if None) simulatedfaultbasedelay 0 Base delay before simulating fault Section-based timeout monitoring covers setup, training steps, checkpointing, and out-of-section time independently. Timeouts are saved to ftstate.json for subsequent runs when calcfttimeouts=True. NVRx straggler detection Parameter Default Description --- --- --- enabled False Enable straggler detection reporttimeinterval 300.0 Seconds between straggler checks calcrelativegpuperf True Compare ranks against each other calcindividualgpuperf True Track per-rank degradation over time gpurelativeperfthreshold 0.7 Threshold for relative performance (0-1) gpuindividualperfthreshold 0.7 Threshold for individual performance (0-1) stopifdetected False Terminate training on straggler numgpuperfscorestoprint 5 Number of best/worst scores to print profilinginterval 1 Profiling interval for detector Preemption Plugin (Slurm) Plugin parameter Default Description --- --- --- preempttime 60 Seconds before job limit to send signal enableexithandler True Enable signal handler in training enableexithandlerfordataloader False Enable for …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nvidia/nemo-mbridge-resiliency