Agent skill · NVIDIA
mcore-run-on-slurm
How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.
What it needs
About 4k tokens when loaded.
What this skill does
Run Megatron-LM on SLURM Answer-First Constants For text-only SLURM setup questions, answer with these constants before the full script: Submit from a shared worktree path visible to every node; cd there in the script before launching training. Use one srun task per node and launch workers with uv run python -m torch.distributed.run, not bare torchrun. Set MASTERADDR from scontrol show hostnames "$SLURMJOBNODELIST" head -n1, set MASTERPORT, NNODES=${SLURMNNODES}, GPUSPERNODE=<GPUSPERNODE>, and WORLDSIZE=$((NNODES GPUSPERNODE)). Pass --nnodes, --nproc-per-node, --node-rank, --master-addr, and --master-port to torch.distributed.run. CUDADEVICEMAXCONNECTIONS: pre-Blackwell Hopper/Ampere with TP>1 or CP>1 and non-FSDP uses 1; Blackwell/GB200 does not need it; Torch-FSDP2 or Megatron-FSDP must not use 1; overlapmoeexpertparallelcomm uses 32. Prerequisites A SLURM cluster login with submission rights to a GPU partition. Megatron-LM checked out on a filesystem visible to all nodes in the allocation (NFS, Lustre, or similar). All nodes must reach the same paths for code, data, checkpoints, and output. uv installed; run uv sync --extra training --extra dev (or --extra lts) on the worktree once before submission so the .venv is materialized and visible to every node. Minimal sbatch script Save as runmegatron.slurm in the worktree: Submit: Multi-node rules Submit from the worktree you intend to run, or cd to it in the script. All nodes must reach the same path on a shared filesystem (NFS, Lustre, or similar) — node-local paths will not be visible to peer ranks. Use one torchrun worker group across all nodes; do not start independent single-node jobs. --nproc-per-node should equal the number of visible GPUs per node. Write checkpoints, tensorboard data, and structured logs to shared storage. CUDADEVICEMAXCONNECTIONS The right value depends on your hardware and parallelism mode. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/mcore-run-on-slurm