Agent skill · NVIDIA
nemo-automodel-distributed-training
Guide for selecting and configuring distributed training strategies in NeMo AutoModel, including FSDP2, Megatron FSDP, DDP, and parallelism settings.
What it needs
About 9k tokens when loaded.
What this skill does
Distributed Training in NeMo AutoModel Purpose NeMo AutoModel uses PyTorch-native distributed training. All parallelism is orchestrated through a single MeshContext object that holds device meshes, strategy configs, and axis names. <!-- NVSkills catalog signing requested after PR #2937 (2026-07-31). --> Instructions For conceptual distributed-training questions, answer directly from the quick patterns in this skill without inspecting the repository. Start with the strategy choice, then list only the YAML fields and constraints relevant to the question. Use direct action verbs in the final answer: recommend the strategy, show the minimal YAML, state the sizing constraint, and name the unsupported strategies. Do not discuss model onboarding, recipes, Slurm, SkyPilot, or checkpointing unless the user asks. Examples TP plus PP for a large multi-node model Recommend strategy: fsdp2. Mention tpsize, ppsize, cpsize, epsize, and the pipeline sub-config. State that dpsize is inferred from worldsize / (tpsize ppsize cpsize). MoE expert parallelism Recommend strategy: fsdp2 with epsize > 1. Say this creates a separate moemesh; include the moe sub-config when relevant; state that epsize must divide dpsize cpsize. Do not recommend megatronfsdp or ddp. MegatronFSDP limitations Say no for pipeline parallelism, expert parallelism, and sequenceparallel. Recommend fsdp2 for PP, EP, or sequenceparallel; mention that DDP is only simple data parallelism. Strategy Selection Three strategies are available, selected via the distributed.strategy YAML key: Strategy YAML value Best for --- --- --- FSDP2 fsdp2 General use, recommended default. Supports TP, PP, CP, EP, HSDP. MegatronFSDP megatronfsdp NVIDIA Megatron-style FSDP. No PP, no EP, no sequenceparallel. DDP ddp Simple data parallelism only. No TP, PP, CP, or EP. Decision tree: Single GPU: no distributed config needed (FSDP2Manager skips parallelization when worldsize=1). Multi-GPU single node: fsdp2 (default). …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/nemo-automodel-distributed-training