Agent skill · research science · wshobson
spark-training-gotchas
Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.
Why this skill is useful
Adds detailed troubleshooting procedures and specific commands for diagnosing and fixing common failure modes in ML training on NVIDIA DGX Spark that are not widely documented.
What it needs
Requires nvidia-smi installed locally. About 5k tokens when loaded. Last updated 2026-08-05. 38,570 stars on the source repository.
What this skill does
Spark Training Gotchas DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified memory, aarch64) has ten recurring failure modes across launch, memory, thermals, bandwidth, and precision. Each is named G1–G10 so it can be checked by number — the numbering is load-bearing for tooling that runs these checks. Read this before a long run, not after hour six. When to Use This Skill A training run fails to start, with an import error or a segfault that doesn't point at the real cause. A run OOMs while nvidia-smi still shows headroom. Throughput degrades partway through a run that started fine. Before any multi-hour or multi-epoch job on GB10. Wiring two Sparks together, before picking a parallelism strategy. Choosing between FP8 and NVFP4 for a Spark-hosted run. Common Issues Quick Reference # Symptom Fix --- --- --- G1 undefined symbol / segfault cu130 wheel or container G2 flash-attn wrong backend used skip pip build; monkeypatch on NGC G3 OOM despite headroom drop page cache G4 throughput drop / reboot expect ~100W sustained cap G5 memory-bound step slow budget 180–192 GB/s G6 cache evicted mid-run one GPU server at a time G7 NVFP4 slower than FP8 stay FP8 unless sm121a G8 playbook fails outright check upstream issues G9 env breaks after install use a container G10 2-Spark TP hangs DDP/FSDP only, never TP The Ten Gotchas G1: CUDA 12/13 ABI Mismatch SYMPTOM: ImportError: undefined symbol naming a CUDA function, or a segfault on the first .cuda() call. CAUSE: most PyPI wheels link libcudart.so.12; Spark ships CUDA 13. pip never checks CUDA ABI, so it surfaces only at import or first kernel launch. CHECK: references/gotcha-checks.md G1 — the wheel's CUDA build tag. FIX: reinstall from download.pytorch.org/whl/cu130 or use a matched container. G2: flash-attn — Skip the pip Build, Watch Unsloth's Auto-Detect SYMPTOM: pip install flash-attn still fails/hangs. Unsloth may also silently train flash-attn over an explicitly requested SDPA. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills wshobson/spark-training-gotchas