Agent skill · data analytics · wshobson
spark-memory-thermal-ops
Manage unified memory and thermals during long-running ML jobs on NVIDIA DGX Spark. Use when planning memory headroom for a training run on GB10, when a job OOMs on unified memory, or when monitoring temperature and power during multi-hour training.
Why this skill is useful
Provides specific domain knowledge on managing memory and thermal operations for NVIDIA DGX Spark during ML jobs, which the AI wouldn't reliably generate on its own.
What it needs
About 5k tokens when loaded. Last updated 2026-08-05. 38,570 stars on the source repository.
What this skill does
Spark Memory & Thermal Ops DGX Spark's GB10 chip has one 128GB unified memory (UMA) pool shared by CPU and GPU, and a sustained power ceiling well below its rated figure. Both break discrete-GPU assumptions: headroom isn't what nvidia-smi reports, and a run that starts fast will slow down mid-job with nothing misconfigured. This skill covers planning memory headroom, working an actual OOM, and watching thermals across a long job. For launch-time failure modes (ABI mismatches, flash-attn, playbook breakage), see spark-training-gotchas — this skill assumes the job starts. Common Issues Quick Reference Situation Do this --- --- Planning headroom before launch Budget against free -g, not nvidia-smi — see UMA Memory Model Job OOMs on unified memory Work the OOM Ladder in order: flush, then batch/pack, then method downgrade Throughput drops mid-run Check the power/temp log before assuming a config bug — see Thermal Monitoring Trainer + inference server both wanted Run one at a time — see Concurrent Workloads When to Use This Skill Sizing a training run against the 128GB pool before launch — will this model, method, and batch/pack combination fit. A run OOMs mid-load or mid-step and the remediation order matters — what to try first, second, third. Watching temperature and power during a multi-hour job, deciding whether a slowdown is thermal throttling or something else. Planning to run a trainer alongside an inference server (vLLM, Ollama) on the same box. UMA Memory Model Spark has no separate GPU VRAM — the GPU and CPU share one 128GB pool. Two consequences: nvidia-smi and cudaMemGetInfo underreport pressure — or report nothing at all. Both report CUDA-allocator-visible memory, not the pool's actual state — a box can show headroom in nvidia-smi and still OOM, because page-cache and mmap'd pages the allocator doesn't see consume the same pool. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills wshobson/spark-memory-thermal-ops