Agent skill · data analytics · jeremylongshore
coreweave-fabric-diagnostics
Diagnose the most expensive silent failure on a CoreWeave multi-node GPU job: GPUDirect RDMA falling back from InfiniBand to TCP. When NCCL drops from NET/IB to NET/Socket, collectives keep running with NO error but throughput collapses (commonly 5-20x slower) while every GPU still bills at full rate — 5x the GPU bill for the same work, invisibly. Paste an NCCL_DEBUG=INFO log (and/or a pod-spec, ibstat, or all_reduce_perf output) and the bundled deterministic script verdicts whether RDMA is actually engaged, which of the three required conditions is missing, and the fix. Use when multi-node training is slow, when checking whether RDMA/InfiniBand is engaged, or when all-reduce bandwidth looks low. Trigger with "coreweave slow training", "is RDMA working", "NCCL fell back to TCP", "NET/Socket", "GPUDirect RDMA", "infiniband not used", "multi-node training slow".
Why this skill is useful
Adds a deterministic script that diagnoses and fixes silent failures in CoreWeave multi-node GPU jobs, which the AI cannot generate from existing knowledge.
What it needs
Requires kubectl, python3 installed locally. About 8k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
CoreWeave Fabric Diagnostics Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc. Detects when a CoreWeave multi-node GPU job has silently fallen off the InfiniBand fabric onto TCP — the failure that makes distributed training run at a fraction of the hardware's speed while every GPU keeps billing at the full rate — and gives the exact fix. Overview On a CoreWeave multi-node job, NCCL should carry collectives over InfiniBand with GPUDirect RDMA (NET/IB). If any one of three conditions is missing, NCCL silently falls back to TCP sockets (NET/Socket): the job still runs, still converges, and raises no error — but all-reduce throughput collapses (commonly cited as 5-20x slower [[nccl]]) because it now crosses the Ethernet control plane instead of the 400 Gb/s-class fabric. You keep paying full GPU rate for a multi-node run that performs like a badly-connected one. This is the single highest-dollar invisible failure on the platform, and nothing in the default output flags it. The diagnosis is deterministic: the bundled scripts/fabric-check.py greps the pasted NCCLDEBUG=INFO log for the decisive Using network line (and NET/IB vs NET/Socket), parses the pod-spec's resources block for the RDMA device request, reads ibstat port state, and echoes any allreduceperf bus bandwidth — then emits a VERDICT with the rdmaengaged/transport call, the missing conditions, and the fix. The LLM never eyeballs which transport is in use; the script decides. Deep grounding lives in references/, loaded only when a leg of the diagnosis needs it. Prerequisites An NCCLDEBUG=INFO log from the actual run — the primary signal. Re-run the job (or one rank) with NCCLDEBUG=INFO set and capture stderr. The decisive line is Using network IB (good) vs Using network Socket (the fallback). This is the one input the skill really needs; everything else corroborates. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/coreweave-fabric-diagnostics