Agent skill · operations · jeremylongshore
coreweave-incident-runbook
Incident response runbook for CoreWeave GPU workload failures.
Why this skill is useful
Provides specific Kubernetes commands and troubleshooting steps for managing GPU workloads on CoreWeave that the AI wouldn't reliably generate on its own.
What it needs
Requires kubectl installed locally. About 2k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
CoreWeave Incident Runbook Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc. Overview Respond to GPU workload incidents by stabilizing customer impact, preserving redacted evidence, and restoring a known-good state. The incident commander owns communications and escalation; responders use only the access required for triage. Prerequisites An incident ID, named commander, affected namespace/service, and on-call route. Authorized read-only cluster access plus a documented production rollback revision. A redaction policy for logs, prompts, model artifacts, and credentials. Instructions 1. Declare severity and scope, then capture pod, event, node, and service status. 2. Stabilize impact with the documented scale, failover, or rollback action before root-cause work. 3. Collect only redacted, bounded diagnostics and escalate hardware or capacity issues through CoreWeave support. 4. Verify recovery against the service SLO, update stakeholders, and create follow-up work for root cause and prevention. Triage Steps Common Incidents Inference Service Down 1. Check pod status and events 2. If OOMKilled: reduce batch size or upgrade GPU 3. If ImagePullBackOff: check registry credentials 4. If Pending: check GPU quota and availability GPU Node Failure 1. Pods will be rescheduled automatically 2. If no capacity: scale down non-critical workloads 3. Contact CoreWeave support for extended outages Model Loading Failure 1. Check HuggingFace token secret exists 2. Verify model name spelling 3. Check PVC has sufficient storage 4. Review container logs for download errors Rollback Output A time-stamped incident record with scope, owner, stabilization action, and redacted evidence. A verified recovery or an explicit escalation with a safe customer-impact mitigation. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/coreweave-incident-runbook--11cd81