Agent skill · google

gke-ai-troubleshooting-tpu-reservation-stockout

Diagnose and resolve cluster-wide GKE TPU node pool creation or scale-up failures caused by Compute Engine reservation stockouts (GCE_STOCKOUT, ZONE_RESOURCE_POOL_EXHAUSTED) when existing unhealthy or NotReady TPU node pools in the cluster have node auto-repair disabled and hold reserved TPU capacity. Use when TPU node pool creation or scale-up fails with GCE_STOCKOUT, ZONE_RESOURCE_POOL_EXHAUSTED, or ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS despite having a specific Compute Engine TPU reservation. Don't use for general pod scheduling without a TPU reservation or workload-level container crashes.

What it needs

About 7k tokens when loaded.

What this skill does

Troubleshoot GKE TPU reservation stockouts caused by leaked node pool capacity Use this skill to diagnose and resolve GKE TPU node pool creation or scale-up failures that fail with GCESTOCKOUT, ZONERESOURCEPOOLEXHAUSTED, or ZONERESOURCEPOOLEXHAUSTEDWITHDETAILS (or queue due to lack of capacity) even though they target a specific Compute Engine TPU reservation. If node auto-repair is disabled on an existing TPU node pool, GKE doesn't repair its nodes when they become NotReady. While their VMs are still running, they keep consuming slots in the reservation (inUseCount is the number of running instances that consume the reservation). Per Ensure reservation availability, the reservation must have sufficient available TPU chips at the time of node pool creation, and per Multi-host TPU slice node pools, if GKE is unable to deploy one node in a slice, no nodes in the TPU slice node pool are deployed. When the free slots in the reservation (specificReservation.count - specificReservation.inUseCount; see Consumable instances in a reservation) are fewer than the number of nodes that the new TPU node pool needs, creation fails with GCESTOCKOUT or ZONERESOURCEPOOLEXHAUSTED (or queues due to lack of capacity). Attribution MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDKMETRICSREQUESTATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-reservation-stockout)" and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-reservation-stockout) to User-Agent headers. Prerequisites Required IAM Roles: Logs Viewer (roles/logging.viewer): Query Cloud Audit Logs in Cloud Logging. Monitoring Viewer (roles/monitoring.viewer): Query GKE node readiness metrics in Cloud Monitoring. …

How to use it

Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:

@skills google/gke-ai-troubleshooting-tpu-reservation-stockout

View the source on GitHub

Browse the @skills marketplace