Agent skill · data analytics · google
gke-ai-troubleshooting-handle-disruption-gpu-tpu
Diagnoses, predicts, and mitigates node disruptions during Compute Engine host maintenance and hardware or software maintenance events for GPU and TPU workloads on GKE. Use when diagnosing node disruptions, predicting host maintenance events on GPU/TPU nodepools, inspecting node interruption PromQL metrics, auditing node taints, or configuring workload protection strategies (graceful termination, opportunistic maintenance, PodDisruptionBudgets). Don't use for general GKE cluster creation, network policy configuration, or non-disruption workload deployment.
Why this skill is useful
Provides specific PromQL queries and diagnostic workflows for troubleshooting GPU/TPU disruptions on GKE that the AI wouldn't reliably generate on its own.
What it needs
About 3k tokens when loaded. Last updated 2026-08-06. 15,831 stars on the source repository.
What this skill does
Handle Disruption on GPUs and TPUs Troubleshooting Attribution MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDKMETRICSREQUESTATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-handle-disruption-gpu-tpu)" and append agent-skills/1.0 (skill:gke-ai-troubleshooting-handle-disruption-gpu-tpu) to User-Agent headers. 🔍 Diagnostic Workflow Step 0: Context Acquisition Mandatory: When a user asks to debug or investigate an actual workload disruption, node crash, or unexpected restart without providing complete cluster details, you MUST immediately halt and request all missing mandatory parameters (projectid, location, clustername, timestamp) BEFORE delivering theories or general diagnostic commands. Only skip context acquisition if the user explicitly requests a generic reusable runbook or provides a complete static telemetry/log dump for offline analysis. Optional: nodename, workloadname, workloadnamespace, nodepoolname. Step 1: [Low Risk] Check for Upcoming Scheduled Maintenance Action: Propose running kubectl to check if nodes have the scheduled maintenance label indicating an upcoming disruption. Example Command: Interpretation: The SCHEDULED-MAINTENANCE-TIME column shows the Unix epoch time when the VM is scheduled for maintenance. If this label exists, a disruption is guaranteed to occur. Step 2: [Low Risk] Investigation via Cloud Monitoring (PromQL) Action: Call any available monitoring tool or provide PromQL for manual verification. Mandatory Monitoring Rule: Whenever recommending follow-up monitoring or interruption tracking over time, you MUST explicitly present a PromQL query using the metric kubernetesio:nodeinterruptioncount filtered by interruptionreason="HW/SW Maintenance". …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills google/gke-ai-troubleshooting-handle-disruption-gpu-tpu