Agent skill · google

gke-ai-troubleshooting-tpu-performance-degradation

Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (`gcloud alpha mldiagnostics monitored-events` / `hypercomputecluster.googleapis.com/v1alpha`) and 1-minute Cloud Monitoring system metrics (`kubernetes.io/node/accelerator/*`). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or duty cycle drops without crashing pods, when `PERFORMANCE_DEGRADATION` monitored events fire, or when triaging slow multi-slice training steps. Don't use for complete multi-slice XLA execution stalls with `HANG_DETECTED` logs (use gke-ai-troubleshooting-tpu-mxla-hang) or pod eviction/interruption restarts (use gke-ai-troubleshooting-jobset-interruption).

What it needs

About 6k tokens when loaded.

What this skill does

Troubleshoot GKE TPU performance degradation with ML Diagnostics Workload Monitoring Diagnose and mitigate Cloud TPU training throughput drops and step-time regressions (15%+ drop in TPU duty cycle) on Google Kubernetes Engine (GKE) by correlating ML Diagnostics Workload Monitoring MonitoredEvent analyzer reports with 1-minute Cloud Monitoring system metrics and GKE node topology labels. --- Attribution MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDKMETRICSREQUESTATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-performance-degradation)" and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-performance-degradation) to User-Agent headers. Prerequisites Tools: Install the Google Cloud SDK (gcloud with alpha component for gcloud alpha mldiagnostics) and kubectl. Cloud Billing & Project Configuration: Verify an active billing account is linked (gcloud billing projects describe {projectid}), authenticate (gcloud auth login), set the target project (gcloud config set project {projectid}), and ensure container.googleapis.com, monitoring.googleapis.com, and hypercomputecluster.googleapis.com are enabled. Supported workloads and GKE versions: Google Cloud ML Diagnostics only supports JAX on TPUs (see ML Diagnostics platform). Workload Monitoring is enabled by default, supports the jobset and job GKE job types, and is compatible with GKE versions 1.36.0-gke.4681000 and later, as stated in Configure GKE for ML Diagnostics. If a workload uses another framework (such as PyTorch) or another custom resource type, gcloud alpha mldiagnostics won't list ML runs or monitored events for it. On-demand profiling (Step 4 Path A) additionally requires the cluster setup described in that document. …

How to use it

Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:

@skills google/gke-ai-troubleshooting-tpu-performance-degradation

View the source on GitHub

Browse the @skills marketplace