Agent skill · google
gke-ai-troubleshooting-tpu-metrics-monitoring
Monitors and troubleshoots GKE TPU workloads, nodes, and node pools using GKE system metrics and PromQL. Use when monitoring TensorCore duty cycle, TPU memory, node readiness, multi-host TPU node pool availability, host maintenance or preemption interruptions, and calculating MTTR or MTBI metrics for GKE TPUs. Don't use for general non-TPU GKE workload monitoring or non-metric TPU debugging.
What it needs
About 3k tokens when loaded.
What this skill does
GKE TPU Metrics Monitoring Guide This skill enables the agent to monitor GKE TPU workloads, nodes, and node pools using GKE system metrics. It helps diagnose if workload interruptions or performance issues are caused by underlying infrastructure. Step 0: Mandatory Context Independently gather required context (such as cluster details or node pool names) using available GKE and Cloud tools, or use the provided {variable} placeholders: {projectid}: The GCP Project ID. {clustername}: The GKE Cluster Name. {location}: The GKE Cluster Location (region or zone). {nodename}: (Optional) The name of the specific GKE node. {nodepoolname}: (Optional) The name of the GKE node pool. --- Diagnostic Steps Step 1: Verify TPU Runtime Metrics Configuration [Low Risk] [Auto] Before analyzing runtime metrics, verify that the workload is configured to export them. This ensures the cluster and container environment are set up for automated metric scraping and visibility into accelerator health. Action: Verify that the Pod specification and cluster meet the following prerequisites: containerPort: 8431 exposed on the TPU container (required for Prometheus metric scraping). JAX version 0.4.14 or later if using JAX (earlier versions do not export runtime metrics). GKE version is 1.27.4-gke.900 or later (required for TPU runtime metric support). GKE System Metrics are enabled on the cluster (required for Cloud Monitoring ingestion). Step 2: Monitor TPU Runtime Metrics [Low Risk] [Auto] If configured correctly, the following metrics are available in Cloud Monitoring (monitored resources k8snode and k8scontainer): Container Metrics: kubernetes.io/container/accelerator/dutycycle: Percentage of time over the past sampling period (60 seconds) during which the TensorCores were actively processing on a TPU chip. kubernetes.io/container/accelerator/memoryused: Amount of accelerator memory allocated in bytes. kubernetes.io/container/accelerator/memorytotal: Total accelerator memory in bytes. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills google/gke-ai-troubleshooting-tpu-metrics-monitoring