Agent skill · google
gke-ai-troubleshooting-tpu-mxla-hang
Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze without progressing steps, emit `HANG_DETECTED`, or stall in collective operations. Don't use for gradual step-time throughput drops without hangs (use gke-ai-troubleshooting-tpu-performance-degradation) or pod preemption/eviction restarts (use gke-ai-troubleshooting-jobset-interruption).
What it needs
About 7k tokens when loaded.
What this skill does
Troubleshoot GKE TPU multi-slice hangs with the MXLA Hang Analyzer Diagnose Cloud TPU multi-slice training hangs on Google Kubernetes Engine (GKE) by correlating Megascale HANGDETECTED logs and ML Diagnostics Workload Monitoring Megascale XLA (MXLA) Hang Analyzer reports with 1-minute multi-slice latency metrics (kubernetes.io/container/multislice/) and GKE node topology labels. --- Attribution MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDKMETRICSREQUESTATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)" and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang) to User-Agent headers. Prerequisites Tools: Install the Google Cloud SDK (gcloud with alpha component for gcloud alpha mldiagnostics) and kubectl. Cloud Billing & Project Configuration: Verify an active billing account is linked (gcloud billing projects describe {projectid}), authenticate (gcloud auth login), set the target project (gcloud config set project {projectid}), and ensure container.googleapis.com, logging.googleapis.com, monitoring.googleapis.com, and hypercomputecluster.googleapis.com are enabled. Supported workloads and versions: Google Cloud ML Diagnostics only supports JAX on TPUs (see ML Diagnostics platform). Workload Monitoring is enabled by default, supports the jobset and job GKE job types, and is compatible with GKE versions 1.36.0-gke.4681000 and later (Configure GKE for ML Diagnostics, which also covers the cluster setup needed for on-demand profiling in Step 4 Path B). If a workload uses another framework (such as PyTorch) or another custom resource type, gcloud alpha mldiagnostics won't list ML runs or monitored events for it. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills google/gke-ai-troubleshooting-tpu-mxla-hang