Agent skill · NVIDIA
dynamo-troubleshoot
Diagnose failed or unhealthy Dynamo deployments. Use when pods, model-cache jobs, PVCs, workers, frontend/router health, endpoints, or benchmark jobs fail; use recipe-runner/router-starter before this for normal bring-up.
What it needs
About 3k tokens when loaded.
What this skill does
Dynamo Troubleshoot <!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: CC-BY-4.0 --> Purpose Turn a Dynamo failure into a clear problem class, strongest signal, and next action. Start with read-only evidence, avoid secrets, and fix one layer at a time. Prerequisites Python 3.10+ on the operator machine. kubectl configured with read access to the target namespace. Permission to read pods, events, jobs, PVCs, and DynamoGraphDeployment resources (NOT secrets). Network reachability to the cluster API server. Instructions 1. Collect A Read-Only Bundle Run: If the user names a deployment, include it: Do not collect Kubernetes secrets. Do not print Hugging Face tokens. 2. Classify The Failure Use references/failure-decision-tree.md and classify into one primary bucket: cluster/platform namespace/secret model cache/PVC/download image pull/runtime image GPU scheduling/resources operator/DynamoGraphDeployment reconciliation frontend/router worker/backend endpoint/API benchmark/perf job 3. Debug Top Down Check in this order: 1. namespace, storage class, GPU nodes, and HF secret existence 2. PVC and model-download job 3. DynamoGraphDeployment status and events 4. pod status, describe pod, and container logs 5. frontend service and port-forward 6. /v1/models 7. /v1/chat/completions 8. benchmark job only after endpoint smoke test passes 4. Fix One Layer At A Time Prefer the smallest reversible change: create missing namespace or HF secret patch storageClassName patch image tag or image pull secret reduce GPU request only if the recipe can still be valid switch KV router to approximate mode only if workers do not publish events restart failed jobs after fixing the underlying config After each fix, rerun the relevant readiness check before moving deeper. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/dynamo-troubleshoot