Agent skill · software engineering · nvidia
tao-run-on-kubernetes
Kubernetes execution platform — submits TAO container jobs as k8s Jobs with NVIDIA GPU scheduling; single-pod
Why this skill is useful
Adds executable scripts for setting up and validating NVIDIA GPU environments on Kubernetes clusters, which are not commonly found in public documentation.
What it needs
Requires kubernetes, nvidia-tao-sdk installed locally. About 10k tokens when loaded. Last updated 2026-08-07. 2,816 stars on the source repository.
What this skill does
Kubernetes Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery). Submits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS / GKE / AKS / on-prem) or in-cluster service account (when running inside a pod). Single-pod by default; opt into multi-node distributed training via numnodes > 1 (uses Indexed Job + headless Service, see Multi-node training below). Preflight Three checks: GPU host runtime ready, cluster reachable via kubectl, GPU Operator/device plugin present. The GPU node runtime check is mandatory for self-managed nodes. For managed clusters where the client is not running on a GPU worker, verify the provider node image or GPU Operator policy and set TAOK8SSKIPNODERUNTIMECHECK=1 instead of running the installer on the client. The GPU-capacity warning here is a soft check; the submit verb re-checks allocatable nvidia.com/gpu and hard-fails before applying the manifest (there is no gang scheduling, so a too-big Job would sit Pending forever). Credentials & configuration Kubeconfig (one of): ~/.kube/config — default discovery path $KUBECONFIG — alternate path In-cluster service account — used when running inside a pod (no kubeconfig needed) TAOK8SNAMESPACE (optional): default namespace for Job submission. Defaults to default. TAOK8SCONTEXT (optional): kubeconfig context name to switch clusters. NGCKEY (optional): for nvcr.io image pulls. If you've pre-created an image-pull secret in the target namespace, reference its name in the rendered manifest's imagePullSecrets. AWSACCESSKEYID / AWSSECRETACCESSKEY / S3BUCKETNAME / S3ENDPOINTURL (optional): for S3 dataset I/O (storage tier C), injected into the pod via the per-job Secret (envFrom.secretRef), never inline. Legacy ACCESSKEY/SECRETKEY are mapped by tao-data-io. Do not ask for Brev or SLURM credentials for Kubernetes runs. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nvidia/tao-run-on-kubernetes