Agent skill · NVIDIA
jetson-llm-serve
Stand up vLLM or SGLang serving on Jetson, using upstream vLLM on Thor and Orin JetPack 7.2+, and NVIDIA-AI-IOT vLLM on older Orin.
What it needs
About 7k tokens when loaded.
What this skill does
Jetson LLM Serve Encodes the Jetson AI Lab GenAI tutorial: on Orin JetPack 7.2 / L4T r39+, use upstream vLLM 0.20+ (vllm/vllm-openai:latest); on older Orin, pick the NVIDIA-AI-IOT prebuilt vLLM container; on Thor, use upstream vLLM 0.20+ or validated native vLLM 0.20+, and use NVIDIA SGLang 26.01 (nvcr.io/nvidia/sglang:26.01-py3, SGLang 0.5.5.post2) when SGLang is requested. Set MAXN, make Hugging Face credentials/cache available, and launch an OpenAI-compatible server. Works for both LLMs and VLMs. Purpose Provide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step. When to use "Run / serve / host this model on a Jetson." "Start a vLLM server I can hit from Open WebUI / my app." After jetson-inference-mem-tune produced launch flags and the user wants to actually start the server. For recipe-only questions, answer from this document without starting containers. Run live pre-flight checks only when the user asks you to check this device or execute the deployment. Prerequisites Run on the Jetson host or a shell with Docker access to the Jetson GPU runtime. Know the target Jetson generation (thor or orin) and the model identifier or local checkpoint path. Use HFTOKEN only when the model is gated/private; public models should omit the token environment variable. Use jetson-inference-mem-tune first when memory headroom or launch flags are uncertain. Instructions For recipe questions, provide a complete launch recipe instead of trying to call jetson-llm-serve as a tool. A complete answer includes: The Jetson-appropriate runtime path: upstream vLLM 0.20+ (vllm/vllm-openai:latest) or NVIDIA SGLang 26.01 (nvcr.io/nvidia/sglang:26.01-py3, SGLang 0.5.5.post2) on Thor, NVIDIA-AI-IOT vLLM container on older Orin, or upstream vLLM 0.20+ on Orin JetPack 7.2 / L4T r39+. The model checkpoint / Hugging Face repo the user named. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/jetson-llm-serve