Agent skill · NVIDIA
jetson-speculative-decoding
Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.
What it needs
About 3k tokens when loaded.
What this skill does
Jetson Speculative Decoding (vLLM) Speculative decoding lets a small "draft" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win/loss is dominated by VRAM headroom, not by the draft quality. This skill encodes the parts an LLM won't already know. Purpose Tune an existing Jetson vLLM deployment for faster token generation by appending the right --speculative-config and validating whether it improves single-stream decode speed. When to use TPOT/ITL is the bottleneck (TTFT is fine, output is just slow). Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency. Jetson family is Thor or AGX Orin. Do not suggest EAGLE-3 on Orin Nano/NX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup. When NOT to use High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM. Models without a published EAGLE-3 head — do not train one ad-hoc as a "fix". After applying jetson-inference-mem-tune flags that already pushed --gpu-memory-utilization near the ceiling. Free at least ~2 GB first. Prerequisites A working vLLM server recipe from jetson-llm-serve. Enough memory headroom for the draft model or EAGLE-3 head in addition to the target model. A benchmark baseline from jetson-llm-benchmark before enabling speculation. A target model with a compatible EAGLE-3 head, or a small same-family draft model for the fallback path. Instructions Append --speculative-config to the vllm serve command shown in jetson-llm-serve. EAGLE-3 (preferred when a head is published for the target model): Draft-model (fallback — pair a small same-family model): Jetson-specific tuning rules numspeculativetokens: start at 5 on Thor, 3 on AGX Orin. Higher values pay off only if the draft acceptance rate is >0.6. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/jetson-speculative-decoding