Agent skill · NVIDIA
jetson-inference-mem-tune
Pick the serving stack and per-runtime memory flags (vLLM, SGLang, llama.cpp, TensorRT Edge-LLM) for an LLM/VLM workload on any NVIDIA Jetson.
What it needs
About 6k tokens when loaded.
What this skill does
Jetson Inference Memory Tuning Recommends an inference runtime and the specific memory-related flags to pass to it, given the Jetson SKU/variant and the user's workload. Does not include quantization recipe selection — that lives in the model-benchmarking skill — but it does point at the precision floor each runtime can serve efficiently. Purpose Turn a live jetson-memory-audit snapshot into runtime and launch-flag recommendations for LLM/VLM serving on Jetson. Use this when the user needs to fit a model, reduce OOM risk, or switch to a lower-memory serving stack. When to use "Which serving stack should I use on Orin Nano 8 GB to run a 7B model?" "vLLM is OOMing — what should --gpu-memory-utilization and --max-model-len be?" "Same model, less memory — can I switch from vLLM to llama.cpp?" After jetson-memory-audit shows a model server is the top NvMap / PSS consumer. Prerequisites Start with a current jetson-memory-audit/scripts/audit.sh JSON snapshot from the target Jetson. Know the intended workload: llm-server, vlm-server, embedding, or rag. If the user gives a desired free-memory target, pass it as --target-mb; otherwise let the script use SKU defaults. Available Scripts Script Purpose Arguments -------- --------- ----------- scripts/recommend.py Reads an audit JSON and emits runtime plus launch-flag recommendations. --audit PATH, --runtime, --workload, --target-mb, --human. If your agent runtime supports runscript, invoke runscript("scripts/recommend.py", ["--audit", "/tmp/audit.json", "--runtime", "auto", "--workload", "llm-server"]) and summarize the returned JSON. Otherwise run it with python3 from the repository root. Instructions 1. Run jetson-memory-audit/scripts/audit.sh to capture the device baseline. 2. Run scripts/recommend.py --audit /tmp/audit.json --runtime auto --workload llm-server --target-mb 6000 to get a JSON of runtime + flag recommendations. 3. The agent presents the suggested runtime and the exact CLI flags. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/jetson-inference-mem-tune