Agent skill · software engineering · zechenzhangagi
sglang
Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.
Why this skill is useful
Adds executable scripts for launching a high-performance inference server with automatic prefix caching that significantly enhances LLM capabilities.
What it needs
Requires sglang, torch, transformers installed locally. About 5k tokens when loaded. Last updated 2026-06-16. 11,472 stars on the source repository.
What this skill does
SGLang High-performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching. When to use SGLang Use SGLang when: Need structured outputs (JSON, regex, grammar) Building agents with repeated prefixes (system prompts, tools) Agentic workflows with function calling Multi-turn conversations with shared context Need faster JSON decoding (3× vs standard) Use vLLM instead when: Simple text generation without structure Don't need prefix caching Want mature, widely-tested production system Use TensorRT-LLM instead when: Maximum single-request latency (no batching needed) NVIDIA-only deployment Need FP8/INT4 quantization on H100 Quick start Installation Launch server Basic inference Structured JSON output RadixAttention (Key Innovation) What it does: Automatically caches and reuses common prefixes across requests. Performance: 5× faster for agentic workloads with shared system prompts 10× faster for few-shot prompting with repeated examples Zero configuration - works automatically How it works: 1. Builds radix tree of all processed tokens 2. Automatically detects shared prefixes 3. Reuses KV cache for matching prefixes 4. Only computes new tokens Example (Agent with system prompt): Structured generation patterns JSON with schema Regex-constrained generation Grammar-based generation python\n" # EBNF grammar for Python pythongrammar = """ ?start: functiondef functiondef: "def" NAME "(" [parameters] "):" suite parameters: parameter ("," parameter) parameter: NAME suite: simplestmt NEWLINE INDENT stmt+ DEDENT """ s += sgl.gen("code", maxtokens=200, grammar=pythongrammar) s += "\n Agent workflows with function calling Performance benchmarks RadixAttention speedup Few-shot prompting (10 examples in prompt): vLLM: 2.5 sec/request SGLang: 0.25 sec/request (10× faster) Throughput: 4× higher Agent workflows (1000-token system prompt): vLLM: 1.8 sec/request SGLang: 0.35 sec/request (5× faster) JSON decoding: Standard: 45 tok/s SGLang: 135 tok/s (3× faste …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills zechenzhangagi/sglang