Agent skill · software engineering · davila7
sglang
Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.
Why this skill is useful
Adds executable scripts for launching a high-performance inference server with advanced caching and structured output generation that aren't available in public documentation.
What it needs
Requires sglang, torch, transformers installed locally. About 5k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
SGLang High-performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching. When to use SGLang Use SGLang when: Need structured outputs (JSON, regex, grammar) Building agents with repeated prefixes (system prompts, tools) Agentic workflows with function calling Multi-turn conversations with shared context Need faster JSON decoding (3× vs standard) Use vLLM instead when: Simple text generation without structure Don't need prefix caching Want mature, widely-tested production system Use TensorRT-LLM instead when: Maximum single-request latency (no batching needed) NVIDIA-only deployment Need FP8/INT4 quantization on H100 Quick start Installation Launch server Basic inference Structured JSON output RadixAttention (Key Innovation) What it does: Automatically caches and reuses common prefixes across requests. Performance: 5× faster for agentic workloads with shared system prompts 10× faster for few-shot prompting with repeated examples Zero configuration - works automatically How it works: 1. Builds radix tree of all processed tokens 2. Automatically detects shared prefixes 3. Reuses KV cache for matching prefixes 4. Only computes new tokens Example (Agent with system prompt): Structured generation patterns JSON with schema Regex-constrained generation Grammar-based generation python\n" # EBNF grammar for Python pythongrammar = """ ?start: functiondef functiondef: "def" NAME "(" [parameters] "):" suite parameters: parameter ("," parameter) parameter: NAME suite: simplestmt NEWLINE INDENT stmt+ DEDENT """ s += sgl.gen("code", maxtokens=200, grammar=pythongrammar) s += "\n Agent workflows with function calling Performance benchmarks RadixAttention speedup Few-shot prompting (10 examples in prompt): vLLM: 2.5 sec/request SGLang: 0.25 sec/request (10× faster) Throughput: 4× higher Agent workflows (1000-token system prompt): vLLM: 1.8 sec/request SGLang: 0.35 sec/request (5× faster) JSON decoding: Standard: 45 tok/s SGLang: 135 tok/s (3× faste …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/inference-serving-sglang