Agent skill · nousresearch
serving-llms-vllm
vLLM: high-throughput LLM serving, OpenAI API, quantization.
What it needs
About 5k tokens when loaded.
What this skill does
vLLM - High-Performance LLM Serving When to use Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism. Quick start vLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests). Installation: Basic offline inference: OpenAI-compatible server: Common workflows Workflow 1: Production API deployment Copy this checklist and track progress: Step 1: Configure server settings Choose configuration based on your model size: Step 2: Test with limited traffic Run load test before production: Verify TTFT (time to first token) < 500ms and throughput > 100 req/sec. Step 3: Enable monitoring vLLM exposes Prometheus metrics at /metrics on the API port (default 8000): Key metrics to monitor: vllm:timetofirsttokenseconds - Latency vllm:numrequestsrunning - Active requests vllm:gpucacheusageperc - KV cache utilization Step 4: Deploy to production Use Docker for consistent deployment: Step 5: Verify performance metrics Check that deployment meets targets: TTFT < 500ms (for short prompts) Throughput > target req/sec GPU utilization > 80% No OOM errors in logs Workflow 2: Offline batch inference For processing large datasets without server overhead. Copy this checklist: Step 1: Prepare input data Step 2: Configure LLM engine Step 3: Run batch inference vLLM automatically batches requests for efficiency: Step 4: Process results Workflow 3: Quantized model serving Fit large models in limited GPU memory. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nousresearch/serving-llms-vllm--2e9001