Agent skill · software engineering · davila7
serving-llms-vllm
Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.
Why this skill is useful
Adds executable scripts and specific configurations for deploying high-throughput LLM APIs using vLLM, which are not commonly found in public documentation.
What it needs
Requires docker, locust, torch, transformers, vllm installed locally. About 5k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
vLLM - High-Performance LLM Serving Quick start vLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests). Installation: Basic offline inference: OpenAI-compatible server: Common workflows Workflow 1: Production API deployment Copy this checklist and track progress: Step 1: Configure server settings Choose configuration based on your model size: Step 2: Test with limited traffic Run load test before production: Verify TTFT (time to first token) < 500ms and throughput > 100 req/sec. Step 3: Enable monitoring vLLM exposes Prometheus metrics on port 9090: Key metrics to monitor: vllm:timetofirsttokenseconds - Latency vllm:numrequestsrunning - Active requests vllm:gpucacheusageperc - KV cache utilization Step 4: Deploy to production Use Docker for consistent deployment: Step 5: Verify performance metrics Check that deployment meets targets: TTFT < 500ms (for short prompts) Throughput > target req/sec GPU utilization > 80% No OOM errors in logs Workflow 2: Offline batch inference For processing large datasets without server overhead. Copy this checklist: Step 1: Prepare input data Step 2: Configure LLM engine Step 3: Run batch inference vLLM automatically batches requests for efficiency: Step 4: Process results Workflow 3: Quantized model serving Fit large models in limited GPU memory. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/inference-serving-vllm