Agent skill · software engineering · aradotso
club-3090-llm-serving
Recipes and configs for serving LLMs locally on RTX 3090 GPUs using vLLM, llama.cpp, and SGLang with OpenAI-compatible API
Why this skill is useful
Adds multiple executable scripts for deploying LLMs locally with specific configurations and optimizations for RTX 3090 GPUs that aren't widely documented.
What it needs
Requires docker, nvidia-container-toolkit installed locally. About 7k tokens when loaded. Last updated 2026-07-09. 70 stars on the source repository.
What this skill does
club-3090 LLM Serving Skill by ara.so — Daily 2026 Skills collection. Community recipes for serving modern LLMs on RTX 3090 (24 GB) hardware. Supports vLLM, llama.cpp, and SGLang engines with validated Docker Compose configs exposing an OpenAI-compatible API on localhost:8020. Currently ships Qwen3.6-27B configs for 1× and 2× cards. --- Engine Decision Matrix Need Engine Why --- --- --- Max throughput (code/chat) vLLM dual 89–127 TPS, MTP n=3, vision, tools Full 262K context, no crashes llama.cpp single No prefill cliffs, stable tool-use 4 concurrent streams @ 262K vLLM dual turbo Stream isolation, full feature stack Single card, moderate ctx vLLM default ~89 TPS, easiest setup SGLang is currently blocked on Qwen3.6-27B — see models/qwen3.6-27b/sglang/README.md. --- Prerequisites --- Installation & Setup 1. Clone the repo 2. Download and verify a model 3. Launch (interactive wizard) 4. Launch (non-interactive) --- Key Scripts Script Purpose --- --- scripts/setup.sh <model> Preflight checks, model download, SHA verify, Genesis patch clone scripts/launch.sh [--variant X] Interactive or direct variant boot; calls switch.sh + verify-full.sh scripts/switch.sh <variant> Stateless switcher — tears down old compose, brings up new one scripts/health.sh Live health probe: KV %, MTP accept-length, recent TPS, errors scripts/verify.sh Quick smoke test (engine-aware via env vars) scripts/verify-full.sh 8-check functional test (~1–2 min) scripts/verify-stress.sh Boundary stress test: 262K ladder + tool prefill OOM (~5–10 min) scripts/bench.sh Canonical TPS benchmark (3 warm + 5 measured runs) Common script usage --- Variant Names Reference --- API Usage (OpenAI-compatible, port 8020) The server exposes a standard OpenAI-compatible API. Use the openai Python SDK pointed at localhost:8020. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/club-3090-llm-serving