Agent skill · research science · aradotso
turboquant-pytorch
PyTorch implementation of TurboQuant for LLM KV cache compression using two-stage vector quantization (random rotation + Lloyd-Max + QJL residual correction).
Why this skill is useful
Adds a complete PyTorch implementation of TurboQuant for LLM KV cache compression, including executable scripts for quantization and validation that are not widely available.
What it needs
Requires scipy, torch installed locally. About 6k tokens when loaded. Last updated 2026-07-09. 70 stars on the source repository.
What this skill does
TurboQuant PyTorch Skill by ara.so — Daily 2026 Skills collection. From-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) for compressing LLM KV caches. Achieves 5x compression at 3-bit with 99.5% attention fidelity via two-stage vector quantization. What It Does TurboQuant compresses LLM key-value caches to 2–4 bits per coordinate: Stage 1: Random orthogonal rotation + Lloyd-Max scalar quantization (MSE-optimal) Stage 2: QJL residual correction — 1-bit sign projection that makes inner product estimates unbiased Result: attention scores remain accurate even when individual vectors look quite different from originals. The algorithm preserves inner products, not vector fidelity. Compression ratios at 8K context on Qwen2.5-3B (289 MB FP16 baseline): 4-bit → 76 MB (3.8x) 3-bit → 58 MB (5.0x) ← practical sweet spot 2-bit → 40 MB (7.3x) Installation requirements.txt includes: torch>=2.0 scipy (Lloyd-Max codebook computation) transformers, accelerate, bitsandbytes (only for real model validation) Project Structure Key Commands Core API Lloyd-Max Codebook Stage 1: MSE Quantization (TurboQuantMSE) Stage 2: Unbiased Inner Product Estimation (TurboQuantProd) KV Cache Wrapper (TurboQuantKVCache) Production Compressors (for real model tensors) Common Patterns Pattern 1: Compress a Full Model's KV Cache Pattern 2: Validate Compression Quality Pattern 3: Needle-in-Haystack Retrieval Test Pattern 4: Compute Memory Savings Algorithm Details Why Random Rotation? Rotating by a random orthogonal matrix R maps unit vectors to a space where each coordinate follows N(0, 1/d). This makes coordinates nearly independent with known distribution — enabling optimal per-coordinate scalar quantization (Lloyd-Max). Why QJL for Keys but Not Values? Keys: Used in dot products with queries. Bias in inner product estimates directly corrupts attention weights. QJL correction is essential. Values: Used in weighted sums after softmax. Small per-vector MSE errors average out. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/turboquant-pytorch