Agent skill · software engineering · davila7
gptq
Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.
Why this skill is useful
Adds executable scripts for quantizing large models to 4-bit with minimal accuracy loss, enabling deployment on consumer GPUs.
What it needs
Requires auto-gptq, optimum, peft, transformers installed locally. About 6k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
GPTQ (Generative Pre-trained Transformer Quantization) Post-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization. When to use GPTQ Use GPTQ when: Need to fit large models (70B+) on limited GPU memory Want 4× memory reduction with <2% accuracy loss Deploying on consumer GPUs (RTX 4090, 3090) Need faster inference (3-4× speedup vs FP16) Use AWQ instead when: Need slightly better accuracy (<1% loss) Have newer GPUs (Ampere, Ada) Want Marlin kernel support (2× faster on some GPUs) Use bitsandbytes instead when: Need simple integration with transformers Want 8-bit quantization (less compression, better quality) Don't need pre-quantized model files Quick start Installation Load pre-quantized model Quantize your own model Group-wise quantization How GPTQ works: 1. Group weights: Divide each weight matrix into groups (typically 128 elements) 2. Quantize per-group: Each group has its own scale/zero-point 3. Minimize error: Uses Hessian information to minimize quantization error 4. Result: 4-bit weights with near-FP16 accuracy Group size trade-off: Group Size Model Size Accuracy Speed Recommendation ------------ ------------ ---------- ------- ---------------- -1 (per-column) Smallest Best Slowest Research only 32 Smaller Better Slower High accuracy needed 128 Medium Good Fast Recommended default 256 Larger Lower Faster Speed critical 1024 Largest Lowest Fastest Not recommended Example: Quantization configurations Standard 4-bit (recommended) Performance: Memory: 4× reduction (70B model: 140GB → 35GB) Accuracy: ~1.5% perplexity increase Speed: 3-4× faster than FP16 High accuracy (3-bit with larger groups) Trade-off: Memory: 5× reduction Accuracy: ~3% perplexity increase Speed: 5× faster (but less accurate) Maximum accuracy (4-bit with small groups) Trade-off: Memory: 3.5× reduction (slightly larger) Accuracy: ~0.8% perplexity increase (best) Speed: 2-3× faster (kernel overhead) Kernel backends ExLlamaV2 (default, …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/optimization-gptq