Agent skill · software engineering · davila7
awq-quantization
Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.
Why this skill is useful
Adds executable scripts for 4-bit quantization and deployment of large models, significantly enhancing inference speed and memory optimization.
What it needs
Requires autoawq, torch, transformers installed locally. About 4k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
AWQ (Activation-aware Weight Quantization) 4-bit quantization that preserves salient weights based on activation patterns, achieving 3x speedup with minimal accuracy loss. When to use AWQ Use AWQ when: Need 4-bit quantization with <5% accuracy loss Deploying instruction-tuned or chat models (AWQ generalizes better) Want ~2.5-3x inference speedup over FP16 Using vLLM for production serving Have Ampere+ GPUs (A100, H100, RTX 40xx) for Marlin kernel support Use GPTQ instead when: Need maximum ecosystem compatibility (more tools support GPTQ) Working with ExLlamaV2 backend specifically Have older GPUs without Marlin support Use bitsandbytes instead when: Need zero calibration overhead (quantize on-the-fly) Want to fine-tune with QLoRA Prefer simpler integration Quick start Installation Requirements: Python 3.8+, CUDA 11.8+, Compute Capability 7.5+ Load pre-quantized model Quantize your own model Timing: ~10-15 min for 7B, ~1 hour for 70B models. AWQ vs GPTQ vs bitsandbytes Feature AWQ GPTQ bitsandbytes --------- ----- ------ -------------- Speedup (4-bit) ~2.5-3x ~2x ~1.5x Accuracy loss <5% ~5-10% ~5-15% Calibration Minimal (128-1K tokens) More extensive None Overfitting risk Low Higher N/A Best for Production inference GPU inference Easy integration vLLM support Native Yes Limited Key insight: AWQ assumes not all weights are equally important. It protects ~1% of salient weights identified by activation patterns, reducing quantization error without mixed-precision overhead. Kernel backends GEMM (default, batch inference) GEMV (single-token generation) Limitation: Only batch size 1, not good for large context. Marlin (Ampere+ GPUs) Requirements: Compute Capability 8.0+ (A100, H100, RTX 40xx) ExLlamaV2 (AMD compatible) HuggingFace Transformers integration Direct loading Fused modules (recommended) Note: Fused modules cannot combine with FlashAttention2. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/optimization-awq