Agent skill · software engineering · davila7
gguf-quantization
GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.
Why this skill is useful
Provides specific commands and workflows for quantizing models in GGUF format that are not widely known.
What it needs
Requires llama-cpp-python installed locally. About 5k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
GGUF - Quantization Format for llama.cpp The GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options. When to use GGUF Use GGUF when: Deploying on consumer hardware (laptops, desktops) Running on Apple Silicon (M1/M2/M3) with Metal acceleration Need CPU inference without GPU requirements Want flexible quantization (Q2K to Q80) Using local AI tools (LM Studio, Ollama, text-generation-webui) Key advantages: Universal hardware: CPU, Apple Silicon, NVIDIA, AMD support No Python runtime: Pure C/C++ inference Flexible quantization: 2-8 bit with various methods (K-quants) Ecosystem support: LM Studio, Ollama, koboldcpp, and more imatrix: Importance matrix for better low-bit quality Use alternatives instead: AWQ/GPTQ: Maximum accuracy with calibration on NVIDIA GPUs HQQ: Fast calibration-free quantization for HuggingFace bitsandbytes: Simple integration with transformers library TensorRT-LLM: Production NVIDIA deployment with maximum speed Quick start Installation Convert model to GGUF Quantize model Run inference Quantization types K-quant methods (recommended) Type Bits Size (7B) Quality Use Case ------ ------ ----------- --------- ---------- Q2K 2.5 ~2.8 GB Low Extreme compression Q3KS 3.0 ~3.0 GB Low-Med Memory constrained Q3KM 3.3 ~3.3 GB Medium Balance Q4KS 4.0 ~3.8 GB Med-High Good balance Q4KM 4.5 ~4.1 GB High Recommended default Q5KS 5.0 ~4.6 GB High Quality focused Q5KM 5.5 ~4.8 GB Very High High quality Q6K 6.0 ~5.5 GB Excellent Near-original Q80 8.0 ~7.2 GB Best Maximum quality Legacy methods Type Description ------ ------------- Q40 4-bit, basic Q41 4-bit with delta Q50 5-bit, basic Q51 5-bit with delta Recommendation: Use K-quant methods (Q4KM, Q5KM) for best quality/size ratio. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/optimization-gguf