Agent skill · software engineering · zechenzhangagi
gguf-quantization
GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.
Why this skill is useful
Adds executable scripts for quantizing models and converting them to GGUF format, which are not commonly found in public documentation.
What it needs
Requires llama-cpp-python installed locally. About 5k tokens when loaded. Last updated 2026-06-16. 11,472 stars on the source repository.
What this skill does
GGUF - Quantization Format for llama.cpp The GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options. When to use GGUF Use GGUF when: Deploying on consumer hardware (laptops, desktops) Running on Apple Silicon (M1/M2/M3) with Metal acceleration Need CPU inference without GPU requirements Want flexible quantization (Q2K to Q80) Using local AI tools (LM Studio, Ollama, text-generation-webui) Key advantages: Universal hardware: CPU, Apple Silicon, NVIDIA, AMD support No Python runtime: Pure C/C++ inference Flexible quantization: 2-8 bit with various methods (K-quants) Ecosystem support: LM Studio, Ollama, koboldcpp, and more imatrix: Importance matrix for better low-bit quality Use alternatives instead: AWQ/GPTQ: Maximum accuracy with calibration on NVIDIA GPUs HQQ: Fast calibration-free quantization for HuggingFace bitsandbytes: Simple integration with transformers library TensorRT-LLM: Production NVIDIA deployment with maximum speed Quick start Installation Convert model to GGUF Quantize model Run inference Quantization types K-quant methods (recommended) Type Bits Size (7B) Quality Use Case ------ ------ ----------- --------- ---------- Q2K 2.5 ~2.8 GB Low Extreme compression Q3KS 3.0 ~3.0 GB Low-Med Memory constrained Q3KM 3.3 ~3.3 GB Medium Balance Q4KS 4.0 ~3.8 GB Med-High Good balance Q4KM 4.5 ~4.1 GB High Recommended default Q5KS 5.0 ~4.6 GB High Quality focused Q5KM 5.5 ~4.8 GB Very High High quality Q6K 6.0 ~5.5 GB Excellent Near-original Q80 8.0 ~7.2 GB Best Maximum quality Legacy methods Type Description ------ ------------- Q40 4-bit, basic Q41 4-bit with delta Q50 5-bit, basic Q51 5-bit with delta Recommendation: Use K-quant methods (Q4KM, Q5KM) for best quality/size ratio. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills zechenzhangagi/gguf