Agent skill · software engineering · davila7
quantizing-models-bitsandbytes
Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.
Why this skill is useful
Adds executable scripts and detailed workflows for quantizing models, enabling significant memory savings and efficient inference not commonly known.
What it needs
Requires accelerate, bitsandbytes, torch, transformers installed locally. About 5k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
bitsandbytes - LLM Quantization Quick start bitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with <1% accuracy loss. Installation: 8-bit quantization (50% memory reduction): 4-bit quantization (75% memory reduction): Common workflows Workflow 1: Load large model in limited GPU memory Copy this checklist: Step 1: Calculate memory requirements Estimate model memory: Step 2: Choose quantization level GPU VRAM Model Size Recommended ---------- ------------ ------------- 8 GB 3B 4-bit 12 GB 7B 4-bit 16 GB 7B 8-bit or 4-bit 24 GB 13B 8-bit or 70B 4-bit 40+ GB 70B 8-bit Step 3: Configure quantization For 8-bit (better accuracy): For 4-bit (maximum memory savings): Step 4: Load and verify model Workflow 2: Fine-tune with QLoRA (4-bit training) QLoRA enables fine-tuning large models on consumer GPUs. Copy this checklist: Step 1: Install dependencies Step 2: Configure 4-bit base model Step 3: Add LoRA adapters Step 4: Train with standard Trainer Workflow 3: 8-bit optimizer for memory-efficient training Use 8-bit Adam/AdamW to reduce optimizer memory by 75%. Step 1: Replace standard optimizer Manual optimizer usage: Step 2: Configure training Compare memory: Step 3: Monitor memory savings When to use vs alternatives Use bitsandbytes when: GPU memory limited (need to fit larger model) Training with QLoRA (fine-tune 70B on single GPU) Inference only (50-75% memory reduction) Using HuggingFace Transformers Acceptable 0-2% accuracy degradation Use alternatives instead: GPTQ/AWQ: Production serving (faster inference than bitsandbytes) GGUF: CPU inference (llama.cpp) FP8: H100 GPUs (hardware FP8 faster) Full precision: Accuracy critical, memory not constrained Common issues Issue: CUDA error during loading Install matching CUDA version: Issue: Model loading slow Use CPU offload for large models: Issue: Lower accuracy than expected Try 8-bit instead of 4-bit: Or use NF4 with double quantization: Issue: OOM even with 4-bit Enable CPU offload: Advanced topics QLoRA trainin …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/optimization-bitsandbytes