Agent skill · research science · wshobson
lora-qlora-recipes
Configure LoRA and QLoRA supervised fine-tuning with current best-practice hyperparameters. Use when writing or reviewing a LoRA/QLoRA training configuration, choosing rank/alpha/target modules, or deciding between LoRA, QLoRA, and full fine-tuning.
Why this skill is useful
Provides specific hyperparameter configurations and best practices for LoRA and QLoRA fine-tuning that the AI wouldn't reliably generate on its own.
What it needs
About 4k tokens when loaded. Last updated 2026-08-05. 38,570 stars on the source repository.
What this skill does
LoRA & QLoRA Recipes This skill assumes the routing decision already happened — finetuning-method-selection should have already pointed here because the data shape is demonstrations (SFT), not preference pairs or a verifiable reward signal. What follows is the current best-practice recipe for configuring the adapter itself: which modules to target, how to size rank and alpha, what learning rate to use, and when QLoRA buys real headroom versus when it just adds risk. Dataset preparation and quality checks are a separate concern — see dataset-curation. Input: a routing decision (SFT via LoRA/ QLoRA) plus a target size class. Output format: a validated adapter config — the kwarg values below, not free-form advice — that llm-finetuning-training-engineer consumes directly when it generates a runnable script. The Reference Recipe The reference recipe is "LoRA Without Regret" (Thinking Machines/Schulman, 2025-09), now the settled convention for LoRA/QLoRA SFT. Target Modules Target all-linear modules, not just attention: The MLP layers (gateproj, upproj, downproj) matter most — attention-only targeting was the older, weaker convention. Dropping modules to save memory is a Failure Mode below, not a valid optimization. Alpha and Learning Rate loraalpha = 2 r is the settled convention (NeurIPS 2025 "intruder dimensions" result). Don't hand-tune alpha independently of rank — derive it from rank every time. LoRA learning rate ≈ 10x the equivalent full-fine-tune LR. For QLoRA specifically, 2e-4 is the standard starting point. Full hyperparameter tables and worked examples: references/hyperparameters.md. Rank by Task Rank is task-shaped, not a single global default: Task Rank --- --- RL (GRPO/RLVR adapters) 1–32 General default 16–32 SFT at scale up to ~256 Higher rank isn't automatically better — it raises capacity to memorize as fast as it raises capacity to generalize. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills wshobson/lora-qlora-recipes