Agent skill · research science · wshobson
preference-optimization
Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.
Why this skill is useful
Provides specific method selection guidance and hyperparameter configurations for preference optimization that the AI wouldn't reliably generate on its own.
What it needs
About 5k tokens when loaded. Last updated 2026-08-05. 38,570 stars on the source repository.
What this skill does
Preference Optimization This skill assumes finetuning-method-selection already routed here because the data shape is preference pairs or unpaired thumbs-up/down feedback, not demonstrations (that's lora-qlora-recipes) or a verifiable reward signal (that's grpo-rlvr-training). What follows is method selection among the DPO family, the evidence for how much that selection actually matters, the production training pattern, and how to build the pairs in the first place. Input: a routing decision (preference optimization) plus preference pairs or unpaired feedback, usually from an SFT checkpoint. Output format: a validated method choice plus a config — the kwarg values in references/method-configs.md, not free-form advice — that llm-finetuning-training-engineer consumes directly. Method Selection Data shape Method Key parameters --- --- --- Preference pairs, default case DPO β=0.1, LR 5e-7–1e-6, 1–2 epochs Memory-bound or no SFT checkpoint ORPO reference-free, fused SFT+preference in one loss Unpaired thumbs-up/down KTO binary label per example, no pairing needed Length bias observed, sweep budget available SimPO reference-free; see sweep grid below DPO is the safe default. Use β=0.1 and a learning rate of 5e-7 to 1e-6 for 1–2 epochs. This LR is lower than the SFT LR that produced the checkpoint being aligned — porting an SFT- scale LR into a DPO run is the most common misconfiguration here, not an edge case. ORPO routes in when memory is the constraint, or when there's no separate SFT checkpoint to start from — it's reference-free and fuses the SFT and preference objectives into one loss, skipping the separate SFT pass and the reference-model memory cost DPO carries. KTO routes in when feedback is unpaired binary signal (thumbs-up/down) rather than matched preference pairs — don't force unpaired feedback into synthetic pairs to use DPO instead. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills wshobson/preference-optimization