Agent skill · research science · nousresearch
simpo
Reference-free preference alignment, simpler than DPO.
Why this skill is useful
Adds executable scripts and specific training configurations for preference optimization that are not commonly found in public documentation.
What it needs
Requires accelerate, datasets, torch, transformers, trl installed locally. About 2k tokens when loaded. Last updated 2026-08-07. 226,679 stars on the source repository.
What this skill does
SimPO - Simple Preference Optimization Quick start SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model. Installation: Training (Mistral 7B): Common workflows Workflow 1: Train from base model (Mistral 7B) Config (mistral-7b-base-simpo.yaml): Launch training: Workflow 2: Fine-tune instruct model (Llama 3 8B) Config (llama3-8b-instruct-simpo.yaml): Launch: Workflow 3: Reasoning-intensive tasks (lower LR) For math/code tasks: When to use vs alternatives Use SimPO when: Want simpler training than DPO (no reference model) Have preference data (chosen/rejected pairs) Need better performance than DPO Limited compute resources Single-node training sufficient Algorithm selection: SimPO: Simplest, best performance, no reference model DPO: Need reference model baseline, more conservative PPO: Maximum control, need reward model, complex setup GRPO: Memory-efficient RL, no critic Use alternatives instead: OpenRLHF: Multi-node distributed training, PPO/GRPO TRL: Need multiple methods in one framework DPO: Established baseline comparison Common issues Issue: Loss divergence Reduce learning rate: Reduce beta: Issue: Model forgets capabilities Add SFT regularization: Issue: Poor preference separation Increase beta and margin: Issue: OOM during training Reduce batch size: Enable gradient checkpointing: Advanced topics Loss functions: See references/loss-functions.md for sigmoid vs hinge loss, mathematical formulations, and when to use each. Hyperparameter tuning: See references/hyperparameters.md for beta, gamma, learning rate selection guide, and model-size-specific recommendations. Dataset preparation: See references/datasets.md for preference data formats, quality filtering, and custom dataset creation. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nousresearch/simpo