Agent skill · research science · davila7
simpo-training
Simple Preference Optimization for LLM alignment. Reference-free alternative to DPO with better performance (+6.4 points on AlpacaEval 2.0). No reference model needed, more efficient than DPO. Use for preference alignment when want simpler, faster training than DPO/PPO.
Why this skill is useful
Includes executable scripts and specific training configurations for preference optimization that enhance LLM alignment capabilities.
What it needs
Requires accelerate, datasets, torch, transformers, trl installed locally. About 3k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
SimPO - Simple Preference Optimization Quick start SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model. Installation: Training (Mistral 7B): Common workflows Workflow 1: Train from base model (Mistral 7B) Config (mistral-7b-base-simpo.yaml): Launch training: Workflow 2: Fine-tune instruct model (Llama 3 8B) Config (llama3-8b-instruct-simpo.yaml): Launch: Workflow 3: Reasoning-intensive tasks (lower LR) For math/code tasks: When to use vs alternatives Use SimPO when: Want simpler training than DPO (no reference model) Have preference data (chosen/rejected pairs) Need better performance than DPO Limited compute resources Single-node training sufficient Algorithm selection: SimPO: Simplest, best performance, no reference model DPO: Need reference model baseline, more conservative PPO: Maximum control, need reward model, complex setup GRPO: Memory-efficient RL, no critic Use alternatives instead: OpenRLHF: Multi-node distributed training, PPO/GRPO TRL: Need multiple methods in one framework DPO: Established baseline comparison Common issues Issue: Loss divergence Reduce learning rate: Reduce beta: Issue: Model forgets capabilities Add SFT regularization: Issue: Poor preference separation Increase beta and margin: Issue: OOM during training Reduce batch size: Enable gradient checkpointing: Advanced topics Loss functions: See references/loss-functions.md for sigmoid vs hinge loss, mathematical formulations, and when to use each. Hyperparameter tuning: See references/hyperparameters.md for beta, gamma, learning rate selection guide, and model-size-specific recommendations. Dataset preparation: See references/datasets.md for preference data formats, quality filtering, and custom dataset creation. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/post-training-simpo