Agent skill · research science · nousresearch
trl-fine-tuning
TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF.
Why this skill is useful
Adds executable scripts for fine-tuning language models with human preferences using TRL, which are not commonly found in public documentation.
What it needs
Requires accelerate, datasets, peft, torch, transformers, trl installed locally. About 6k tokens when loaded. Last updated 2026-08-07. 226,679 stars on the source repository.
What this skill does
TRL - Transformer Reinforcement Learning Quick start TRL provides post-training methods for aligning language models with human preferences. Installation: Supervised Fine-Tuning (instruction tuning): DPO (align with preferences): Common workflows Workflow 1: Full RLHF pipeline (SFT → Reward Model → RLOO) Complete pipeline from base model to human-aligned model. Note (TRL 1.x): PPO has been removed from TRL — PPOTrainer, PPOConfig, and python -m trl.scripts.ppo no longer exist. Use an online-RL trainer TRL still ships: RLOO (RLOOTrainer / trl rloo) is the closest drop-in for a reward-model-driven RLHF pipeline, and GRPO (GRPOTrainer / trl grpo, see Workflow 3) is the memory-efficient alternative. The step below uses RLOO. Copy this checklist: Step 1: Supervised fine-tuning Train base model on instruction-following data: Step 2: Train reward model Train model to predict human preferences: Step 3: RLOO reinforcement learning Optimize policy using the reward model. PPO was removed in TRL 1.x; use the RLOO CLI (trl rloo) with the trained reward model passed via --rewardmodelnameorpath: Equivalent Python (RLOOTrainer / RLOOConfig): Step 4: Evaluate Workflow 2: Simple preference alignment with DPO Align model with preferences without reward model. Copy this checklist: Step 1: Prepare preference dataset Dataset format: Load dataset: Step 2: Configure DPO Step 3: Train with DPOTrainer CLI alternative: Workflow 3: Memory-efficient online RL with GRPO Train with reinforcement learning using minimal memory. For in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see references/grpo-training.md. A production-ready training script is in templates/basicgrpotraining.py. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nousresearch/trl-fine-tuning