Agent skill · research science · davila7
fine-tuning-with-trl
Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.
Why this skill is useful
Adds executable scripts and detailed workflows for fine-tuning LLMs with reinforcement learning, which are not commonly found in public documentation.
What it needs
Requires accelerate, datasets, peft, torch, transformers, trl installed locally. About 5k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
TRL - Transformer Reinforcement Learning Quick start TRL provides post-training methods for aligning language models with human preferences. Installation: Supervised Fine-Tuning (instruction tuning): DPO (align with preferences): Common workflows Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO) Complete pipeline from base model to human-aligned model. Copy this checklist: Step 1: Supervised fine-tuning Train base model on instruction-following data: Step 2: Train reward model Train model to predict human preferences: Step 3: PPO reinforcement learning Optimize policy using reward model: Step 4: Evaluate Workflow 2: Simple preference alignment with DPO Align model with preferences without reward model. Copy this checklist: Step 1: Prepare preference dataset Dataset format: Load dataset: Step 2: Configure DPO Step 3: Train with DPOTrainer CLI alternative: Workflow 3: Memory-efficient online RL with GRPO Train with reinforcement learning using minimal memory. Copy this checklist: Step 1: Define reward function Or use a reward model: Step 2: Configure GRPO Step 3: Train with GRPOTrainer CLI: When to use vs alternatives Use TRL when: Need to align model with human preferences Have preference data (chosen/rejected pairs) Want to use reinforcement learning (PPO, GRPO) Need reward model training Doing RLHF (full pipeline) Method selection: SFT: Have prompt-completion pairs, want basic instruction following DPO: Have preferences, want simple alignment (no reward model needed) PPO: Have reward model, need maximum control over RL GRPO: Memory-constrained, want online RL Reward Model: Building RLHF pipeline, need to score generations Use alternatives instead: HuggingFace Trainer: Basic fine-tuning without RL Axolotl: YAML-based training configuration LitGPT: Educational, minimal fine-tuning Unsloth: Fast LoRA training Common issues Issue: OOM during DPO training Reduce batch size and sequence length: Or use gradient checkpointing: Issue: Poor alignment quality Tune bet …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/post-training-trl-fine-tuning