Agent skill · research science · davila7
openrlhf-training
High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.
Why this skill is useful
Adds executable scripts for distributed RLHF training that leverage Ray and vLLM, which are not commonly found in public documentation.
What it needs
Requires deepspeed, docker, ray, torch, transformers, vllm installed locally. About 4k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
OpenRLHF - High-Performance RLHF Training Quick start OpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration. Installation: PPO Training (Hybrid Engine): GRPO Training (Group Normalized Policy Optimization): Common workflows Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO) Step 1: Train reward model (DPO): Step 2: PPO training: Workflow 2: GRPO training (no critic model needed) Memory-efficient alternative to PPO: Key GRPO parameters: --advantageestimator groupnorm - Enables GRPO --useklloss - KL loss from GRPO paper --klestimator k3 - Loss function (k2 ≈ k1) --noadvantagestdnorm - Disables std normalization Workflow 3: DPO training (preference optimization) Simpler alternative without reward model: When to use vs alternatives Use OpenRLHF when: Training large models (7B-70B+) with RL Need vLLM inference acceleration Want distributed architecture with Ray Have multi-node GPU cluster Need PPO/GRPO/RLOO/DPO in one framework Algorithm selection: PPO: Maximum control, best for complex rewards GRPO: Memory-efficient, no critic needed RLOO: Modified PPO with per-token KL REINFORCE++: More stable than GRPO, faster than PPO DPO: Simplest, no reward model needed Use alternatives instead: TRL: Single-node training, simpler API veRL: ByteDance's framework for 671B models DeepSpeedChat: Integrated with DeepSpeed ecosystem Common issues Issue: GPU OOM with large models Disable model colocation: Issue: DeepSpeed GPU index out of range Set environment variable: Issue: Training instability Use Hybrid Engine instead of async: Adjust KL coefficient: Issue: Slow generation during PPO Enable vLLM acceleration: Advanced topics Hybrid Engine GPU sharing: See references/hybrid-engine.md for vLLM sleep mode, DeepSpeed sleep mode, and optimal node allocation. Algorithm comparison: See references/algorithm-comparison.md for PPO vs GRPO vs RLOO vs REINFORCE++ benchmarks and hyperparameters. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/post-training-openrlhf