Agent skill · alirezarezvani
spinning-up-deep-rl
Knowledge base from \"Spinning Up in Deep RL\" by Joshua Achiam (OpenAI, MIT-licensed). Use when applying Achiam's frameworks for RL fundamentals and MDPs, the model-free algorithm taxonomy, policy gradient derivations, the six reference algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC), debugging silently-failing RL code, or running rigorous multi-seed RL experiments.
What it needs
About 6k tokens when loaded.
What this skill does
Spinning Up in Deep RL Author: Joshua Achiam (OpenAI) Source: spinningup.readthedocs.io, MIT Chapters: 20 Generated: 2026-08-25 How to Use This Skill No argument — load the core frameworks below A topic — ask about advantage function, target networks, entropy regularization; I resolve it through the Topic Index and read that chapter file chNN — I load that chapter's summary "what chapters do you have?" — the full index When you ask about something not in Core Frameworks, I read the relevant chapter file before answering rather than guessing from the index. --- Core Frameworks & Mental Models The RL problem pi = argmaxpi J(pi), where J(pi) = E{tau~pi}[R(tau)]. Every algorithm approximates this; where it substitutes a different objective (a Bellman residual, a surrogate), that substitution is the source of its failure modes. Four value functions — V^pi, Q^pi, V, Q — all obey Bellman self-consistency, and a(s) = argmaxa Q(s,a) is why Q-learning is a viable family at all. Advantage A^pi(s,a) = Q^pi(s,a) - V^pi(s) is the relative-quality signal policy gradients run on. (ch07) The two branching questions Place any algorithm by asking: does it have or learn a model, and what does it learn (policy, Q-function, value function, model). That generates the whole landscape. (ch08) Policy optimization vs Q-learning — the central trade-off Policy optimization is principled: you directly optimize the thing you want. Stable and reliable. On-policy, so it cannot reuse data, so it is sample-hungry. Q-learning only indirectly optimizes performance, by training Qtheta to satisfy a self-consistency equation. Many failure modes, so less stable. But substantially more sample efficient when it works, because it reuses everything. Satisfying the Bellman equations well carries no guarantee of good policy performance. The two are not exclusive — DDPG and SAC live between them deliberately. (ch08) The policy gradient template grad J = E[ sumt grad log pitheta(at st) Phit ]. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills alirezarezvani/spinning-up-deep-rl