Agent skill · vasilyu1983
ai-scaling-laws
Sizes models and token budgets using Kaplan/Chinchilla scaling laws. Use when reasoning about compute-optimal N and D, tokens-per-parameter ratios, or over-training tradeoffs.
What it needs
About 9k tokens when loaded.
What this skill does
AI Scaling Laws — Compute-Optimal Sizing Skill Functional reference for pre-training researchers and engineers who need to reason cold about compute, token, and parameter tradeoffs. Covers Kaplan et al. (2020), Chinchilla / Hoffmann et al. (2022), GPT-3 sizing, over-training for inference efficiency, and the mechanics of budget allocation for a from-scratch run. This is a standard interview probe. Know the key ratios and be ready to work through a concrete sizing calculation without a lookup. Quick Reference Concept Formula / Heuristic Notes --------- --------------------- ------- Compute budget C ≈ 6 N D N = non-embedding params, D = training tokens; approximate, constant ≈6 accounts for forward + backward Chinchilla-optimal ratio D ≈ 20 × N From Hoffmann et al. 2022; holds compute constant Kaplan (2020) ratio D ≈ 1.7–2 × N (roughly) Pre-Chinchilla; model-heavy. Difference from Chinchilla is methodological (FLOP counting, warmup, optimizer tuning), not simply "wrong" — see post-Chinchilla ref Optimal N given C N ≈ (C / 120)^0.5 Approximate; from Chinchilla Table A3 Optimal D given C D ≈ (C / 0.3)^0.5 Paired with above; verify against Hoffmann et al. Table A3 numbers Over-training (Llama-style) D ≫ 20 × N Trades higher training loss for cheaper inference; standard for deployed open models. Llama 3 8B: 15T tokens ≈ 1,875 tok/param (dense example). Llama 4 (2025) is the current MoE example — apply the ratio to activated, not total, params GPT-3 (175B) training tokens ~300B tokens 175B params × ~1.7 tok/param (Kaplan-era; undercooked by Chinchilla standard) Chinchilla (70B) training tokens ~1.4T tokens 70B × 20; compute-matched to GPT-3 but smaller and more accurate GPT-2 (124M) repro budget ≈1–3 B tokens minimum See worked example below Published fit constants α≈0.336, β≈0.283 Corrected by Besiroglu et al. 2024 to α≈0.35, β≈0.37 — original fit had convergence/rounding errors Key distinction: Chinchilla-optimal minimizes validation loss for a given compute budget. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills vasilyu1983/ai-scaling-laws