Agent skill · research science · aradotso
nanochat-llm-training
Train your own GPT-2 level LLM for under $100 using nanochat, Karpathy's minimal hackable harness covering tokenization, pretraining, finetuning, evaluation, inference, and chat UI.
Why this skill is useful
Adds executable scripts for training and fine-tuning LLMs using nanochat, which are not available in public documentation.
What it needs
Requires uv installed locally. About 5k tokens when loaded. Last updated 2026-07-09. 70 stars on the source repository.
What this skill does
nanochat LLM Training Skill by ara.so — Daily 2026 Skills collection. nanochat is Karpathy's minimal, hackable harness for training LLMs end-to-end on a single GPU node. It covers tokenization, pretraining, SFT finetuning, RL, evaluation (DCLM CORE score), inference with KV cache, and a ChatGPT-like web UI. A single complexity dial (--depth) auto-configures all other hyperparameters (width, heads, LR, training horizon, weight decay) for compute-optimal training. You can reproduce GPT-2 capability (~$43,000 in 2019) for ~$48 on an 8×H100 node (~2 hours). Installation nanochat uses uv for dependency management: Key Commands Full GPT-2 Speedrun (8×H100 node, ~2–3 hours, ~$48) Pretraining (distributed) Pretraining (single GPU) Quick Research Iteration (~5 min, GPT-1 scale) CPU / Apple Silicon (tiny model, ~minutes) Serve Chat UI CLI Chat Scaling Laws / Miniseries The Depth Dial The single most important parameter. Everything else is derived automatically: --depth Approximate model scale Notes ----------- ------------------------ ------- 6–8 Tiny (toy) CPU/MPS feasible 12 GPT-1 size ~5 min on 8×H100, great for research iteration 16 Medium ~15 min on 8×H100 24–26 GPT-2 size ~2 hrs on 8×H100, ~$48 Precision / dtype Configuration nanochat uses explicit dtype management via COMPUTEDTYPE in nanochat/common.py. No torch.amp.autocast. Hardware Default Override ---------- --------- --------- CUDA SM 80+ (A100, H100) bfloat16 NANOCHATDTYPE=float32 CUDA SM < 80 (V100, T4) float32 NANOCHATDTYPE=float16 CPU / MPS float32 — How it works: Weights stored in fp32 (optimizer precision), custom Linear casts to COMPUTEDTYPE in forward pass, embeddings stored directly in COMPUTEDTYPE to save memory. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/nanochat-llm-training