Agent skill · research science · aradotso
how-to-train-your-gpt
how-to-train-your-gpt — an agent skill for coding agents.
Why this skill is useful
Includes detailed executable scripts and domain-specific knowledge for building a LLaMA-style GPT model from scratch in PyTorch.
What it needs
Requires datasets, matplotlib, numpy, tiktoken, torch installed locally. About 10k tokens when loaded. Last updated 2026-07-09. 70 stars on the source repository.
What this skill does
bash git clone https://github.com/raiyanyahya/how-to-train-your-gpt.git cd how-to-train-your-gpt python -m venv gptenv source gptenv/bin/activate # Mac/Linux gptenv\Scripts\activate # Windows pip install torch tiktoken datasets numpy matplotlib bash python -c "import torch; print(f'CUDA: {torch.cuda.isavailable()}, Device: {torch.cuda.getdevicename(0) if torch.cuda.isavailable() else \"CPU\"}')" chapters/ ├── 00overview.md # Big picture: what is a GPT? ├── 01setup.md # Environment, PyTorch basics ├── 02tokenization.md # BPE walkthrough ├── 03embeddings.md # Token embeddings, semantic space ├── 04positionalencoding.md # RoPE math + numeric example ├── 05attention.md # ⭐ Core: Q,K,V, scaling, causal mask (713 lines) ├── 06transformerblock.md # RMSNorm, SwiGLU, residuals ├── 07gptmodel.md # Full 124M parameter model ├── 08training.md # AdamW, cosine warmup, mixed precision ├── 09inference.md # KV cache, sampling strategies ├── 10fullscript.md # Runnable main.py (copy this to run) └── 11glossary.md # Architecture provenance table python import tiktoken Load GPT-4 BPE tokenizer (same family as used by OpenAI) enc = tiktoken.getencoding("cl100kbase") Encode text to token IDs tokens = enc.encode("Hello, how are you?") → [9906, 11, 1268, 527, 499, 30] Decode back to text text = enc.decode(tokens) → "Hello, how are you?" vocabsize = enc.nvocab # 100,277 for cl100kbase python import torch import torch.nn as nn Embedding table: maps token ID → dense vector vocabsize tokens, each represented as dmodel floats embedding = nn.Embedding(vocabsize, dmodel) tokenids shape: (batchsize, seqlen) tokenids = torch.tensor([[9906, 11, 1268]]) # (1, 3) x shape: (batchsize, seqlen, dmodel) x = embedding(tokenids) # (1, 3, 768) python def precomputeropefreqs(dmodel: int, maxseqlen: int, theta: float = 10000.0): """ Precompute rotation frequencies for RoPE. LLaMA rotates Q and K vectors by position-dependent angles so attention scores encode relative distance automatically. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/how-to-train-your-gpt