Agent skill · vasilyu1983
ai-pretraining
Builds a transformer/GPT and BPE tokenizer from scratch. Use when implementing autograd, self-attention, a nanoGPT-style pretraining loop, or a byte-level tokenizer.
What it needs
About 9k tokens when loaded.
What this skill does
Pretraining From Scratch Domain: building a transformer/GPT and a BPE tokenizer from first principles — the from-first-principles training-layer competency. Does NOT cover applications-layer fine-tuning, RLHF, or inference optimization; those belong to sibling skills. Canonical teachers: Karpathy "Neural Networks: Zero to Hero" (micrograd → makemore → "Let's build GPT" → "Let's build the GPT Tokenizer" → "Let's reproduce GPT-2"), Karpathy nanochat (full-stack from-scratch successor to nanoGPT, 2025), Raschka "Build a Large Language Model From Scratch", nanoGPT, minbpe, "Attention Is All You Need". GPT-2 is the pedagogical spine here — the right thing to build first. The 2026 from-scratch baseline then swaps four components onto that spine (RoPE, RMSNorm, SwiGLU, GQA) and runs attention through FlashAttention/SDPA; see Modern Architecture Deltas. ASCII Flow When to Use This Skill Activate when the user asks about: Implementing autograd / backprop from scratch (micrograd-style) Building makemore (bigram, MLP, WaveNet-style character LMs) Implementing self-attention, multi-head attention, causal masking Building the transformer block (pre-norm vs post-norm, residual, FFN) Stacking blocks into a GPT with an LM head and weight tying Writing the pretraining loop: cross-entropy, bf16 mixed precision, gradient accumulation, gradient checkpointing, cosine LR schedule with warmup, model checkpointing Building a BPE tokenizer from scratch: byte-level, merge algorithm, vocab construction, encode/decode (minbpe-style) Reproducing GPT-2 (124M) from scratch end-to-end (nanoGPT path) Implementing temperature scaling and top-k sampling for text generation Scope Boundaries (Use These Skills for Depth) LLM lifecycle, fine-tuning, provider selection, deployment -> ai-llm Multi-GPU training: DDP, FSDP, tensor/pipeline parallelism -> ai-distributed-training Token/param budget, Chinchilla scaling, compute-optimal runs -> ai-scaling-laws Dataset curation, deduplication, quality filtering f …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills vasilyu1983/ai-pretraining