Agent skill · research science · davila7
nanogpt
Educational GPT implementation in ~300 lines. Reproduces GPT-2 (124M) on OpenWebText. Clean, hackable code for learning transformers. By Andrej Karpathy. Perfect for understanding GPT architecture from scratch. Train on Shakespeare (CPU) or OpenWebText (multi-GPU).
Why this skill is useful
Provides executable scripts for training and fine-tuning GPT models, which extend the AI's capabilities in natural language processing.
What it needs
Requires datasets, tiktoken, torch, transformers, wandb installed locally. About 4k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
nanoGPT - Minimalist GPT Training Quick start nanoGPT is a simplified GPT implementation designed for learning and experimentation. Installation: Train on Shakespeare (CPU-friendly): Output: Common workflows Workflow 1: Character-level Shakespeare Complete training pipeline: Config (config/trainshakespearechar.py): Training time: ~5 minutes (CPU), ~1 minute (GPU) Workflow 2: Reproduce GPT-2 (124M) Multi-GPU training on OpenWebText: Config (config/traingpt2.py): Training time: ~4 days (8× A100) Workflow 3: Fine-tune pretrained GPT-2 Start from OpenAI checkpoint: Example config (config/finetuneshakespeare.py): Workflow 4: Custom dataset Train on your own text: Train: When to use vs alternatives Use nanoGPT when: Learning how GPT works Experimenting with transformer variants Teaching/education purposes Quick prototyping Limited compute (can run on CPU) Simplicity advantages: ~300 lines: Entire model in model.py ~300 lines: Training loop in train.py Hackable: Easy to modify No abstractions: Pure PyTorch Use alternatives instead: HuggingFace Transformers: Production use, many models Megatron-LM: Large-scale distributed training LitGPT: More architectures, production-ready PyTorch Lightning: Need high-level framework Common issues Issue: CUDA out of memory Reduce batch size or context length: Issue: Training too slow Enable compilation (PyTorch 2.0+): Use mixed precision: Issue: Poor generation quality Train longer: Lower temperature: Issue: Can't load GPT-2 weights Install transformers: Check model name: Advanced topics Model architecture: See references/architecture.md for GPT block structure, multi-head attention, and MLP layers explained simply. Training loop: See references/training.md for learning rate schedule, gradient accumulation, and distributed data parallel setup. Data preparation: See references/data.md for tokenization strategies (character-level vs BPE) and binary format details. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/model-architecture-nanogpt