🧠 tiny-gpt

A from-scratch, character-level GPT in pure numpy. Every forward and backward pass is written by hand and commented, so you can read exactly how a transformer learns.

Most "GPT from scratch" tutorials still lean on PyTorch's autograd for the hard part. tiny-gpt doesn't: the only dependency is numpy, and every layer (embeddings, LayerNorm, multi-head causal self-attention, GELU MLP, cross-entropy) has an explicit backward() method. Finite-difference gradient checks in the test suite prove the maths is right.

It trains in about a minute on a laptop CPU on a small corpus of original verses about London bundled in data/london_verses.txt, and ships with a trained checkpoint so you can sample straight away.

📸 Demo

With only ~7.5k characters of training text the model quickly starts to memorise, which is visible as the validation curve turning upwards. Training keeps the best-validation weights (early stopping), and dropout plus weight decay hold off overfitting. The samples pick up the corpus's shape: stanzas, line breaks, capitalised titles, common words like "the", "and", "river". That's about the limit for 111k parameters and a tiny dataset. Point it at a larger text file (--corpus) and it gets much better.

✨ Features

• Pure numpy transformer: token + position embeddings → N pre-norm blocks → LayerNorm → linear head
• Hand-written backprop for every layer, with comments explaining each gradient
• Multi-head causal self-attention with masking, written out step by step (no fused kernels to hide behind)
• AdamW with bias correction, decoupled weight decay, global-norm gradient clipping, warmup + cosine LR schedule
• Dropout and early stopping (keeps the best validation checkpoint)
• Sampling with temperature, top-k and greedy decoding
• Gradient-checked: finite-difference tests for each layer and the full model in float64
• Compressed .npz checkpoints with config, vocabulary and loss history
• Optional LLM comparison: tinygpt compare puts tiny-gpt's output next to any OpenAI-compatible model's (offline mock when no key is set)

🚀 Quick start

git clone https://github.com/hbtabi/tiny-gpt.git
cd tiny-gpt
pip install -r requirements.txt          # just numpy

python -m tinygpt sample --prompt "The river"          # uses the bundled checkpoint
python -m tinygpt train                                 # ~1-2 min on a laptop CPU
python -m tinygpt sample --prompt "THE NIGHT BUS" --temperature 0.6 --top-k 5

Train on your own text (any UTF-8 file):

python -m tinygpt train --corpus my_book.txt --steps 4000 --n-layer 4 --n-embd 128 --block-size 128 --out checkpoints/book.npz
python -m tinygpt sample --checkpoint checkpoints/book.npz --prompt "Chapter"

All training flags: python -m tinygpt train --help (--steps, --batch-size, --block-size, --n-layer, --n-head, --n-embd, --dropout, --lr, --warmup, --weight-decay, --grad-clip, --eval-every, --seed, ...).

Compare with a hosted LLM (optional)

export OPENAI_API_KEY=sk-...                   # or LLM_API_KEY
export OPENAI_BASE_URL=https://api.openai.com/v1   # any OpenAI-compatible endpoint, e.g. Ollama http://localhost:11434/v1
export OPENAI_MODEL=gpt-4o-mini
python -m tinygpt compare --prompt "The night bus"

Without a key, compare prints tiny-gpt's sample and a clearly labelled offline placeholder instead of calling anything.

Plot the loss curve

pip install matplotlib
python scripts/plot_loss.py checkpoints/tiny-gpt.json docs/loss.png

🔬 How it works

idx (B, T) ──► wte[idx] + wpe[:T] ──► dropout
           ──► [ x + Dropout(Attn(LN(x)))  ;  x + Dropout(MLP(LN(x))) ] × n_layer
           ──► LayerNorm ──► Linear ──► logits (B, T, vocab) ──► cross-entropy vs idx shifted by one

File	What to read it for
tinygpt/layers.py	Linear, LayerNorm, GELU, Dropout, CausalSelfAttention, MLP, Block, each with forward + backward
tinygpt/model.py	Embeddings, loss, the full backward pass, sampling, save/load
tinygpt/optim.py	AdamW, gradient clipping, LR schedule
tinygpt/train.py	The training loop, evaluation and early stopping
tests/test_gradients.py	Numerical gradient checks

Two details worth reading in the code:

• Softmax backward inside attention: dscores = att * (datt - sum(datt * att)). Masked (future) positions have att == 0, so they receive no gradient automatically.
• Cross-entropy backward collapses to (softmax - one_hot) / N, which is why logits and loss are fused.

🗂️ Project structure

tiny-gpt/
├── tinygpt/
│   ├── layers.py        # layers with hand-written backward passes
│   ├── model.py         # TinyGPT model, loss, generation, checkpoints
│   ├── optim.py         # AdamW, clipping, LR schedule
│   ├── data.py          # corpus loading and batching
│   ├── tokenizer.py     # character-level tokenizer
│   ├── train.py         # training loop with early stopping
│   ├── remote.py        # optional OpenAI-compatible comparison
│   └── __main__.py      # CLI: train / sample / compare
├── data/london_verses.txt   # original training corpus (written for this repo)
├── checkpoints/             # bundled trained model + loss history
├── scripts/plot_loss.py
├── tests/                   # gradient checks, model, CLI tests
└── docs/                    # demo image, loss curve, training log

🧪 Tests

pip install pytest
pytest -q          # 20 tests: gradient checks, causal masking, overfitting, sampling, checkpoints, CLI

🛣️ Roadmap

• Weight tying between token embeddings and the output head
• KV-cache for faster sampling
• Byte-pair encoding tokenizer
• Notebook walkthrough that visualises attention maps per head
• Optional mixed precision (float16 weights, float32 accumulators)

📄 Licence

MIT © 2026 Mohammed Hassan bin Tayyeb · Built in London at Hasenix. The bundled corpus data/london_verses.txt is original writing released under the same licence.