AI Engineering · Deep Learning
GPT from Scratch
Using LLMs well means understanding what happens inside them — attention, tokenization, and why training behaves the way it does.
- My role
- Solo learning project, written up as a build log.
- When
- 2026 · in progress
10.8M
parameters
1.47
validation loss
5,000
training steps
1 GPU
~45 min of training
Approach
A GPT-style model built step by step in PyTorch — bigram baseline, self-attention, multi-head attention, full transformer blocks — trained on a French text corpus, with attention maps visualised.
Results
A 10.8M-parameter decoder-only transformer reaching a validation loss of 1.47 after 5,000 steps (about 45 minutes on one GPU). Each stage — bigram, single head, multi-head, full blocks — is kept as a separate checkpoint so the gain from every idea is measurable.
Technical highlights
- 01Every component implemented by hand: tokenizer, attention, residuals, layer norm
- 02Ablations showing what each architectural step buys in validation loss
- 03Attention maps plotted to see what individual heads learn
Stack
Related case studies
AI Engineering · Agents
AI Agents Toolkit for Index Operations
46
reusable skills across 5 agents, in daily use
AI Engineering · Agents
Aloi, a Market Research Agent that Checks its Figures
27/28
eval questions passed with no figure missing from the fetched data (one run)
AI Engineering · Developer Tools
AgentDeck, One Place to Answer Your Coding Agents
1 place
to answer Claude Code and Codex, on the PC or from a phone