Building a Large Language Model from Scratch: From tensors and tokenizers to a DeepSeek-style Mixture-of-Experts reasoning model - reading the open-source code at every step

You can read about how large language models work - or you can build one.

Most introductions stop at GPT-2. Building a Large Language Model from Scratch goes all the way to the architecture running today - the Mixture-of-Experts, Multi-head Latent Attention, FP8 training, and reinforcement-learned reasoning behind frontier open models like DeepSeek-V3 and DeepSeek-R1. Beginning with a single scalar and a hand-written autograd engine, you will assemble, line by line, a complete modern model: trained, aligned, taught to reason, and ready to serve.

This is a working engineer's book, not a survey. Every idea is built in real, runnable code - and the pivotal components are reproduced directly from the canonical open-source projects (micrograd, nanoGPT, minbpe, and DeepSeek's own model code), so you study the actual source rather than a paraphrase of it.

Across 32 chapters and six appendices, you'll learn to:

• Build the transformer from first principles - attention, RoPE, RMSNorm, SwiGLU, and a full GPT
• Implement DeepSeek's signature innovations: Mixture-of-Experts, Multi-head Latent Attention, multi-token prediction, and sparse attention
• Train at scale with FP8 precision, ZeRO/FSDP, and pipeline and expert parallelism - and grasp the economics that make it affordable
• Turn a base model into a helpful assistant with SFT, LoRA/QLoRA, RLHF, DPO, and the GRPO recipe behind R1's reasoning
• Serve models efficiently with KV caching, paged attention, and vLLM - then evaluate them honestly
• Assemble your own small DeepSeek-style model, end to end

Written for developers, ML engineers, and serious students, it assumes programming fluency and a little linear algebra, then builds everything else from there. Rigorous without being academic, practical without being shallow, it is candid about what actually works, what it costs, and where the field is heading.

The frontier labs have more compute - but, thanks to open weights and open methods, no longer a monopoly on how. The surest way to understand modern AI is to build it.

Go build something.