Open the hood of modern AI.Everyone is learning to use large language models. Almost no one understands how they actually run. This book closes that gap - carrying you, without a single hand-wave, from "what is a token?" to a working, batched, quantized, multi-GPU inference server you built yourself.
An LLM's weights hold the intelligence, but they do nothing on their own. The inference engine is the software that decides how those billions of numbers are moved, cached, batched, and multiplied against a stream of concurrent users. Get it wrong and a state-of-the-art model crawls and falls over at ten users. Get it right and the same weights on the same GPU serve dozens - at a fraction of the latency and cost.
What you will learn:
- Why decoding is memory-bandwidth-bound - the one law that explains PagedAttention, quantization, and FlashAttention as inevitable consequences, not tricks.
- The full anatomy of an engine: tokenizer, scheduler & continuous batching, memory manager, KV cache & PagedAttention, quantization (INT8/INT4/FP8), sampler, and Mixture-of-Experts routing.
- The hot path: fused GPU kernels and FlashAttention, writing your own kernels in Triton/CUDA/Rust, speculative decoding, and multi-GPU tensor/pipeline/expert parallelism.
- Real engines dissected: vLLM, TensorRT-LLM, llama.cpp, MLC-LLM, and SGLang - and how to deploy Llama, Qwen, and DeepSeek on each.
- Production skills: benchmarking, profiling, cost/SLO capacity planning, and eighteen-plus real-world war stories.
Built for three altitudes. Beginners get the concept and the mental model; intermediate engineers get the algorithms, the math, and runnable Python; advanced engineers get kernel-level detail, C++/Rust considerations, failure modes, and the frontier.
Who it's for: LLM & inference engineers, AI platform engineers and architects, forward-deployed engineers, researchers, and technology leaders who must reason about latency, throughput, memory, and cost - not just call an API.
You will build DwarfStar, a minimal but real inference engine, growing it stage by stage into a batched, quantized, C++/CUDA-accelerated server. Close this book and you will open the source of vLLM or TensorRT-LLM and feel at home.
