This book23.5 x 19.1 x 1.3 cm9.25 x 7.52 x 0.51 in
Ship to
Taiwan
Select your location
Shipping costs will be calculated based on this address throughout the site.
Shipping costs will be calculated based on this address throughout the site.
Select your country
Americas
Argentina
Brazil
Canada
Chile
Colombia
Costa Rica
Dominican Republic
Ecuador
El Salvador
Mexico
Peru
U.S.A.
Uruguay
Europe
Austria
Belgium
Croatia
Czech Republic
Denmark
Finland
France
Germany
Greece
Hungary
Ireland
Italy
Latvia
Malta
Netherlands
Norway
Poland
Portugal
Serbia
Slovakia
Slovenia
Spain
Sweden
Switzerland
United Kingdom
Rest of the world
THE LLM ENGINEER. Building Transformer Language Models with Python and PyTorch
Lachlan James
THE LLM ENGINEER. Building Transformer Language Models with Python and PyTorch
Lachlan James (Author) · Independently published · Paperback
THE LLM ENGINEER. Building Transformer Language Models with Python and PyTorch
Lachlan James
Most developers working with large language models today are flying blind. They understand the interface but they don't really understand the machine. And there lies the gap - between knowing how to use a model and knowing how to build, debug, and adapt one - is where real engineering capability lives.
The LLM Engineer is a hands-on implementation guide that closes that gap completely. Starting from a single sentence - predict the next token - and ending with a quantized model serving requests through an OpenAI-compatible API, this book walks you through every layer of the modern LLM stack. No black boxes. No magic. Every concept is introduced once, explained precisely, and immediately followed by complete, runnable code.
The architecture you build matches the design of Llama 3, Mistral, and Gemma at the blueprint level - rotary position embeddings, grouped-query attention, SwiGLU activations, RMSNorm - not a toy approximation, but the real thing at teachable scale.
Inside this book, you'll learn how to:
Implement byte-pair encoding from scratch and understand the tokenizer quirks that cause real production bugsBuild scaled dot-product attention, multi-head attention, and grouped-query attention (GQA) - the memory-efficient variant used by every major open-weight model since 2022Construct a complete transformer block using pre-norm RMSNorm, SwiGLU feed-forward layers, RoPE positional encodings, and residual connectionsDesign and run a full training pipeline: packed sequences, AdamW with parameter-group weight decay, cosine warmup scheduling, gradient clipping, mixed-precision bfloat16, and distributed data parallelismFine-tune models efficiently using LoRA and QLoRA - implemented entirely from scratch, not just called from a libraryTrain for human preference alignment using Direct Preference Optimization (DPO), the technique that replaced PPO-based RLHF in most production pipelinesQuantize models to INT8 and 4-bit precision using GPTQ, AWQ, and GGUF for CPU deployment with llama.cppServe models at scale using vLLM with PagedAttention and continuous batching, and expose an OpenAI-compatible APIAlong the way, you'll build:
A byte-pair encoding tokenizer that handles Unicode, byte-level encoding, and the edge cases that break naive implementationsA complete GPT-style transformer language model - architecturally identical to Llama 3 - trained from scratch on real text dataA full training loop with Weights & Biases experiment tracking, checkpointing, and distributed GPU support via PyTorch DDPAn inference engine with greedy decoding, temperature sampling, top-k, top-p, repetition penalties, speculative decoding, and structured output generationLoRA and QLoRA adapters injected and merged into a pre-trained model, reducing trainable parameters by over 99%A DPO-aligned instruct model trained on preference pairs, starting from an SFT checkpointA production-ready serving stack: quantized model exported to GGUF, served locally via Ollama, and deployed at scale with vLLMEvery chapter includes working, runnable code, common bug sections drawn from real implementation failures, and exercises that push your understanding beyond what the text alone can teach.
Do you have a question about the book? Login to be able to add your own question.