> For the complete documentation index, see [llms.txt](https://handbook.modular.com/llms.txt).
> Markdown versions of all pages are available by appending .md to any URL.

# LLM Inference Handbook

_LLM Inference Handbook_ is your technical glossary, guidebook, and reference -
all in one. It covers everything you need to know about LLM inference, from core
concepts and performance metrics (e.g.,
[Time to First Token and Tokens per Second](https://handbook.modular.com/llm-inference-basics/llm-inference-metrics.md)),
to optimization techniques (e.g.,
[continuous batching](https://handbook.modular.com/inference-optimization/static-dynamic-continuous-batching.md)
and [prefix caching](https://handbook.modular.com/inference-optimization/prefix-caching.md)),
[GPU architecture](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals.md), and
deployment patterns like [BYOC](https://handbook.modular.com/getting-started/bring-your-own-cloud.md) and
[on-prem](https://handbook.modular.com/getting-started/on-prem-llms.md).

## Motivation

We wrote this handbook to solve a common problem facing developers: LLM
inference knowledge is often fragmented; it’s buried in academic papers,
scattered across vendor blogs, hidden in GitHub issues, or tossed around in
Discord threads. Worse, much of it assumes you already understand half the
stack.

There aren’t many resources that bring it all together — like how
[inference differs from training](https://handbook.modular.com/llm-inference-basics/training-inference-differences.md),
why
[goodput matters more than raw throughput](https://handbook.modular.com/llm-inference-basics/llm-inference-metrics.md#goodput)
for meeting SLOs, or how
[prefill-decode disaggregation](https://handbook.modular.com/inference-optimization/prefill-decode-disaggregation.md)
works in practice.

So we started pulling it all together.

## Who this is for

This handbook is for engineers deploying, scaling or operating LLMs in
production, whether you're fine-tuning a small open model or running large-scale
deployments on your own stack.

If your goal is to make LLM inference faster, cheaper, or more reliable, this
handbook is for you.

## How to use this

You can read it start-to-finish or treat it like a lookup table. There’s no
wrong way to navigate. We’ll keep updating the handbook as the field evolves,
because LLM inference is changing fast, and what works today may not be best
tomorrow.

## Interactive tools

This handbook provides various interactive tools to help you learn by trying the
concepts directly:

- [LLM Inference Visualizer](https://handbook.modular.com/llm-inference-basics/what-is-llm-inference.md): Walk
  through the request lifecycle and see how tokens flow through prefill and
  decode.
- [LLM Lifecycle Visualizer](https://handbook.modular.com/llm-inference-basics/training-inference-differences.md):
  See where training and inference sit in the model lifecycle, and how inference
  runs on every request.
- [Token-by-Token Decode Loop](https://handbook.modular.com/llm-inference-basics/how-does-llm-inference-work.md#decode):
  Step through autoregressive decoding and watch each new token extend the
  sequence and KV cache.
- [Latency Timeline Visualizer](https://handbook.modular.com/llm-inference-basics/how-does-llm-inference-work.md#decode):
  See how every decode step is followed by detokenization, and which stages
  TTFT, ITL, and E2EL span.
- [Context Window Simulator](https://handbook.modular.com/llm-inference-basics/how-does-llm-inference-work.md#what-is-a-context-window-and-how-does-it-work-in-llm-inference):
  See how the full conversation is re-sent each turn and fills the context
  window.
- [Latency Metrics Playground](https://handbook.modular.com/llm-inference-basics/llm-inference-metrics.md#latency):
  Explore TTFT, E2EL, TPOT, and SLO-based goodput.
- [Top-p vs Top-k Filter](https://handbook.modular.com/model-interaction/inference-parameters.md#top-p-and-top-k-sampling):
  Compare how each filter handles peaky, mixed, and flat distributions.
- [Model Explorer](https://handbook.modular.com/getting-started/choosing-the-right-model.md): Browse popular
  open-source LLMs and compare their architecture, scale, context, and typical
  GPU deployment.
- [GPU Comparison Table](https://handbook.modular.com/getting-started/choosing-the-right-gpu.md#matching-gpus-to-open-source-llms):
  Match popular open-source LLMs to suitable NVIDIA and AMD GPUs.
- [GPU Memory Calculator](https://handbook.modular.com/getting-started/calculating-gpu-memory-for-llms.md#):
  Estimate VRAM requirements for serving an LLM.
- [Quantization Memory Impact Visualizer](https://handbook.modular.com/model-preparation/llm-quantization.md#quantization-formats):
  Compare weight memory across quantization formats.
- [Batching Strategy Simulator](https://handbook.modular.com/inference-optimization/static-dynamic-continuous-batching.md):
  Compare static, dynamic, and continuous batching behavior.
- [Chunked Prefill Scheduler](https://handbook.modular.com/inference-optimization/static-dynamic-continuous-batching.md#chunked-prefill):
  See how a whole prefill stalls active decodes, and how chunking lets them
  continue.
- [KV Cache Memory Calculator](https://handbook.modular.com/inference-optimization/kv-cache-offloading.md#how-to-calculate-the-kv-cache-size):
  Estimate how much memory the KV cache consumes.
- [GPU Execution and Memory Map](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals.md):
  Visualize how threads, warps, SMs, and the GPU memory hierarchy fit together.
- [Warp Divergence Visualizer](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/threads-warps-blocks.md#warps):
  See how a branch splits a warp into serialized passes that leave lanes idle.
- [GPU Memory Explorer](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/gpu-memory.md):
  Understand different memory tiers and see how reuse scope changes as data moves.
- [Memory Coalescing Visualizer](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/gpu-memory.md#memory-coalescing):
  Compare contiguous, misaligned, strided, and scattered access, and see how
  many sectors each one forces the GPU to fetch.
- [Streaming Multiprocessor Explorer](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/streaming-multiprocessors.md#what-an-sm-contains):
  Explore the units inside an SM and see how many of each an H100 has.
- [Warp Scheduler Visualizer](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/streaming-multiprocessors.md#how-warp-scheduling-hides-latency):
  Compare too few, enough, and extra resident warps on a scheduler timeline.
- [Kernel Fusion Visualizer](https://handbook.modular.com/kernel-optimization/kernel-optimization-for-llm-inference.md#what-is-kernel-fusion):
  See how separate kernels round-trip every intermediate through HBM, and what
  fusing them saves.

## Contributing

We welcome contributions! If you spot an error, have suggestions for
improvements, or want to add new topics, please open an issue or submit a pull
request on our
[GitHub repository](https://github.com/modular/llm-inference-handbook).