# LLM Inference Handbook > A practical handbook for engineers building, optimizing, scaling, and operating LLM inference systems in production. Markdown versions of all pages are available by appending `.md` to any URL. - [LLM Inference Handbook](https://handbook.modular.com/index.md): A practical handbook for engineers building, optimizing, scaling, and operating LLM inference systems in production. ## Table of Contents - [Bring Your Own Cloud (BYOC)](https://handbook.modular.com/getting-started/bring-your-own-cloud.md): Bring Your Own Cloud (BYOC) is a deployment model where vendors run software in your cloud, combining managed orchestration with complete data cont... - [Calculating GPU memory for serving LLMs](https://handbook.modular.com/getting-started/calculating-gpu-memory-for-llms.md): Learn how to calculate GPU memory for serving LLMs. - [Choosing the right GPU](https://handbook.modular.com/getting-started/choosing-the-right-gpu.md): Select the right NVIDIA or AMD GPUs (e.g., L4, A100, H100, B200, MI250X, MI300X, MI350X) for LLM inference. - [Choosing the right inference framework](https://handbook.modular.com/getting-started/choosing-the-right-inference-framework.md): Learn what LLM inference frameworks do, why raw model execution is not enough for production, and how to choose the right inference frameworks for ... - [Choosing the right model](https://handbook.modular.com/getting-started/choosing-the-right-model.md): Select the right models for your use case. - [Planning your deployment](https://handbook.modular.com/getting-started.md): Before you can run an LLM in production, you first need to make a few key - [On-prem LLM deployments](https://handbook.modular.com/getting-started/on-prem-llms.md): On-prem LLMs are large language models deployed within an organization’s own infrastructure, such as private data centers or air-gapped environment... - [Serverless vs. self-hosted LLM inference](https://handbook.modular.com/getting-started/serverless-vs-self-hosted-llm-inference.md): Understand the differences between serverless LLM APIs and self-hosted LLM deployments. - [Where is LLM inference run?](https://handbook.modular.com/llm-inference-basics/cpu-vs-gpu-vs-tpu.md): Learn the differences between CPUs, GPUs, and TPUs and where you can deploy them. - [How does LLM inference work?](https://handbook.modular.com/llm-inference-basics/how-does-llm-inference-work.md): Learn how LLM inference works, from tokenization to prefill and decode stages, with tips on performance, KV caching, and optimization strategies. - [Foundations](https://handbook.modular.com/llm-inference-basics.md): LLM inference is where models meet the real world. It powers everything from - [Key metrics for LLM inference](https://handbook.modular.com/llm-inference-basics/llm-inference-metrics.md): Measure key metrics like latency and throughput to optimize LLM inference performance. - [Training vs. inference](https://handbook.modular.com/llm-inference-basics/training-inference-differences.md): LLM training builds the model while LLM inference applies it to generate real-time outputs from new inputs. - [What is LLM inference?](https://handbook.modular.com/llm-inference-basics/what-is-llm-inference.md): LLM inference is the process of using a trained language model to generate responses or predictions based on prompts. - [Data, tensor, pipeline, expert and hybrid parallelisms](https://handbook.modular.com/inference-optimization/data-tensor-pipeline-expert-hybrid-parallelism.md): Understand the differences between data, tensor, pipeline, expert and hybrid parallelisms. - [Inference optimization](https://handbook.modular.com/inference-optimization.md): Running an LLM is just the starting point. Making it fast, efficient, and - [Inference routing](https://handbook.modular.com/inference-optimization/inference-routing.md): Route LLM requests using cache locality, queue depth, KV cache pressure, and worker state for lower latency and better utilization. - [KV cache offloading](https://handbook.modular.com/inference-optimization/kv-cache-offloading.md): Learn how KV cache offloading improves LLM inference by reducing GPU memory usage, lowering latency, and cutting compute costs. - [LLM performance benchmarks](https://handbook.modular.com/inference-optimization/llm-performance-benchmarks.md): LLM performance benchmarks are standardized tests that measure how LLMs perform under specific conditions. Unlike leaderboards that rank the best L... - [Offline batch inference](https://handbook.modular.com/inference-optimization/offline-batch-inference.md): Run predictions at scale with offline batch inference for efficient, non-real-time processing. - [PagedAttention](https://handbook.modular.com/inference-optimization/pagedattention.md): Improve LLM memory usage with block-based KV cache storage via PagedAttention. - [Prefill-decode disaggregation](https://handbook.modular.com/inference-optimization/prefill-decode-disaggregation.md): Disaggregate prefill and decode for better parallel execution, resource allocation, and scaling. - [Prefix caching](https://handbook.modular.com/inference-optimization/prefix-caching.md): Prefix caching speeds up LLM inference by reusing shared prompt KV cache across requests. - [Speculative decoding](https://handbook.modular.com/inference-optimization/speculative-decoding.md): Speculative decoding accelerates LLM inference with draft model predictions verified by the target model. - [Static, dynamic and continuous batching](https://handbook.modular.com/inference-optimization/static-dynamic-continuous-batching.md): Optimize LLM inference with static, dynamic, and continuous batching for better GPU utilization. - [FlashAttention](https://handbook.modular.com/kernel-optimization/flashattention.md): FlashAttention is a fast, memory-efficient attention algorithm for Transformers that accelerates LLM training and inference and helps achieve longe... - [GPU architecture fundamentals](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals.md): Understand GPU architecture fundamentals for kernel optimization, including threads, warps, streaming multiprocessors, memory hierarchy, and tensor... - [Kernel optimization](https://handbook.modular.com/kernel-optimization.md): Kernel optimization is about making GPU kernels run faster and more efficiently - [Kernel optimization for LLM inference](https://handbook.modular.com/kernel-optimization/kernel-optimization-for-llm-inference.md): Kernel optimization for LLM inference improves GPU utilization and performance by writing or generating optimized kernels tailored to the compute p... - [Choosing the right kernel optimization tool](https://handbook.modular.com/kernel-optimization/kernel-optimization-tools.md): Compare the main tools for kernel optimization in LLM inference, from cuBLAS and cuDNN to TVM, XLA, Triton, custom CUDA kernels, Mojo and MAX. - [Model preparation](https://handbook.modular.com/model-preparation.md): Model preparation is the process of preparing the model for inference. - [LLM distillation](https://handbook.modular.com/model-preparation/llm-distillation.md): Learn how LLM distillation works, how it compares to quantization, and how to use it to build smaller, faster, and more efficient models for infere... - [LLM fine-tuning](https://handbook.modular.com/model-preparation/llm-fine-tuning.md): Understand LLM fine-tuning and different fine-tuning frameworks. - [LLM quantization](https://handbook.modular.com/model-preparation/llm-quantization.md): Understand LLM quantization and different quantization formats and methods. - [Anthropic-compatible API](https://handbook.modular.com/model-interaction/anthropic-compatible-api.md): An Anthropic-compatible API mirrors Anthropic's Messages API so Claude-based clients, SDKs, and agent tools can use another model or provider with ... - [Function calling](https://handbook.modular.com/model-interaction/function-calling.md): Learn what function calling is and its use case. - [Model interaction](https://handbook.modular.com/model-interaction.md): Model interaction is the process of interacting with the model to get the - [LLM inference parameters](https://handbook.modular.com/model-interaction/inference-parameters.md): LLM inference parameters are request-time settings that control randomness, output length, repetition, stopping behavior, reproducibility, and stru... - [Model Context Protocol](https://handbook.modular.com/model-interaction/model-context-protocol.md): Learn what Model Context Protocol (MCP) is and its use case. - [OpenAI-compatible API](https://handbook.modular.com/model-interaction/openai-compatible-api.md): An OpenAI-compatible API implements the same request and response formats as OpenAI's official API, allowing developers to switch between different... - [Prompt engineering](https://handbook.modular.com/model-interaction/prompt-engineering.md): Understand prompt engineering for LLM inference. Learn system & user prompts, zero-shot & few-shot prompting, KV cache impact, token costs, and pro... - [Structured outputs](https://handbook.modular.com/model-interaction/structured-outputs.md): Structured outputs are model responses in defined formats like JSON or XML, making AI-generated data predictable, machine-readable, and easy to int... - [Build and maintenance cost](https://handbook.modular.com/infrastructure-and-operations/build-and-maintenance-cost.md): Building LLM infrastructure in-house is costly, complex, and slows AI product development and innovation. - [LLM observability](https://handbook.modular.com/infrastructure-and-operations/comprehensive-observability.md): LLM observability provides end-to-end visibility into LLM inference, using metrics, logs, and events to ensure reliable, efficient, and scalable mo... - [What is distributed inference?](https://handbook.modular.com/infrastructure-and-operations/distributed-inference.md): Distributed inference is the practice of running model inference across multiple GPUs, workers, nodes, or regions to achieve scalable, reliable, an... - [Fast scaling](https://handbook.modular.com/infrastructure-and-operations/fast-scaling.md): Fast scaling enables AI systems to handle dynamic LLM inference workloads while minimizing latency and cost. - [Infrastructure and operations](https://handbook.modular.com/infrastructure-and-operations.md): LLMs don't run in isolation. They need robust infrastructure behind them, from - [InferenceOps and management](https://handbook.modular.com/infrastructure-and-operations/inferenceops-and-management.md): Scale LLM inference confidently with InferenceOps workflows and infrastructure best practices. - [Multi-cloud and cross-region inference](https://handbook.modular.com/infrastructure-and-operations/multi-cloud-and-cross-region-inference.md): Multi-cloud and cross-region inference is the practice of running LLM workloads across multiple cloud providers or regions to improve latency, avai... - [Multi-model inference pipelines](https://handbook.modular.com/infrastructure-and-operations/multi-model-inference-pipelines.md): Multi-model inference pipelines chain multiple models into one application path, improving specialization and control, but at the cost of extra lat... - [What is LLM inference infrastructure?](https://handbook.modular.com/infrastructure-and-operations/what-is-llm-inference-infrastructure.md): Deploy, scale, and manage LLMs with purpose-built inference infrastructure.