> For the complete documentation index, see [llms.txt](https://handbook.modular.com/llms.txt).
> Markdown versions of all pages are available by appending .md to any URL.

# GPU architecture

Before writing or tuning GPU kernels, you need a working model of how a GPU runs
code. Without it, suggestions like “increase occupancy” or "reduce shared memory
bank conflicts" are just a set of rules to memorize. You don't fully understand
when they apply and when they don't.

This section explains modern GPU architecture at the level needed for kernel
work. The details lean toward NVIDIA hardware because CUDA dominates much of the
LLM inference ecosystem today. However, the core concepts apply broadly to AMD
GPUs and other parallel accelerators as well.

## Hardware and the execution model

The hardware is what physically exists on the GPU:
[streaming multiprocessors (SMs)](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/streaming-multiprocessors.md),
[Tensor Cores](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/tensor-cores.md),
and the registers, caches, shared memory, and high bandwidth memory (HBM) that
form the
[GPU memory hierarchy](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/gpu-memory.md).
These resources determine how much work and data the GPU can keep close to the
compute units.

The execution model is the abstraction used to write a kernel. Code creates
threads and organizes them into warps, blocks, and grids. The GPU schedules that
work across the available hardware.

[Kernel optimization](https://handbook.modular.com/kernel-optimization/kernel-optimization-for-llm-inference.md)
connects these two views. A fast kernel maps the execution model onto the
hardware and respects limits on compute, memory bandwidth, registers, shared
memory, and scheduling capacity.

## GPU execution and memory map

The visual below follows work from individual threads to an SM, then shows the
memory layers that feed the compute units. Four terms establish the execution
hierarchy:

- A
  [thread](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/threads-warps-blocks.md#threads)
  is the smallest logical unit of work and usually handles one element or a
  small group of elements.
- A
  [warp](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/threads-warps-blocks.md#warps)
  is a group of 32 threads that an NVIDIA SM schedules together.
- A
  [thread block](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/threads-warps-blocks.md#thread-blocks)
  groups threads that can cooperate through shared memory and synchronization.
- A
  [streaming multiprocessor (SM)](https://handbook.modular.com/kernel-optimization/gpu-architecture-fundamentals/streaming-multiprocessors.md#what-an-sm-contains)
  is the on-chip compute unit that accepts blocks and issues instructions from
  ready warps.

## What's next

The four pages below separate the execution model from the physical hardware,
then show how the two views meet during kernel execution.

```mdx-code-block

```