IMPORTANT: To view this page as Markdown, append `.md` to the URL. For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL.

GPU architecture

Before writing or tuning GPU kernels, you need a working model of how a GPU runs code. Without it, suggestions like “increase occupancy” or "reduce shared memory bank conflicts" are just a set of rules to memorize. You don't fully understand when they apply and when they don't.

This section explains modern GPU architecture at the level needed for kernel work. The details lean toward NVIDIA hardware because CUDA dominates much of the LLM inference ecosystem today. However, the core concepts apply broadly to AMD GPUs and other parallel accelerators as well.

Hardware and the execution model​

The hardware is what physically exists on the GPU: streaming multiprocessors (SMs), Tensor Cores, and the registers, caches, shared memory, and high bandwidth memory (HBM) that form the GPU memory hierarchy. These resources determine how much work and data the GPU can keep close to the compute units.

The execution model is the abstraction used to write a kernel. Code creates threads and organizes them into warps, blocks, and grids. The GPU schedules that work across the available hardware.

Kernel optimization connects these two views. A fast kernel maps the execution model onto the hardware and respects limits on compute, memory bandwidth, registers, shared memory, and scheduling capacity.

GPU execution and memory map​

The visual below follows work from individual threads to an SM, then shows the memory layers that feed the compute units. Four terms establish the execution hierarchy:

  • A thread is the smallest logical unit of work and usually handles one element or a small group of elements.
  • A warp is a group of 32 threads that an NVIDIA SM schedules together.
  • A thread block groups threads that can cooperate through shared memory and synchronization.
  • A streaming multiprocessor (SM) is the on-chip compute unit that accepts blocks and issues instructions from ready warps.
GPU Execution and Memory Map
Understand how work is organized and where data lives, from individual threads up to global memory. Hover over any section to highlight it.
→→→
On-chipGPU die
SM 0core compute unit
Thread Block 0
Warp 032 threads, lockstep
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Warp 132 threads
⋮ up to 32 warps (1024 threads max)
Thread Block 1Runs on the same SM
Register file
256 KB · private per thread
Shared memory
per block · programmer-managed
L1 cache
per SM · hardware-managed
Warp schedulers
issue ready warps each cycle
SM 1
Thread Block 2
Registers · Shared mem · L1 · Schedulers
SM 2
Thread Block 3
Registers · Shared mem · L1 · Schedulers
SM 3
Thread Block 4
Registers · Shared mem · L1 · Schedulers
up to 132 SMs
(H100 SXM)
L2 Cache50 MB · shared across all SMs · hardware-managed
↕memory bus
Off-chipHBM (Global Memory / VRAM)
80 GB · 3.35 TB/s (H100 SXM)
Model weights
KV cache
Activations
Intermediate buffers
Largest but slowest. ~400+ cycle access latency. Kernel optimization focuses on minimizing round-trips here.

What's next​

The four pages below separate the execution model from the physical hardware, then show how the two views meet during kernel execution.

Edit this page