Try Chat Free Docs Pricing AI Analysis Repo Insights Free Tools About Blog
Sign in Get started free
< Back to blog
Research Velona Team ·24 October 2026 ·9 min read

The Memory Wall: Why AI Response Speed Is Limited by Memory Bandwidth, Not Compute

Review copy. The full designed version, with diagrams, charts and FAQ schema, is in memory-wall.html and should be pasted in through the admin console before publishing.

When an AI model types its answer slowly, the chip is rarely running out of maths. It is waiting for memory. To generate each new token, a language model has to read every active weight from memory once. On an NVIDIA H100 serving one user, that leaves about 99.7% of the chip's compute idle. Researchers call this the memory wall: over two decades, peak server compute grew about 60,000 times while memory bandwidth grew about 100 times.

This report explains the memory wall from first principles, gives you the arithmetic to estimate any model's speed on any hardware, and reviews the six main ways the industry is pushing through it in 2026. It ends with what all of this means for the speed you get and the rupees you pay.

The short answer: why are AI models slow to respond?

A language model answers in two phases. First it reads your whole prompt at once, which is called prefill. Then it writes the reply one token at a time, which is called decode. Prefill processes thousands of tokens in parallel, so the chip does a lot of maths for every byte it reads. Decode produces one token per step, so it reads the entire set of active weights to do a tiny amount of maths, then reads them all again for the next token.

That makes prefill compute-bound and decode memory-bandwidth-bound. The speed at which words appear on your screen is set almost entirely by how fast the hardware can stream weights out of memory. A 500-token reply runs the decode loop 500 times.

The physics: the roofline model in plain words

The roofline model, introduced by Williams, Waterman and Patterson in 2009, needs two numbers about a chip and one about a workload.

The best speed a workload can reach is min(F, B × I). The crossover, F ÷ B, is the ridge point, about 295 FLOP per byte on an H100 at dense BF16. Decode at batch size 1 runs at roughly one FLOP per byte, which uses about 0.3% of peak. At batch 32 it reaches about 11%. Batching helps because weights read once serve many users, but every user brings their own KV cache, and those reads are never shared.

The back-of-envelope formula anyone can use

tokens per second (ceiling) ≈ memory bandwidth ÷ bytes of active weights read per token
bytes of weights = active parameters × bytes per parameter
Example: 70B dense model at 8-bit = 70 GB read per token
On an H100 (3,350 GB/s): 3,350 ÷ 70 ≈ 48 tokens per second, at best

Cerebras used exactly this arithmetic when launching its inference service: a 70B model at 16-bit is 140 GB, so 10 tokens per second needs 1.4 TB/s, and 1,000 tokens per second would need 140 TB/s.

HardwareBandwidth8B at 4-bit (4 GB)70B at 8-bit (70 GB)
Apple M4 Max546 GB/s~137~8 (128 GB model)
Apple M3 Ultra819 GB/s~205~12
NVIDIA RTX 5090 (32 GB)1,790 GB/s~448won't fit
NVIDIA H100 (80 GB)3,350 GB/s~838~48 (tight fit)
NVIDIA H200 (141 GB)4,800 GB/s~1,200~69
NVIDIA B200 (180 GB)8,000 GB/s~2,000~114
NVIDIA Rubin R100 (288 GB)22,000 GB/s~5,500~314

When you compare hardware for inference, compare memory bandwidth per rupee, not teraflops per rupee. Our guide to local AI versus cloud AI for coding in India applies the same arithmetic to buying or renting a GPU.

Why the wall keeps getting taller

In AI and Memory Wall, Gholami and colleagues at UC Berkeley found peak compute scaling about 3.0 times every two years, against 1.6 times for DRAM bandwidth and 1.4 times for interconnect bandwidth. Over 20 years: 60,000× compute, 100× DRAM bandwidth, 30× interconnect. The root cause is energy: Horowitz's 2014 measurements put a 32-bit DRAM read at roughly 640 picojoules, against a few picojoules for a 32-bit multiply.

NVIDIA GPUMemoryBandwidth
A100HBM2e2.0 TB/s
H100HBM33.35 TB/s
H200HBM3e4.8 TB/s
B200 and B300HBM3e8.0 TB/s
Rubin R100 (2026)HBM422 TB/s

B300 raised FP4 compute about 1.5× over B200 while keeping the same 8 TB/s: the memory wall in miniature.

The second wall: the KV cache

Models store a key-value (KV) cache of vectors for every token in every layer, and each decode step reads all of it. It grows with context and belongs to one user, so batching cannot share it.

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value
Llama 3 70B: 2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 0.33 MB
At 128K tokens: ≈ 43 GB for one user
At 1M tokens: ≈ 328 GB for one user

Past about 427,000 tokens, one user's cache outgrows the 140 GB of BF16 weights. This is why long-context serving is expensive and why our companion report on the future of AI model architecture focuses so much on the KV cache.

Six ways the industry is pushing through the wall

Every technique either reads fewer bytes per token or gets more useful work out of each byte read.

1. Quantization

8-bit or 4-bit weights halve or quarter bytes per token. An FP8 KV cache gives close to a 2× decode gain when memory-bound. The cost is accuracy, as our 2026 research review showed for quantized reasoning models.

2. Sparsity (mixture-of-experts)

Total parameters decide memory needed. Active parameters decide bytes read per token. At 8-bit: a 70B dense model reads 70 GB per token, DeepSeek V4 Pro (1.6T total, 49B active) reads 49 GB, DeepSeek V3 (671B, 37B active) reads 37 GB, and DeepSeek V4 Flash (284B, 13B active) reads 13 GB. This underpins our benchmark of cheap LLMs for production agents.

3. KV cache compression

GQA shares KV heads. DeepSeek-V2's Multi-head Latent Attention cut the KV cache 93.3% and raised maximum throughput 5.76×. DeepSeek V4 reports needing 10% of V3.2's KV cache and 27% of its per-token compute at 1M tokens.

4. Faster memory: HBM4 and SRAM

SystemFast memoryBandwidthTrade-off
NVIDIA Rubin GPU288 GB HBM422 TB/sHuge capacity, strong batched throughput
Groq 3 LP30 chip~500 MB SRAM150 TB/sTiny capacity
Groq 3 LPX rack128 GB SRAM~40 PB/sDecode co-processor beside a Rubin rack
Cerebras WSE-344 GB SRAM21 PB/sFastest single-user speed, trails GPUs at high concurrency

At GTC in March 2026 NVIDIA unveiled the Groq 3 LPX rack, built on technology from a reported $20 billion Groq deal. Rubin GPUs handle prefill and decode attention, while feed-forward and expert layers move to SRAM-based LPUs. Cerebras has similar deals with OpenAI and AWS.

5. Batching

One weight read serves many users, which makes tokens cheap, but per-user speed falls as batches grow and the KV cache caps how many users fit.

6. Parallel decoding

Speculative decoding reported 2× to 3× speedups with identical outputs (Leviathan et al., 2023), with gains shrinking on busy servers. Diffusion language models report around 1,000 tokens per second on standard GPUs.

What this means for you as a developer

  1. Pick a low active-parameter model for anything interactive.
  2. Cap output length with max_tokens.
  3. Prune the context, as our context engineering guide recommends.
  4. Stream every reply.
  5. Measure tokens per second yourself at p50 and p95.

Measure the memory wall in five minutes

"""decode_speed.py: time to first token and decode speed per model."""
import os
import time

from openai import OpenAI

client = OpenAI(api_key=os.environ["VELONA_API_KEY"], base_url="https://velona.in/v1")
MODELS = ["deepseek/deepseek-v4-flash", "openai/gpt-5-mini", "anthropic/claude-sonnet-5"]
PROMPT = "Explain how a hash map works, in about 300 words."


def measure(model):
    start = time.perf_counter()
    first = None
    chunks = 0
    stream = client.chat.completions.create(
        model=model, messages=[{"role": "user", "content": PROMPT}],
        max_tokens=500, stream=True,
    )
    for event in stream:
        if event.choices and event.choices[0].delta.content:
            if first is None:
                first = time.perf_counter()
            chunks += 1
    end = time.perf_counter()
    decode_s = max(end - (first or end), 1e-6)
    return {
        "model": model,
        "time_to_first_token_s": round((first or end) - start, 2),
        "decode_chunks_per_s": round(chunks / decode_s, 1),
    }


for m in MODELS:
    print(measure(m))

Limits of this analysis

The speed ceilings count weight reads only, so real speeds are lower. SRAM and HBM bandwidth are not directly comparable. Several 2026 figures come from vendor announcements and press coverage. Velona prices are live list prices on 24 October 2026.

Frequently asked questions

Why are large language models slow to generate text?

Because token generation is limited by memory bandwidth. For each token the model must read all of its active weights and its KV cache from memory, so compute sits mostly idle.

What is the memory wall in AI?

The growing gap between how fast chips compute and how fast they move data: roughly 60,000× compute growth against 100× DRAM bandwidth growth over 20 years.

Is LLM inference compute-bound or memory-bound?

Prefill is compute-bound. Decode is memory-bandwidth-bound.

How do I estimate a model's tokens per second?

Divide memory bandwidth by bytes of active weights per token. A 70B model at 8-bit on an H100 tops out near 48 tokens per second for one user.

Why are mixture-of-experts models faster?

Only a small fraction of parameters are read per token. DeepSeek V4 Flash activates 13 billion of 284 billion.

Why do output tokens cost more than input tokens?

Each output token needs its own pass through memory, so output often costs 3 to 5 times more than input.

Sources