Latency
How long does one request take? Crucial for voice, search, and interactive agents.
MEMORY × COMPUTE
Your computer is
mostly waiting.
The math is fast. Fetching the numbers is not. Chip design is the story of shrinking that wait.
ORIGIN
Built from a conversation about Cerebras, GPUs, SRAM, model weights, and why memory keeps moving closer to compute.
Read the source chat ↗
01
Compute transforms.
FIRST PRINCIPLE
Compute transforms.
Memory remembers.
A processor does arithmetic on values. Memory holds the instructions, model weights, pixels, and intermediate results that arithmetic needs.
The catch: storage gets larger as it moves farther from the arithmetic—but also slower and more expensive to reach. Engineers build a hierarchy instead of choosing one kind of memory.
The central trade: tiny and immediate at the top; vast and patient at the bottom.
× +COMPUTE
REG
Inside the execution unit
The number already in your hand.
Registers are the smallest, fastest working locations. They are not cache; they hold the values an instruction is operating on now.
02
THE REAL COST
Distance is the tax.
Slide memory away from compute.
Short wires, little waiting, high bandwidth.
LATENCYLOW
ENERGY / MOVELOW
CAPACITYTINY
This is a conceptual model, not a benchmark. Actual latency and energy depend on architecture, process, packaging, access pattern, and workload.
03Same problem.
FOUR ANSWERS
Same problem.
Different geometry.
THE CLASSIC MODULAR COMPUTER
Keep compute and bulk memory separate.
Separate parts are easier to manufacture, cool, replace, and scale. The price is a narrower path between them.
- Optimizes for
- cost, flexibility, yield
- Pays with
- data movement and latency
- Good fit
- general-purpose computing
Why did small squares win?
A wafer accumulates manufacturing defects. Cutting it into many dies lets a defect ruin one die instead of an entire wafer. Smaller chips are also easier to cool and pair with replaceable memory. Cerebras made the opposite bet possible with redundancy, routing around defects, custom cooling, and system-level design.
04Where do a trillion
THE MODEL-SIZE PROBLEM
Where do a trillion
weights go?
Inference needs the model’s weights available, but not necessarily on one chip.
Systems split layers or tensors across processors, reduce precision through quantization, and stream activations between devices. More memory capacity is helpful; faster movement is also helpful. Neither number alone decides the winner.
Weight storage calculator
WEIGHTS ALONE140 GB
NVIDIA B200
180 GB HBM3E / GPU 1GPU minimum
180 GB HBM3E / GPU 1GPU minimum
CEREBRAS WSE-3
44 GB SRAM / wafer 4wafer systems minimum
44 GB SRAM / wafer 4wafer systems minimum
Capacity-only illustration. Real serving also needs KV cache, activations, runtime buffers, headroom, interconnect, and a workable mapping.
05
HOW TO COMPARE
There is no “fastest chip.”
There is only a fastest system for a specific job, scale, budget, and software stack.
Throughput
How much total work finishes per second? High throughput can coexist with slow individual responses.
Bandwidth
How quickly can weights and activations reach compute? Especially important during token-by-token decode.
Efficiency
Performance per watt and per dollar determine what can run economically at scale.
Scale
How well do many chips cooperate? Interconnects and parallelism turn parts into a system.
Ecosystem
Compilers, frameworks, libraries, operators, and developer familiarity can outweigh raw silicon.
THE THROUGH-LINE
Every architecture is negotiating with data movement.
CPU caches anticipate what comes next. Apple lets CPU and GPU share a pool. NVIDIA stacks HBM beside accelerators and links many GPUs. Groq schedules data deterministically. Cerebras keeps vast compute and SRAM on one wafer. Different tactics; the same enemy.
06
KEEP GOING
Sources & caveats
CEREBRASWSE-3 launch and technical specifications↗
CEREBRASInference, memory bandwidth, and multi-system mapping↗
NVIDIAHGX H200, B200, and B300 memory specifications↗
APPLEUnified memory architecture on Apple silicon↗
GROQDeterministic execution in the LPU architecture↗
ORIGIN CHATThe conversation that prompted this explanation↗
Vendor performance claims depend on model, precision, batch size, latency target, software, system configuration, and benchmark methodology. Use independent, workload-matched tests before purchasing or investing.
NEXT EXPLANATIONHow the internet finds a website →