Attention turns scores into weights with softmax, and softmax needs one exponential per score. That is a small share of the arithmetic, but it sits on the critical path: every layer, for every token, must finish it before the next matrix multiply can start.
From H100 to B200, BF16 tensor throughput rose from about 1 to 2.25 PFLOPs, while the units that compute exponentials did not grow. Per streaming multiprocessor, that is 8,192 tensor operations per cycle against 16 exponentials. In FlashAttention-4's forward pass, the matrix work and the exponentials each take about 1,024 cycles per tile, so neither can hide behind the other.
FlashAttention-4's answer is to compute part of the exponentials in software, on the general multiply-add units, with a short polynomial:
The lesson for anyone designing inference silicon: the few percent of work that is not matrix math decides whether the matrix units are busy or waiting.
The newest model families add steps that a multiply-accumulate array was never built for, and each asks something different of the hardware.
| Model feature | Where it appears | What the hardware must do | Why matrix units alone fall short |
|---|---|---|---|
| Sparse attention | DeepSeek V3.2 | Score every past token, then keep the top 2,048 for each new token | Picking the top 2,048 from a long history is selection, closer to sorting than multiplying |
| State space layers | Hybrid Mamba models | Update a fixed-size state every token, and scan across the sequence during prefill | Without a parallel scan, prefill becomes one long sequential loop |
| Latent attention | DeepSeek V3 and later | Let all 128 heads attend to one compressed cache of 576 values per token per layer | Decode shifts from waiting on memory to waiting on compute |
| Mixture of experts | DeepSeek, gpt-oss, Qwen3 | Route each token to a few experts that live on different chips | Every layer adds an all-to-all exchange, so link latency sits on the critical path |
None of these is exotic anymore. Each is in a model that providers serve today, as the vLLM write-up of DeepSeek's sparse attention shows, and the next generation will add more.
Every past token leaves an entry in the key-value cache, and every new token reads the whole cache again. So memory grows with context, per user, and the bandwidth to read it grows with it.
| Model | Cache per token | Per user at 128K tokens | Per user at 1M tokens |
|---|---|---|---|
| Llama 3.3 70B, 8-bit cache | about 164 KB | about 21 GB | about 164 GB |
| DeepSeek V3.2, latent attention | about 40 KB | about 5 GB | about 40 GB |
| DeepSeek V4-Pro, compressed attention | about 5 KB | about 0.7 GB | about 5 GB |
The Llama figures follow from its published shape: 80 layers, 8 key-value heads of 128 values each, keys plus values, one byte per value. DeepSeek V3.2 stores 656 bytes per token per layer across 61 layers. DeepSeek V4 compresses the cache along the sequence, and in vLLM's 8-bit layout its Pro model needs about 5 KB per token.
A thirty-fold difference in per-user memory, between models served side by side today, changes what the right chip looks like. Hardware sized for one generation's memory ratio is mis-sized for the next.
A compiler can teach a chip new math after tape-out. It cannot teach it new data movement.
New math is the easy case. A new activation, a new normalization or a different precision can be built from multiply-add and a polynomial with fitted coefficients, and loaded as software. It may run a few times slower than a dedicated circuit, but it is a small share of the work.
Data movement is different. When the work grows with the data, emulation multiplies the cost by the size of the data.
| What the model needs | How you would emulate it | What it costs |
|---|---|---|
| Top 2,048 of a long history | Repeat a maximum search 2,048 times | About 2,048 passes over the data instead of a handful |
| Fetch only the selected entries | Read the whole cache, then mask | Undoes the point of sparse attention |
| Running scan | Step through the sequence one token at a time | A long prefill becomes a sequential loop |
| Expert routing across chips | Relay messages through other chips | Extra link hops on every expert layer |
The decisions that decide which future models a chip can serve well are mostly about moving data, and they are made before the first wafer.
The same model can run with 4-bit or 8-bit weights, with a hardware exponential or a polynomial one, with sums added in a different order. Floating-point addition is not associative, so each choice shifts the result slightly. On one machine that is a tuning detail. Across thousands of nodes from different owners, it becomes a question of trust: which model, at which precision, on which hardware, produced this answer?
Supporting a model at scale therefore includes proving what ran where, and that proof is strongest when it is anchored in the silicon itself rather than in software that can be copied. At UBIQS we are building distributed inference from the silicon up, on a silicon-proven hardware identity in every node: silicon-intrinsic, with no stored secret.
A platform that answers all five is supporting models. One that cannot is fitting weights.
Decentralized AI networks offer cheap, resilient inference capacity, but a buyer cannot trust a result from a machine they have never seen. Staking.
Read →Confidential computing protects a workload in memory while it runs; it doesn't prove which enrolled machine ran it, under whose authority. That's a.
Read →We're talking with teams building verifiable, component-level trust into infrastructure and devices.
