← Frontier Inference Margins · all research reports
The complete, unedited response of the roofline throughput-model consultation (58m52s,
gpt-5-6-pro, Pro-verified, 2026-07-10; conversation). Provenance: recovered 2026-07-11 from the session transcript that received it — original bytes, not a regeneration. The adopted-verdict wrapper states the decision the site took (anchor fits stay); this page is the full derivation behind it. The runnable diagnostic is at /tests/roofline-diagnostic.mjs. SHA-256 (response body below this header):c3b6e650d6e80e11ff9c7c03b502563401a45fc0d99e9dbac6f2415e37ae891d
I’ll derive an implementable roofline, audit current assumptions, and run platform-level holdouts with explicit batch/MTP sensitivity and downstream margin propagation.
The key transfer variable is decode reuse: weight traffic scales roughly as 1/B, while MTP alters useful tokens and work per iteration.
A critical comparability issue is that GB200’s 10.1k result appears to use NVFP4/FP8 at batch 1024, not an 8-bit-equivalent operating point.
The explicit roofline is a retrospective near-miss: worst decode error is 28.5%, rising to 50.5% with literal GB200 precision; H800-only fitting invalidates the formal gate.
Do not replace the anchor-fit model with this roofline yet.
A favorable retrospective implementation gives respectable predictions at the throughput-oriented operating points: H20 within 11–16%, GB200 within about 2% under an optimistic NVFP4 assumption, and Ascend’s 50 ms point within 10%. But the council-compliant test must hold out and score the entire platform. Once Ascend’s documented 15 ms operating point is included, the model predicts 324 vs 538 tokens/s/NPU, a −39.8% error. That exceeds the existing ~25% gate.
More fundamentally, the preregistered experiment is presently unrunnable:
The existing single-MFU test failed at 59% worst error; this roofline improves the favorable retrospective result but still fails the whole-platform gate and cannot be called a preregistered validation. citeturn692601view0turn525695view0
For platform (j) and operating point (o), define:
DeepSeek-R1/V3 supplies the physical architecture constants: 37B active parameters, 61 layers, hidden size 7,168, 128 attention heads, 512-dimensional compressed KV, 64-dimensional decoupled RoPE key, 58 MoE layers, and eight routed experts per token. citeturn848796view0turn692601view9
Including core attention work rather than only parameter FLOPs:
2P_{\rm active} + 2N_LN_H(d_{qk}+d_v)L ]
Using (P_{\rm active}=37\text{B}), (d_{qk}=128+64=192), and (d_v=128):
74\times10^9 + 4{,}997{,}120L \quad\text{FLOPs.} ]
Thus:
[ t_C=\frac{bq\Phi(L)}{F_j}. ]
For genuinely mixed precision, replace that with:
[ t_C=bq\sum_r\frac{\Phi_r(L)}{F_{j,r}}. ]
One complete scan of the latent KV cache for one sequence and one target position is:
N_L(512+64)s_{KV}L. ]
Therefore:
Let (W_{\rm iter}) be active weight bytes fetched during one iteration. Then:
\frac{W_{\rm iter}+bqK(L)}{B_{H,j}}. ]
For the hand calculation, I use the favorable simplification (W_{\rm iter}=37) GB for the 8-bit runs: active weights are fetched once and reused across the batch.
A production MoE implementation should instead calculate expected distinct expert-weight traffic:
W_{\rm always} + \sum_e W_e\left[1-(1-p_e)^{bqk}\right], ]
with routing probabilities and expert placement fixed independently of end-to-end throughput. The simple 37 GB assumption is especially questionable at small batch and large expert-parallel degree.
A gross BF16 dispatch-plus-combine payload is:
13.304\text{ MB} ]
per target position.
With a remote-traffic fraction (r) and independently measured round-trip collective startup (\alpha_j):
N_{\rm MoE}\alpha_j + bqD \left( \frac{1-r}{B_{\ell,j}} + \frac{r}{B_{r,j}} \right). ]
The available table does not provide consistent collective-startup measurements, so the hand calculation sets (\alpha_j=0) and uses scale-up bandwidth. This is a favorable lower bound. It is not innocuous: H20 EP16 spans two nodes and approximately half of its MoE traffic remains on NVLink while the rest goes inter-node. citeturn692601view2
[ t_{\rm roof}=\max(t_C,t_H,t_N) ]
[ t_{\rm iter}=\frac{t_{\rm roof}}{\eta}+t_0 ]
[ \boxed{ \widehat T=\frac{ba}{t_{\rm iter}} } ]
and the predicted latency is:
[ \widehat{\rm TPOT}=\frac{t_{\rm iter}}{a}=\frac{b}{\widehat T}. ]
Do not put achieved TPOT into the throughput equation. Since (T=b/\mathrm{TPOT}), doing that would simply restate the measured result. TPOT should be an output and an SLO check.
HBM capacity is a separate feasibility condition:
[ W_{\rm resident} + bL,N_L(512+64)s_{KV} + W_{\rm workspace} \leq M_{\rm HBM}. ]
const ARCH = Object.freeze({
active: 37e9,
layers: 61,
heads: 128,
qkDim: 192,
vDim: 128,
kvDim: 576,
moeLayers: 58,
topK: 8,
hidden: 7168,
});
export function decodeRoof(hw, op, etaRoof, t0 = 0) {
const attnFlops =
2 * ARCH.layers * ARCH.heads * (ARCH.qkDim + ARCH.vDim) * op.context;
const flopsPerPos = 2 * ARCH.active + attnFlops;
const kvPerScan =
ARCH.layers * ARCH.kvDim * op.kvBytes * op.context;
const fabricPerPos =
2 * ARCH.moeLayers * ARCH.topK * ARCH.hidden *
op.activationBytes * (op.dispatchScale ?? 1);
const tCompute = op.batch * op.targetPositions *
(op.computeSecondsPerPos ?? flopsPerPos / hw.flops);
const kvScans = op.kvScans ?? op.targetPositions;
const tHbm = (op.weightBytesIter +
op.batch * kvScans * kvPerScan) / hw.hbmBps;
const invFabric =
op.localFraction / hw.localFabricBps +
(op.remoteFraction ? op.remoteFraction / hw.remoteFabricBps : 0);
const tFabric = ARCH.moeLayers * (op.fabricRttSec ?? 0) +
op.batch * op.targetPositions * fabricPerPos * invFabric;
const tRoof = Math.max(tCompute, tHbm, tFabric);
const tIter = t0 + tRoof / etaRoof;
const tps = op.batch * op.acceptedTokens / tIter;
const residentKv = op.batch * op.context *
ARCH.layers * ARCH.kvDim * op.kvBytes;
const feasible = op.residentWeightBytes + residentKv +
op.workspaceBytes <= hw.hbmBytes;
return { tps, tpot: tIter / op.acceptedTokens,
tCompute, tHbm, tFabric, feasible };
}
| Platform/row | (b) | (L) | (q) | (a) | Weight traffic | KV dtype |
|---|---|---|---|---|---|---|
| H800 calibration | 96 | 4,989 | 1 | 1.00 | 37 GB | FP8 |
| H20 Pro | 32 | 4,096 | 3 | 1.85 | 37 GB | FP8 |
| H20 Base | 48 | 4,096 | 3 | 1.85 | 37 GB | FP8 |
| GB200 | 128 | average 3,000 | 1 | 1.00 | 18.5 GB, optimistic lower bound | FP8 |
| Ascend, 50 ms | 96 | 4,096 | 2 | 1.70 | 37 GB | BF16 assumed |
| Ascend, 15 ms | 8 | 4,096 | 2 | 1.70 | 37 GB | BF16 assumed |
Important qualifications:
At (L=4{,}989):
98.931\text{ GF/position}. ]
An FP8 KV scan is:
175.294\text{ MB/sequence}. ]
For (b=96,q=1):
4.797\text{ ms}. ]
16.068\text{ ms}. ]
3.193\text{ ms}. ]
So HBM is the roof:
[ t_{\rm roof}=16.068\text{ ms}. ]
The observed iteration time under (a=1) is:
51.892\text{ ms}. ]
Hence the one shared residual is:
0.30965 } ]
or 30.96% roof utilization.
This is not an MFU: it multiplies the active bottleneck, whether compute, HBM, or fabric.
At 4K context:
[ \Phi(4096)=94.468\text{ GF}, \qquad K_{\rm FP8}(4096)=143.917\text{ MB}. ]
30.638\text{ ms}. ]
\frac{37\text{ GB}+32(3)(143.917\text{ MB})} {4.0\text{ TB/s}} = 12.704\text{ ms}. ]
Using the supplied 900 GB/s scale-up bandwidth:
1.419\text{ ms}. ]
Compute is the roof:
\boxed{598\text{ tokens/s/GPU}}. ]
Measured: 675, so error is −11.4%.
[ t_C=45.958\text{ ms},\qquad t_H=14.431\text{ ms},\qquad t_N=2.129\text{ ms}. ]
Because the row is compute-bound, batch cancels in (ba/t_C):
\boxed{598}. ]
Measured: 714, so error is −16.2%.
For a 2K-input/2K-output run, use (L_{\rm avg}=3{,}000):
[ \Phi(3000)=88.991\text{ GF}, \qquad K_{\rm FP8}(3000)=105.408\text{ MB}. ]
Using the favorable 18.5 GB all-active-FP4 weight lower bound and 10 PF FP4 compute:
1.139\text{ ms}. ]
3.999\text{ ms}. ]
NVFP4 dispatch divides the nominal activation payload by four:
0.237\text{ ms}. ]
HBM is the roof:
\boxed{9{,}911}. ]
Measured: 10,108.4, so error is −2.0%.
That near-match should not be overinterpreted: 18.5 GB is the minimum physically possible active-weight traffic. The actual mixed-precision checkpoint is heavier.
At 4K with BF16 KV:
[ K_{\rm BF16}(4096)=287.834\text{ MB}. ]
12.060\text{ ms}. ]
28.833\text{ ms}. ]
3.258\text{ ms}. ]
\boxed{1{,}753}. ]
Measured: 1,943, so error is −9.8%.
Notably, the HBM time increases from about 20.2 ms without MTP to 28.8 ms with two KV scans, a 43% increase. That closely matches the paper’s measured approximately 44% MTP iteration-latency increase, although it does not prove the decomposition is unique. citeturn848796view4
Use the same physical configuration but (b=8):
13.002\text{ ms}, \qquad t_N=0.272\text{ ms}. ]
\boxed{324}. ]
Measured: 538, so error is:
[ \boxed{-39.8%}. ]
This is the load-bearing failure. At batch eight, the constant shared residual and simple “37 GB once per iteration” traffic assumption do not describe the platform.
Let:
[ e=\frac{\widehat T}{T}-1. ]
The calculator’s cost is inversely proportional to throughput, and its contribution margin is (M=1-C/R). Therefore, if (\widehat M) is the margin calculated from predicted throughput:
1-(1-\widehat M)\frac{\widehat T}{T}, ]
so:
(1-\widehat M)e }. ]
The following percentage-point values assume the throughput error applies to the entire relevant serving cost. For a blended workload where decode is only fraction (\omega_d) of direct cost, multiply the values by (\omega_d). citeturn525695view0
| Held-out row | Predicted | Measured | Throughput error | Margin error at 80% | Margin error at 90% |
|---|---|---|---|---|---|
| H20 Pro | 598 | 675 | −11.4% | −2.27 pp | −1.14 pp |
| H20 Base | 598 | 714 | −16.2% | −3.24 pp | −1.62 pp |
| GB200, optimistic FP4 bound | 9,911 | 10,108 | −2.0% | −0.39 pp | −0.20 pp |
| Ascend, 50 ms | 1,753 | 1,943 | −9.8% | −1.96 pp | −0.98 pp |
| Ascend, 15 ms | 324 | 538 | −39.8% | −7.96 pp | −3.98 pp |
Define a platform-level score as:
[ E_j=\max_{o\in j}|e_{jo}|. ]
Then:
At an 85% nominal margin, the worst-row downstream error is approximately 5.97 percentage points.
The tempting result is “16.2% worst” after looking only at H20, GB200, and Ascend’s throughput-oriented 50 ms row. That is not the council-compliant score: it excludes Ascend’s 15 ms observation from a held-out platform after seeing it.
The separate 1,303 tokens/s “neutral Ascend” figure should not be scored using the optimized batch-96/MTP constants. Doing so gives 1,753 vs 1,303, or +34.5%, but the comparison is invalid until the neutral row’s batch, context, MTP, EP placement, and precision are specified.
The favorable result is not stable.
The H800 batch is undisclosed. Holding every other assumption fixed:
| Assumed H800 batch | Fitted shared (\eta) | Worst error over all held-out rows |
|---|---|---|
| 80 | 35.22% | 31.5% |
| 96 | 30.96% | 39.8% |
| 128 | 25.64% | 50.1% |
All held-out predictions scale almost linearly with (\eta). Even the favorable batch-80 case remains above the 25% whole-platform gate.
The central calculation assumes no H800 MTP. DeepSeek’s model report says one additional token can have 85–90% acceptance and produce approximately a 1.8× TPS gain, but that does not establish whether the 1,850 production disclosure used it. citeturn848796view2
If H800 instead had (q=2,a=1.85):
[ \eta=21.97% ]
rather than 30.96%, and the Ascend 15 ms prediction falls to about 230 tokens/s, a −57.3% error.
Central assumption:
[ q=3,\qquad a=1.85 ]
for one base plus two drafted positions.
If the target-equivalent work is instead (q=2):
With (q=3), changing acceptance from 1.8 to 1.9 moves the Pro prediction only from 582 to 614. The large sensitivity is to the number of expensive target positions, not to the narrow acceptance range.
At average context 3K:
| Assumed active weight traffic | Prediction | Error |
|---|---|---|
| 18.5 GB, all active path FP4 lower bound | 9,911 | −2.0% |
| 23 GB mixed-precision approximation | 8,689 | −14.0% |
| 37 GB strict 8-bit counterfactual | 6,280 | −37.9% |
With 18.5 GB weights, changing average context from 2K to 4K moves the prediction from 11,532 to 8,690, or from +14.1% to −14.0%.
Consequently, the GB200 row cannot be a clean “8-bit held-out platform.” It is a different precision treatment that must either be modeled layer by layer or excluded from an apples-to-apples 8-bit gate.
At batch 96:
At batch eight:
The low-batch failure survives either KV assumption.
The retrospective calculation does this only when all H20 and Ascend rows are scored. On that basis, the worst platform error is 39.8%, so the approximately 25% gate fails.
Selecting only the 50 ms Ascend row would be row selection, not a whole-platform holdout.
The proposed equation has no per-platform (\eta): only one shared (\eta).
However, this condition is satisfied only if batch, MTP work, acceptance, precision mix, KV dtype, expert occupancy, topology fractions, and collective startup are fixed from independent configuration records or microbenchmarks before throughput is revealed. In the present dataset, several are unknown. Post-hoc choices among them would function as undeclared platform-specific fits.
For decode:
[ p=1,\qquad n_{\rm train}=1,\qquad 1<1\text{ is false}. ]
Adding the optional shared (t_0) would make (p=2), which is even less identifiable.
The only H800-only model satisfying this condition has (p=0). Setting (\eta=1) gives an ideal H800 prediction of approximately 5,975 tokens/s versus 1,850 measured, a +223% error. So the zero-parameter alternative is formally runnable but plainly fails.
Reported above:
There is none among H20, GB200, and Ascend. Their outcomes have already informed:
Therefore:
[ \boxed{\text{The preregistered held-out experiment is unrunnable on the current observations.}} ]
For a 4K prompt, the average attention history during prefill is approximately (L/2). A favorable compute-roof approximation is therefore:
84.234\text{ GF/input token}. ]
A full prefill roofline would also include FlashAttention I/O, prompt-token batch size, TP collectives, EPLB, and weight traffic. Unfortunately, the H800 4,026 row has batch and input length recorded as N/A, while the comparable Ascend rows use 16,384 total tokens and 4,096-token prompts. citeturn131213view0
Using the requested H800 fresh-prefill anchor anyway:
17.13%. ]
At the same prompt length, the FLOPs/token cancel, so predictions reduce to peak-compute ratios:
\boxed{602}. ]
H20 measured input throughput is approximately (16.5\text{k}/8=2{,}062.5) per GPU, giving −70.8% error. citeturn459091view0
\boxed{3{,}058}. ]
Against Ascend:
The prefill worst-platform error is therefore 70.8%, equivalent to approximately 14.16 pp of margin at 80% or 7.08 pp at 90%, under the same whole-cost convention.
Using the more comparable H800 profile result of 7,839 at 16K total tokens and 4K prompts would predict 1,172 on H20 and 5,954 on Ascend. That still misses H20 by 43%. The prefill residual also does not transfer.
Fitting separate decode and prefill residuals would mean (p=2) residuals from one H800 platform group, violating the parameter-count precondition even more clearly.
Freeze the present per-platform anchor-fit calculator. Keep its current language that anchored rows are interpolations rather than validated physics. The roofline can be added as an experimental diagnostic or uncertainty band, but not as the production replacement.
A valid future test needs:
The roofline is directionally better than scalar MFU because it correctly changes bottlenecks across H20, GB200, and Ascend. But with the available metadata it is still an operating-point reconstruction, not a transferable throughput law.