Methods note: leave-one-anchor-out validation of the effective-MFU abstraction

Run 2026-07-10, as prescribed by the external review (the empiricist's "load-bearing experiment"). Code: tests/loao.mjs in the project tree. Predictions were fixed before comparison.

Question

The calculator compresses each accelerator's serving behavior into one scalar — an "effective decode MFU" against dense 8-bit FLOPS. Is that a predictive abstraction (one coefficient transfers across hardware), or an interpolating one (each platform needs its own fitted coefficient)?

Design

Fit a single global decode MFU using only the DeepSeek H800 production anchor (1,850 tok/s/GPU on a 37B-active model, 1.98 PF dense FP8 ⇒ 6.91%). Predict every other platform's published decode measurement from its spec sheet and that one coefficient. Pass bar (set in advance by the review): central error ≤ ~25%.

Result

Mean |error| 37%, worst 59% — the abstraction FAILS the transfer test.

Interpretation and consequences (adopted in methodology v2)

1. A scalar compute-MFU does not transfer across platforms, because decode is memory/interconnect-bound: the H20's tiny FLOPS denominator makes its fitted "MFU" (17%) 2.5× the H800's (7%) for the same physical workload. The coefficient absorbs bandwidth, batch regime, serving stack, and latency target — it is not a hardware constant.
2. The calculator's per-platform MFU values are therefore anchor fits — each reproduces its own published measurement, and the page no longer describes this as "calibration" in the predictive sense. Reproducing the datum used to select a coefficient is an identity, not a validation.
3. Platforms without a published anchor carry materially lower confidence: TPU v7 Ironwood, Trainium 2/3, and the Rubin projection have no public serving measurement for a frontier MoE and their MFU values are analyst estimates. This is now stated in the calculator.
4. The model's honest domain of validity: short-to-moderate context, throughput-oriented serving, interpolation near the anchored operating points. Context length, TTFT/TPOT targets, KV-cache lifecycle, and speculative-decoding acceptance are not modeled; the interactivity multipliers (1.0/0.70/0.35) are a coarse stand-in for a latency curve that published data (CloudMatrix: 1,943 → 538 tok/s from 50 ms → 15 ms TPOT) shows is steep.

What would upgrade the model

A roofline formulation (compute + HBM + interconnect terms with per-platform physical constants) fit jointly on all anchors with one shared efficiency residual — then re-run this experiment. Until that passes, cross-platform margin comparisons inherit anchor-fit uncertainty, and the per-provider ranges in §10 should be read accordingly.