37,152 tensors, one flat landscape, one wrong index

The sensitivity map: 37,152 instruments, one flat landscape

We did not invent the recipe we started from. LibertAI published it, and our baseline (P1) is an exact replica of their card: ModelOpt 0.45.0, weight-only NVFP4, blocks of sixteen, 4-bit E2M1 weights with an FP8 scale per block and one FP32 scale per tensor, no calibration data, weights streamed shard by shard through CPU memory. Only the routed-expert feed-forward weights get squeezed — 37,152 tensors, about 97% of the parameters. The other 1,618 tensors stay in BF16: attention, the shared experts, the routers, the embeddings, the output head, the norms, the MTP head, and the first dense layers. That matches LibertAI’s 43-entry ignore list exactly. The one thing we added was a tap: every tensor, as it was quantized, wrote down how far the round trip moved it — cosine, relative error, and what happened to its scales — to a JSONL file.

Everyone who quantizes a mixture-of-experts model has the same instinct, and we had it too: some experts must be fragile, some layers must be sensitive, and the clever recipe keeps those in higher precision. We expected the tap to tell us which tensors to protect. It told us not to bother.

If the scale-and-grid mechanics behind those block scales are new to you — per-tensor versus per-channel versus group-wise scales, and why the bucket size is a tradeoff rather than a setting — Chapter 27 of Under The Hood builds them from scratch and is free to read.

The error landscape is flat. Not roughly flat — flat in a way that made us re-run the analysis looking for a bug. What the map does not show:

• Full-model per-tensor cosine spans only 0.99544–0.99604. The worst tensor is 1.07× the mean.
• The worst 1% of tensors hold 1.06% of total rel².
• 86% of error variance is idiosyncratic per tensor — not explained by layer, expert index, projection type, or any structural covariate we tested.
• No hot-expert effect: one-way ANOVA across the 288 expert indices gives p = 0.52, and between-index variance is 0.993× the sampling-noise null. Expert-index rankings between adjacent layers correlate at 0.0001.
• amax is a bad exemption selector: 14% overlap with an oracle worst-list.
• The best exemption list we could construct (worst-372 tensors by rel², kept in higher precision) buys −1.06% mean rel_l2 for +2.6% checkpoint size. Exempting all of layer 45 buys −1.23% for +5.9%. Both are a poor trade.

One genuine structural signal survived: MTP layer 45 is a uniform mild outlier (1.057× mean error, a level shift with no tail), and all 245 scale clamps in the model live there. Whether layer 45 deserves BF16 experts is a functional question (MTP acceptance), not a weight-space one. On our serving stack MTP acceptance measured 51.6% of draft tokens, accept length 2.07/4 (cumulative /metrics over the gate runs), with layer-45 experts quantized and the head in BF16.

A footnote that cost us an afternoon. LibertAI’s card reports round-trip rel err 0.0925 and cosine 0.99665. Our replica matched their error (0.0921). But those two numbers cannot both be per-tensor averages: for this statistic rel² ≈ 2(1−cos), and 0.0925 implies cos ≈ 0.99575, not 0.99665. We spent that afternoon assuming our quantizer was wrong. It was not; their cosine was aggregated some other way. Our P1 pair (0.0921 / 0.99576) satisfies the identity.

The recipe decision: fix the quantizer, not the exemption list

If there is no tail to protect, the only place left to find error is in the quantizer itself. So we rewrote ModelOpt’s stock quantizer in plain code, checked it produced bit-identical output before changing anything, and tried levers on a stratified sample of 872 tensors: the worst 372, 400 at random, and 100 from layer 45.

Table 1. Lever bake-off on the n=872 stratified sample, all variants data-free and CPU-only. The per-block MSE scale search won on 872/872 tensors (Wilcoxon p ≈ 1e-144), tail included (worst tensor rel_l2 0.09562 → 0.08488).

The winner is boring and old: instead of setting each block’s scale from its largest value, try nine multipliers on that scale and keep whichever reconstructs the block best. We built the full checkpoint with it (P2) and the numbers were lovely: mean rel² 0.8487% → 0.6335% (−25.4%), with 37,152 of 37,152 tensors improved. Not most of them. All of them.

Then we ran the behavioral gates and the model was worse. The gates are a small battery of things we actually use this model for. One hands it a codebase and asks for every call site of a function across files, at 32k and again at 100k tokens of context; the stock checkpoint went 6/6 on both, and P2 went 0/4. A tool-call probe went 0/2. A quarter less weight error, and the model got worse at its job. We had written down before building P2 that this was the outcome we were most afraid of — that better weights would not mean better behavior — but predicting a failure is not the same as understanding it.

The explanation came from looking at what the scale search was doing to individual blocks. Multipliers below 1.0 shrink a block’s scale, which makes the grid finer for the bulk of its values (hence the lower error) but leaves the block’s largest weights off the grid, clipped. On this model the clipping was diffuse — ~4% of L2 mass at ≤20% depth — but present on 8.8–20.5% of blocks in every tensor. A little deep clipping, everywhere. The objective was delighted to trade a few large weights for many precise small ones. The model was not. So P3 forbids multipliers below 1.0 (mse_ge1): no clipping is possible by construction, at a priced cost of ~2.5 points of the gain. This was the experiment we had pre-registered to decide the question: if the gates came back, clipping was the cause.

Table 2. Full-model weight-space aggregates, all 37,152 quantized tensors, round-trip vs the BF16 master. P3 keeps nearly all of P2’s gain with zero deep clipping. The deep-clipping hypothesis was supported when lc-01 recovered 0/4 → 2/3 + 2/3 and tc-10 0/2 → 1/3 on the gates.

The corruption bug: one global scale for a two-scale GEMM

If you read one section of this note, read this one. Serve this family of checkpoints on an unpatched vLLM and the model will, now and then, emit U+FFFD — the Unicode replacement character, the little diamond with a question mark in it — and tool calls come back malformed. It happens mostly in Korean, Tamil, Japanese, anything that takes more than one byte per character, and almost never in English, which is exactly why an English smoke test passes. The server logs nothing. The probe that catches it is tonyd2wild’s, who first measured the pattern on this model family: 4/9/8 events across three passes on ModelOpt-packed checkpoints versus 0/0/0 on a compressed-tensors control. On our own P3, unpatched: 8 U+FFFD events across 6 passes (Korean tool-call probe 1/3/4; English code+JSON 0/0/0).

The cause is one line. Remember the two-level scales from the top of this note, one per block and one per tensor? Each expert has a “gate” projection and an “up” projection, and vLLM fuses the two into one matrix for speed. When ModelOptNvFP4FusedMoE.process_weights_after_loading (in vllm/model_executor/layers/quantization/modelopt.py, ~line 1527) repacks that fused matrix for the marlin GEMM, it grabs the global scale like this:

w13_weight_scale_2 = layer.w13_weight_scale_2[:, 0]

Index zero. The gate (w1) projection’s scale, applied to both halves. vLLM even knows this might be wrong — a warning_once fires when the scales differ (“w1_weight_scale_2 must match w3_weight_scale_2. Accuracy may be affected.”) — and then carries on. The up (w3) half is dequantized mis-scaled by ws2_gate / ws2_up, separately for every expert. Whether the two scales happen to match is a property of the checkpoint, not the code, and you can count it without touching a GPU:

Table 3. Gate/up global-scale census, measured with a plain-python safetensors header reader over both checkpoints; no GPU involved. Any ModelOpt NVFP4 checkpoint quantized with separate gate/up projections and per-tensor amax scaling hits this bug — including the LibertAI checkpoint.

Numeric proof, local (BF16 master + ModelOpt dequant, no serving), layer 10 expert 0 (gate/up scale ratio 1.23), up-proj round-trip rel_l2:

Table 4. The mis-scaling is worth ~2.7× the up-projection’s quantization error on this expert. Controls behaved as predicted: expert 2 (ratio 0.93): 0.1124 → 0.0924 compensated; expert 1 (ratio 1.0): unaffected.

The fix is a ~20-line hunk ahead of the repack: rescale the up half’s FP8 block scales by ws2_up / ws2_gate, clamped to the FP8-E4M3 max (448). It ships as patches/modelopt_gscale_fixed.py, bind-mounted over the image’s modelopt.py — no image rebuild. Boot-time stats on our checkpoint: compensation ratio mean 1.0197 / max 2.0000 per layer, clamps ≈ 0.03% of up-half scales. Two implementation notes from getting it right: the fused scale tensor is 3-D and TP-sharded, so the half-split must be rank-aware (our first version failed on exactly this), and requantizing both halves to max(ws2_gate, ws2_up) is an untested alternative.

Table 5. The battery covered the original Korean and English probes at 10 passes each, eight additional script families (Japanese, Chinese, Tamil, Arabic, Russian, Hindi, emoji-heavy, code-switching) at 3 passes each, and two 8,000-token Korean long-form generations (2/2 clean). Post-fix outputs were verified non-vacuous — real structured Korean, coherence intact.

Zero is easy to fake with an empty string, so we read the outputs; the post-fix Korean is real, structured, coherent Korean. And because changing the numerics could have changed behavior, we re-ran the gates on the patched build against the same probes, budgets, and grader: lc-01-32k 2/3 → 3/3, lc-01-100k 2/3 → 1/3 (noise at temp 1.0 with three samples), tc-10 1/3 → 2/3, ce-01 0/2 → 0/2. Neutral to slightly better, with the corruption gone. The patch is on by default.

We reported the root cause, both proofs, and the fix upstream in vllm#54150, answering the reporter’s open question — damaged weights (A) versus a wrong vLLM ModelOpt path (B) — with (B). The original report was on SM120 (4× RTX PRO 6000, same vLLM dev commit, 86 U+FFFD/6 runs on the LibertAI checkpoint and 94/6 on dealignai’s, 0/6 on RedHatAI); our comment extends it to SM121. The mechanism is not GB10-specific — the mis-scaling happens at repack, before any kernel runs — but Blackwell marlin NVFP4 MoE is where this code path executes.

Ecosystem context

None of this happened alone, and the people whose work we leaned on deserve a straight account of how ours compares.

• LibertAI published the W4A16 recipe this work replicates as its P1 baseline — the partition, the format, the data-free CPU streaming approach. Their checkpoint is affected by the same repack bug: the corruption itself is documented in vllm#54150 and the mechanism applies by construction, but we did not re-run their scale census before their weights were removed from our cluster, so that number for their checkpoint is inferred, not measured.
• RedHatAI’s compressed-tensors W4A4 checkpoint shares gate/up scales by construction and is immune; we verified it clean (0/6) on our own hardened stack, replicating the issue reporter’s control. Their card lists GPQA 90.57 / AIME25 86.67 / MATH-500 94.87 / GSM8K-Plat 97.74 (third-party numbers, no baseline column). It has its own trade-offs: W4A4, and on our stack a chat-parser quirk on long-code prompts that we did not resolve before release.
• inco.ai shipped the day-0 NVFP4 checkpoint and the DFlash 2 drafter, reporting FP8-vs-NVFP4 accuracy parity across their suite (GPQA 91.1→91.2, AIME 94.3→95.1, MATH-500 95.6→95.2, HLE 35.9→35.2, AA-LCR 73.6→73.0) and up to 4.4× end-to-end throughput at concurrency 1 versus native FP8 autoregressive decoding (their numbers, from their launch post, on their engine). DFlash 2 is a block-diffusion drafter — lossless by rejection sampling, ~2.3 GiB — and community measurements put its acceptance length at 5.3–5.86 versus MTP’s 4.26–5.06 (third-party, SGLang on GB300); because it consumes token IDs, its acceptance survives target quantization. On our NVFP4 stack MTP itself held up (51.6% draft-token acceptance), so we shipped with MTP and did not gate on DFlash 2.
• EXL3 (Mia-AiLab’s TR3-4bpw recipe) was the decisive sparring partner. Its published KLD advantage (0.0246 versus NVFP4 0.0605 — third-party panel by malaiwah, likely cross-lane since that panel mixes KV-cache formats, and we have no KLD harness of our own) did not survive into gate behavior on our battery (Table 6).

Table 6. Behavioral gates head-to-head, same probes, budgets, and grader; n=2–3 per cell. † Under the hardened grader (zero-content-at-cap check), Murai-P3’s ce-01 cell revises from a historical 1/2 to 0/2 — the pass was a zero-content artifact. EXL3’s dj-01 failures were total reasoning starvation: >200k characters of reasoning, zero content, at both temp 1.0 and 0.7. EXL3’s real edges are operational (DFlash2 decode speed, 1M context, 1.75M-token KV pool), not output quality on this battery.

• voska’s W4A16 card independently converged on our central negative result: they tried “MSE-optimal clip search” and discarded it — “~9% lower weight L2, no better in practice; optimising weight-space error is the wrong objective at 4 bits” (their card, third-party). Our gates add the mechanism nuance: the problem was not weight-space optimization per se, but the deep-clipping side effect of one specific lever. Remove the clipping and the weight-space gain becomes behavior-neutral-to-positive instead of harmful.
• KVarN (huawei-csl, arXiv:2606.03458), a calibration-free KV-cache quantizer — 4-bit keys, 2-bit values, Hadamard rotation plus Sinkhorn normalization — reporting 2.77× cache capacity at accuracy parity on GLM-4.7-Flash, looked like the obvious next lever, and nobody had tried composing it with NVFP4 weights. It does not run on a GB10. Its sparse-MLA path bottoms out in a compiled flash_mla_sparse_fwd kernel whose body uses tcgen05 tensor-memory instructions the GB10 does not have; the runtime gate accepts only SM90a and SM100f, there is no Triton fallback for sparse MLA, and adding sm_121 to the build yields a silent empty stub. A one-hour spike settled it. Re-evaluate when FlashInfer or FlashMLA ships a sparse-MLA kernel for sm_12x.
• The serving stack also carries the community’s GB10 runtime fixes as bind-mounts: tonyd2wild’s SM121 top-k >24K hard-kill fix and vcruz305’s K-pool tail OOB fix. Serving this model long-context on this vLLM lineage without them is not recommended.

What we are taking away from the week

The map was worth building because it killed the plan. We instrumented 37,152 tensors expecting to find the ones to protect, and the answer was that there are none. That is a cheap thing to learn from a JSONL file and an expensive thing to learn by shipping a checkpoint with an exemption list that buys nothing.

P2 improved every tensor in the model and got worse at the job. The quantizer optimizes weight error because weight error is what it can compute, and that works right up until the lever you pull has a side effect the number cannot see. The behavioral gates are not a check you run after the quantization. They are the experiment.

And we spent most of the week inside the quantizer while the biggest quality problem on the whole path was [:, 0] in a function nobody here had read. The single most useful thing we did was run someone else’s probe against our own checkpoint and take the answer seriously when it came back dirty.

What you can take home today: a 181.3 GiB W4A16 checkpoint that serves TP=2 on 2× DGX Spark with the corruption fixed (0 U+FFFD across 46 passes / ~90k tokens where the unpatched stack emitted 8 in 6), weight-space error 22.8% below the stock recipe with zero deep clipping, MTP intact, and a regression probe in the repo so you can check all of it on your own stack in minutes.

What we are not claiming

If you serve any ModelOpt-packed NVFP4 MoE checkpoint on vLLM — ours, LibertAI’s, anyone’s — run the probe. If it comes back dirty, the patch is 20 lines and the mechanism, the proof, and the upstream trail are all in the repo.

Artifacts

Related

Raw markdown (ARTICLE.md) · Figure 1 · Figure 2 · Figure 3 · Author’s explainer on ramchandk.com