September 2, 2026

### GLM-5.3-Flash-NVFP4-Murai — Weight-Space Error Across 37,152 Quantized Tensors

Flat error landscape

Per-tensor round-trip error vs BF16 master

All 37,152 routed-expert FFN tensors

cosine span

0.99544–0.99604

Worst 1% of tensors, share of total rel²

1.06%

[FIGURE 1: scatter/line of per-tensor round-trip rel_l2 across all 37,152 quantized tensors ordered by layer, showing the flat distribution with the mild uniform level shift at MTP layer 45]

Figure 1. Round-trip weight-space error for every quantized tensor in GLM-5.3-Flash-NVFP4-Murai, measured against the BF16 master on the quant workstation (ModelOpt 0.45.0, CPU-only shard streaming, data-free). The error landscape is flat: no hot experts, no hot layers except a mild uniform shift at the MTP layer.

## Overview

GLM-5.3-Flash is a ~320B-parameter MoE (18B active), released by Z.ai on August 26, 2026 under MIT. This article accompanies **GLM-5.3-Flash-NVFP4-Murai**, a weight-only NVFP4 (W4A16) quantization — 181.3 GiB across 120 safetensors shards — that serves tensor-parallel (TP=2) on 2× NVIDIA DGX Spark nodes (GB10, sm_121, 128 GB unified memory each) over a 200G ConnectX-7 fabric. It does not fit on one node at this precision.

Three things came out of this project, and each is documented with measurements rather than impressions:

1. **A sensitivity map across all 37,152 routed-expert tensors** — the first per-tensor quantization sensitivity measurement at this scale that we are aware of — with a negative result that redirects the field's usual instinct: per-expert exemption lists are a poor trade, because the error landscape is flat.
2. **A data-free recipe ("Murai-P3")** that cuts mean weight-space rel² from 0.849% to 0.655% versus the stock ModelOpt recipe, with zero deep clipping — and a behavioral gate suite showing which of those weight-space gains did and did not cash out.
3. **A serving-time corruption fix.** Unpatched vLLM dequantizes this checkpoint lineage with the wrong global scale on 69.1% of expert gate/up pairs, emitting U+FFFD replacement characters in multibyte scripts. We root-caused it to a single-gscale assumption in vLLM's ModelOpt marlin repack, validated a ~20-line compensation patch (8 U+FFFD events in 6 passes → 0, then 0 across a 46-pass / ~90k-token battery), and reported it upstream as [vllm-project/vllm#54150 (comment)](https://github.com/vllm-project/vllm/issues/54150#issuecomment-5503200584).

Weights live at [murai-labs/GLM-5.3-Flash-NVFP4-Murai](https://huggingface.co/murai-labs/GLM-5.3-Flash-NVFP4-Murai); the patch, regression probe, launcher, and full writeup live at [Murai-Labs/GLM-5.3-Flash-NVFP4-Murai](https://github.com/Murai-Labs/GLM-5.3-Flash-NVFP4-Murai). Every number below traces to project logs; third-party numbers are labeled as such, with the measurement lane they were taken in.

**Measurement conditions, stated once:** quantization from the BF16 master (598.5 GiB, 120 shards — verified bytewise to be the master, not an upcast of the FP8 release); serving on image `radixark/vllm-glm53-flash:sm121-v8` (vLLM `0.1.dev20051+g487ecf187`, FlashInfer `0.6.18.dev20260819`), TP=2, marlin MoE backend, fp8_e4m3 KV cache (672,606-token pool), `--max-model-len 262144`, `--block-size 2304`, `--gpu-memory-utilization 0.85`, MTP speculative decode with `num_speculative_tokens=4`. Behavioral gates run at temp 1.0 / top_p 0.95, n=2–3 per probe unless noted; the corruption probe runs at temp 0, non-streaming.

## The sensitivity map: 37,152 instruments, one flat landscape

The baseline (P1) is an exact replication of LibertAI's published card: ModelOpt 0.45.0, weight-only NVFP4, group-16, E2M1 weights + FP8-E4M3 block scales + FP32 global scale, data-free, CPU-only shard streaming. The partition quantizes only routed-expert FFN weights — 37,152 tensors, ~97% of parameters — and passes 1,618 tensors through in BF16 (attention, shared experts, routers, embeddings, lm_head, norms, the MTP head, the first dense layers), matching LibertAI's 43-entry ignore list with zero XOR violations. Every quantized tensor was instrumented at quantize time: round-trip cosine, rel_l2, scale statistics, written to JSONL.

The map's headline is what it does **not** show:

- Full-model per-tensor cosine spans only **0.99544–0.99604**.
- The worst 1% of tensors hold **1.06%** of total rel².
- **86%** of error variance is idiosyncratic per tensor — not explained by layer, expert, or any structural covariate we tested.
- No hot-expert effect (ANOVA p=0.52). amax is a bad exemption selector (14% overlap with an oracle worst-list).
- The best exemption list we could construct (worst-372 tensors by rel², kept in higher precision) buys **−1.06%** mean rel_l2 for **+2.6%** checkpoint size. A poor trade.

[FIGURE 2: cumulative share of total rel² versus fraction of tensors excluded, showing the near-linear accumulation that makes exemption lists futile]

Figure 2. Error mass accumulates nearly linearly in the number of exempted tensors — there is no tail worth exempting. Measurement: P1 instrumentation over all 37,152 tensors.

One genuine structural signal survived: MTP layer 45 is a uniform mild outlier (1.057× mean error, a level shift with no tail), and all 245 scale clamps in the model live there. Whether layer 45 deserves BF16 experts is a functional question (MTP acceptance), not a weight-space one — and on our serving stack MTP acceptance measured **51.6% of draft tokens, accept length 2.07/4** (cumulative `/metrics` over the gate runs), with layer-45 experts quantized and the head in BF16.

A note on published numbers, because this map lets us check them: LibertAI's card reports round-trip rel err 0.0925 and cosine 0.99665. Our replica's rel_l2 matches theirs (0.0921), but for per-tensor statistics rel² ≈ 2(1−cos), and their pair is not self-consistent under that identity (0.0925 would imply cos ≈ 0.99575, not 0.99665). Their cosine was computed with some other aggregation; we say so plainly because we initially burned time chasing that gap. Our P1 pair (0.0921 / 0.99576) is self-consistent.

## The recipe decision: fix the quantizer, not the exemption list

With exemptions priced out, we ran a data-free lever bake-off on an n=872 stratified sample (worst-372 + 400 random + 100 layer-45 tensors), with the stock quantizer re-implemented and verified bitwise-equal to ModelOpt's before touching anything.

[TABLE 1]

| variant | mean rel² | Δ vs stock | cost |
|---|---|---|---|
| stock (ModelOpt 0.45.0) | 0.8755% | — | 1× |
| s46 (4/6 adaptive block scaling, arXiv:2512.02010, paper-exact) | 0.7238% | −17.3% | 2× |
| **mse (per-block scale search, 9-point grid 0.70–1.75)** | **0.6659%** | **−23.9%** | 8.7× |
| s46_mse | 0.6685% | −23.6% | 8.7× |

Table 1. Lever bake-off on the n=872 stratified sample, all variants data-free and CPU-only. The per-block MSE scale search won on 872/872 tensors (Wilcoxon p ≈ 1e-144), tail included (worst tensor rel_l2 0.09562 → 0.08488).

The MSE build (P2) delivered full-model: mean rel² **0.8487% → 0.6335% (−25.4%)**, with **37,152 of 37,152 tensors improved**. And then the behavioral gates regressed: lc-01 (long-code cross-file call-site discovery at 32k and 100k context) went 0/4 against the baseline's 6/6 + 6/6, and tc-10 (a tool-call probe) went 0/2. Weight-space improvement did not trivially cash out — exactly the falsification scenario we had pre-registered for.

Clip analysis found the mechanism: the MSE grid's downward scale moves (< 1.0 multipliers) cause **deep clipping** — diffuse, ~4% of L2 mass at ≤20% depth, but present on 8.8–20.5% of blocks in every tensor. P3 restricts the grid to ≥ 1.0 (`mse_ge1`), eliminating deep clipping by construction, at a priced cost of ~2.5 points of the rel² gain. P3 was the pre-registered discriminating experiment: if the gates recovered, clipping was the cause.

[TABLE 2]

| checkpoint | mean rel² | mean rel_l2 | mean cos | notes |
|---|---|---|---|---|
| Murai-P1 (stock replication) | 0.849% | 0.0921 | 0.99576 | matches LibertAI's published rel err |
| Murai-P2 (MSE scale search) | 0.634% | 0.0796 | 0.996829 | −25.4% rel²; gates regressed |
| **Murai-P3 (mse_ge1 — shipped)** | **0.655%** | **0.0809** | **0.996727** | zero deep clipping; 245 clamps, 0 zeroed blocks |

Table 2. Full-model weight-space aggregates, all 37,152 quantized tensors, round-trip vs the BF16 master. P3 keeps nearly all of P2's gain with zero deep clipping; the deep-clipping hypothesis was supported when lc-01 recovered 0/4 → 2/3 + 2/3 and tc-10 0/2 → 1/3 on the gates.

## The corruption bug: one global scale for a two-scale GEMM

The bigger find was not in the weights at all. Serving this checkpoint lineage on unpatched vLLM produces **U+FFFD replacement characters and tool-call desync** — worst in multibyte scripts, invisible in ASCII, and silent server-side (nothing in the logs). The probe method is tonyd2wild's, who first measured the split on this model family (4/9/8 events across three passes on ModelOpt-packed checkpoints versus 0/0/0 on a compressed-tensors control). On our Murai-P3 serving, unpatched: **8 U+FFFD events across 6 passes** (Korean tool-call probe 1/3/4; English code+JSON 0/0/0).

The mechanism, proven end-to-end:

In `vllm/model_executor/layers/quantization/modelopt.py`, `ModelOptNvFP4FusedMoE.process_weights_after_loading` (~line 1527) repacks the fused [gate;up] expert weights for the marlin GEMM with **one** global scale:

```python
w13_weight_scale_2 = layer.w13_weight_scale_2[:, 0]
```

That is gate's (w1) global scale, applied to both halves. A `warning_once` fires when the scales differ ("w1_weight_scale_2 must match w3_weight_scale_2. Accuracy may be affected."), but the repack proceeds, and the up (w3) half is dequantized mis-scaled by `ws2_gate / ws2_up` per expert. Whether gate and up share one scale is a property of the **checkpoint**, not the code:

[TABLE 3]

| checkpoint | producer | gate/up expert pairs | pairs sharing one scale |
|---|---|---|---|
| Murai-P3 (ours) | ModelOpt 0.45.0, per-tensor amax, W4A16 | 12,384 (43 layers incl. MTP 45) | **30.9%** — mismatch on 69.1%; ratio mean 1.1454, median 1.077, p90 1.286, max 10.0 |
| RedHatAI/GLM-5.3-Flash-NVFP4 | llm-compressor (compressed-tensors, W4A4, group-16) | 12,096 (42 layers, no MTP) | **100.0%** — shared by construction |

Table 3. Gate/up global-scale census, measured with a plain-python safetensors header reader over both checkpoints; no GPU involved. Any ModelOpt NVFP4 checkpoint quantized with separate gate/up projections and per-tensor amax scaling hits this bug — including the LibertAI checkpoint.

Numeric proof, local (BF16 master + ModelOpt dequant, no serving), layer 10 expert 0 (gate/up scale ratio 1.23), up-proj round-trip rel_l2:

[TABLE 4]

| condition | up-proj rel_l2 |
|---|---|
| correct dequant | 0.0929 |
| single-gscale repack (unpatched vLLM behavior) | 0.2549 |
| single-gscale + compensation (the fix) | 0.0929 — restored exactly |

Table 4. The mis-scaling is worth ~2.7× the up-projection's quantization error on this expert. Controls behaved as predicted: expert 2 (ratio 0.93): 0.1124 → 0.0924 compensated; expert 1 (ratio 1.0): unaffected.

**The fix** is a ~20-line hunk ahead of the repack: rescale the up half's FP8 block scales by `ws2_up / ws2_gate`, clamped to the FP8-E4M3 max (448). It ships as `patches/modelopt_gscale_fixed.py`, bind-mounted over the image's `modelopt.py` — no image rebuild. Boot-time stats on our checkpoint: compensation ratio mean 1.0197 / max 2.0000 per layer, clamps ≈ 0.03% of up-half scales. Two implementation notes from getting it right: the fused scale tensor is 3-D and TP-sharded, so the half-split must be rank-aware (our first version failed on exactly this), and requantizing both halves to `max(ws2_gate, ws2_up)` is an untested alternative.

Runtime proof — same checkpoint, same probe, only repack numerics changed:

[TABLE 5]

| measurement | unpatched | patched |
|---|---|---|
| U+FFFD events, Korean ×3 + English ×3 (temp 0) | **8 / 6 passes** | **0 / 6 passes** |
| Full battery: 46 passes / ~90k tokens | — | **0 U+FFFD, 0 surrogates, 0 control chars, 0 repetition loops** |

Table 5. The battery covered the original Korean and English probes at 10 passes each, eight additional script families (Japanese, Chinese, Tamil, Arabic, Russian, Hindi, emoji-heavy, code-switching) at 3 passes each, and two 8,000-token Korean long-form generations (2/2 clean). Post-fix outputs were verified non-vacuous — real structured Korean, coherence intact.

[FIGURE 3: bar chart of U+FFFD events per probe pass, unpatched (8 events / 6 passes) versus patched (0/6), with the 46-pass battery shown as a third zero bar]

Figure 3. U+FFFD corruption on the ModelOpt marlin path before and after the gscale compensation, temp 0, non-streaming. Probe method after tonyd2wild.

A fix that changes numerics could change behavior, so we re-ran a gates spot-check on the patched build against the same probes, budgets, and grader: lc-01-32k 2/3 → 3/3, lc-01-100k 2/3 → 1/3 (within temp-1.0 noise), tc-10 1/3 → 2/3, ce-01 0/2 → 0/2. **Behavior-neutral to slightly better.** Combined with 8/6 → 0/6 → 0/46, the fix is pure win, and it is the ship default.

We reported the root cause, both proofs, and the fix upstream: [vllm#54150#issuecomment-5503200584](https://github.com/vllm-project/vllm/issues/54150#issuecomment-5503200584), answering the reporter's open question — damaged weights (A) versus a wrong vLLM ModelOpt path (B) — with **(B)**. The original report was on SM120 (4× RTX PRO 6000, same vLLM dev commit, 86 U+FFFD/6 runs on the LibertAI checkpoint and 94/6 on dealignai's, 0/6 on RedHatAI); our comment extends it to SM121. The mechanism is not GB10-specific — the mis-scaling happens at repack, before any kernel runs — but Blackwell marlin NVFP4 MoE is where this code path executes.

## Ecosystem context

This release exists inside a fast-moving community, and several lanes deserve explicit credit and honest comparison:

- **LibertAI** published the W4A16 recipe this work replicates as its P1 baseline — the partition, the format, the data-free CPU streaming approach. Their checkpoint is affected by the same repack bug (69.1%-class mismatch is a property of the ModelOpt packing, not of our recipe) (unverified — we measured the mismatch on our checkpoint; for LibertAI's, the corruption itself is documented in vllm#54150 and the mechanism applies by construction, but we did not re-run their scale census before their weights were removed from our cluster).
- **RedHatAI**'s compressed-tensors W4A4 checkpoint shares gate/up scales by construction and is immune; we verified it clean (0/6) on our own hardened stack, replicating the issue reporter's control. Their card lists GPQA 90.57 / AIME25 86.67 / MATH-500 94.87 / GSM8K-Plat 97.74 (third-party numbers, no baseline column on the card). It has its own trade-offs: W4A4, and on our stack a chat-parser quirk on long-code prompts that we did not resolve before release.
- **inco.ai** shipped the day-0 NVFP4 checkpoint and the DFlash 2 drafter, reporting FP8-vs-NVFP4 accuracy parity across their suite (GPQA 91.1→91.2, AIME 94.3→95.1, MATH-500 95.6→95.2, HLE 35.9→35.2, AA-LCR 73.6→73.0) and up to 4.4× end-to-end throughput at concurrency 1 versus native FP8 autoregressive decoding (their numbers, from [their launch post](https://inco.ai/blog/glm-5-3/), measured on their engine). DFlash 2 is a block-diffusion drafter — lossless by rejection sampling, ~2.3 GiB — and community measurements put its acceptance length at 5.3–5.86 versus MTP's 4.26–5.06 (third-party, SGLang on GB300); because it consumes token IDs, its acceptance survives target quantization, which makes it the ecosystem's standard mitigation for quant-degraded MTP. On our NVFP4 stack MTP itself held up (51.6% draft-token acceptance), so we shipped with MTP and did not gate on DFlash 2.
- **EXL3 (Mia-AiLab's TR3-4bpw recipe)** was the decisive sparring partner. Its published KLD advantage (0.0246 versus NVFP4 0.0605 — third-party panel by malaiwah, and likely cross-lane: that panel mixes KV-cache formats, and we have no KLD harness of our own) did **not** survive into gate behavior on our battery:

[TABLE 6]

| probe (max arm, temp 1.0) | LibertAI NVFP4 | Murai-P2 | Murai-P3 | EXL3 TR3-4bpw |
|---|---|---|---|---|
| lc-01-32k (long-code) | 6/6 | 0/4 | 2/3 | 2/3 |
| lc-01-100k (long-code) | 6/6 | 0/4 | 2/3 | 1/3 |
| tc-10 (tool-call) | pass | 0/2 | 1/3 | 2/3 |
| ce-01 (code-exec) | starved | 2/2 | 1/2 † | 0/2 |
| dj-01 (diorama @64k) | 0/5 | only working scene ever rendered | 0/3 (1 unrenderable) | 0/3 (zero-content ×3) |

Table 6. Behavioral gates head-to-head, same probes, budgets, and grader; n=2–3 per cell. † Under the hardened grader (zero-content-at-cap check), Murai-P3's ce-01 cell revises from 1/2 to 0/2 — the historical pass was a zero-content artifact. EXL3's dj-01 failures were total reasoning starvation: >200k characters of reasoning, zero content, at both temp 1.0 and 0.7. EXL3's real edges are operational (DFlash2 decode speed, 1M context, 1.75M-token KV pool), not output quality on this battery.

- **voska**'s W4A16 card independently converged on our central negative result: they tried "MSE-optimal clip search" and discarded it — "~9% lower weight L2, no better in practice; optimising weight-space error is the wrong objective at 4 bits" (their card, third-party). Our gates add the mechanism nuance: the problem was not weight-space optimization per se, but the deep-clipping side effect of one specific lever — remove the clipping and the weight-space gain becomes behavior-neutral-to-positive instead of harmful.
- The serving stack also carries the community's GB10 runtime fixes as bind-mounts: tonyd2wild's SM121 top-k >24K hard-kill fix and vcruz305's K-pool tail OOB fix. Serving this model long-context on this vLLM lineage without them is not recommended.

## The bottom line

A week ago the open questions were whether a measured recipe could beat the stock NVFP4 quant of a 320B MoE on behavior rather than perplexity, and whether weight-space error means anything at 4 bits. The answers, measured: the sensitivity map says the flat landscape kills exemption-list strategies; the recipe arc says weight-space gains are real but commute into behavior only when the lever's side effects are removed; and the corruption saga says the largest output-quality bug in this serving path was never in the weights at all — it was one wrong index (`[:, 0]`) in a repack function.

What you can take home today: a 181.3 GiB W4A16 checkpoint that serves TP=2 on 2× DGX Spark with corruption fixed (0 U+FFFD across 46 passes / ~90k tokens where the unpatched stack emitted 8 in 6), weight-space error 22.8% below the stock recipe with zero deep clipping, MTP intact, and a regression probe in the repo so you can verify all of it on your own stack in minutes.

What we are not claiming: this is not a benchmark suite. Gate cells are n=2–3 at temp 1.0 and single-cell differences are sampling noise. ce-01-class code execution is 0/2 on this checkpoint under the hardened grader — P2 passed 2/2 there, so treat that workload as at-risk. Very long single-shot generation (the 64k diorama probe) is unproven on P3 — the only checkpoint that ever rendered that scene was P2, once. Historical gate numbers predate the runtime hardening and cannot be retro-attributed. We have no KLD measurement of our own. And this W4A16 checkpoint carries no activation scales: it is validated only on the marlin MoE backend, and will degenerate on the default flashinfer NVFP4 path — do not drop `--moe-backend marlin`.

If you serve any ModelOpt-packed NVFP4 MoE checkpoint on vLLM — ours, LibertAI's, anyone's — run the probe. If it comes back dirty, the patch is 20 lines and the mechanism, the proof, and the upstream trail are all in the repo.

---

Weights: [huggingface.co/murai-labs/GLM-5.3-Flash-NVFP4-Murai](https://huggingface.co/murai-labs/GLM-5.3-Flash-NVFP4-Murai)
Patch, probe, launcher, recipe and upstream writeups: [github.com/Murai-Labs/GLM-5.3-Flash-NVFP4-Murai](https://github.com/Murai-Labs/GLM-5.3-Flash-NVFP4-Murai)
Upstream report: [vllm-project/vllm#54150#issuecomment-5503200584](https://github.com/vllm-project/vllm/issues/54150#issuecomment-5503200584)