# Two Frontier MoEs on Two DGX Sparks: GLM-5.3-Flash vs Qwen3.8-Flash-Next Build the Shore Temple

**Status: complete through blind scoring.** All phases done: deployment, single-shot, repair, blind scoring (6/6 artifacts scored).
Last updated: 2026-08-27 (dates in this doc are the days the work happened, 2026-08-26/27).

**Authors/operators:** Ramchand Kumaresan (Murai Labs), with Kimi Code CLI as the executing agent.
**Task corpus context:** the eval prompt draws on [Thiruvaasal](https://github.com/mechramc/Thiruvaasal), a research system for probabilistic reconstruction of the lost temples of Mahabalipuram.

---

## 1. What we set out to do

Two open-weight models launched within 48 hours of each other:

- **GLM-5.3-Flash** (Z.ai) — 320B total / 18B active MoE, hybrid sparse+linear attention, natively multimodal
- **Qwen3.8-Flash-Next** (Alibaba) — 180B total (125B LM, 6B active + 51B n-gram embedding + 4B MTP), hybrid Gated DeltaNet + Qwen Sparse Attention, multimodal

Both are too new to have community bake-offs. We wanted a **work-quality** comparison, not a benchmark-table comparison: give both models the same substantial creative-engineering task and compare what they actually produce, how they think, and what they cost in tokens and time.

**The task:** a single self-contained HTML file rendering an interactive 3D voxel-art scene of the **Shore Temple at Mahabalipuram** (8th-century Pallava, granite, three shrines, Nandi-topped enclosure wall, facing the Bay of Bengal at sunrise) — with orbit controls, a click-triggered temple-bell/petal-burst interaction, animated waves, and thousands of particles. Reference point: the voxel-art scene genre exemplified by [wesche.com's Qwen pagoda scene](https://wesche.com/dgx/qwen/pagodas/scenes/flashnext/) (itself model-generated; we deliberately chose a different theme to avoid familiarity effects).

**The twist that makes it more than a vibe test:** both models received a curated **reference evidence pack** from the Thiruvaasal research repo — traced vimana/sikhara profile curves, parametric dimensions, a measured granite color palette (#786654 family), and two CC-licensed photos of the actual temple (both models are natively multimodal). So the eval also measures *how well a model uses supplied domain evidence*.

## 2. Hardware and deployment

### 2.1 Cluster

Two NVIDIA DGX Spark (GB10 Grace-Blackwell, sm_121) nodes, 128 GB unified LPDDR5X each, joined by a 200G ConnectX-7 QSFP DAC (measured 109 Gb/s RoCE at eval time; ~13.6 GB/s NCCL all-reduce). CUDA 13.0, driver 580.173.02, Ubuntu 24.04 (DGX OS), Docker 29 + NVIDIA Container Toolkit.

### 2.2 Checkpoint selection

Neither model's BF16 weights fit (GLM: 599 GiB; Qwen: 360 GB), and official FP8 (GLM ~306 GiB) also exceeds the 242 GB usable pair. We surveyed every published quant on Hugging Face on launch week:

| Checkpoint | Size | Verdict |
|---|---|---|
| `zai-org/GLM-5.3-Flash` (FP8, official) | ~306 GiB | doesn't fit |
| **`LibertAIDAI/GLM-5.3-Flash-NVFP4`** | **181 GiB** | **chosen** — weight-only NVFP4 (experts only, rest BF16), documented method, GB10 notes included |
| `axiomofmind/…-W4A16-NVFP4` | 191 GiB | undocumented, no evals |
| `dealignai/…-UNCENSORED-NVFP4` | 181 GiB | abliterated — wrong for a quality eval |
| GGUFs (unsloth et al.) | various | llama.cpp had no `glm5_next` support at eval time |
| `Qwen/Qwen3.8-Flash-Next-FP8` (official) | ~180 GB | needs TP=2, heavier than needed |
| **`RadixArk/Qwen3.8-Flash-Next-NVFP4`** | **135 GB** | **chosen** — W4A4 experts-only NVFP4, calibrated, with GSM8K/AIME evals on the exact checkpoint |
| `unsloth/Qwen3.8-Flash-Next-GGUF` | 67–104 GiB | requires unmerged llama.cpp PR #27742 — held as fallback |

Attribution caveat that applies to everything below: **we measured these checkpoints, not the abstract models.** The GLM quant card carries no downstream coding evals; quant-vs-model attribution is unresolved. See §7.

### 2.3 Getting GLM-5.3-Flash to serve: six walls, one recipe

The vendor-day-0 vLLM image (`vllm/vllm-openai:glm53-flash-arm64-cu130`) failed on GB10 in six distinct ways before serving:

1. Follower node must run `--headless` (rayless `mp` multi-node) — else `AssertionError: collective_rpc should not be called on follower node`
2. In-container hostname resolves to 127.0.0.1 → Gloo full-mesh failure; fix: `VLLM_HOST_IP` per node
3. NCCL "unhandled system error" on IB without `--cap-add=IPC_LOCK`
4. RoCE `IBV_WC_RETRY_EXC_ERR` mid-init (later solved properly by the recipe's NCCL env: `NCCL_IB_ADDR_RANGE`, `NCCL_IB_ROCE_VERSION_NUM=2`, `memlock=-1`, full `/dev/infiniband` passthrough)
5. Free memory (110.26 GiB) below the image's 0.92 default GMU target → `--gpu-memory-utilization 0.88`
6. **The hard one:** GLM-5.3's sparse MLA is NoPE (`qk_rope_head_dim=0`), and the only stock sm_12x sparse backend requires the packed `fp8_ds_mla` layout that hardcodes DeepSeek's `pe_dim=64` → assert death in KV warmup. No config flag fixes a missing kernel.

At that point we found [tonyd2wild's day-zero deployment repo](https://github.com/tonyd2wild/GLM-5.3-Flash-NVFP4-262K-2x-DGX-Spark) — the same checkpoint on the same 2×GB10 topology, with **seven day-zero bugs patched** in a Docker image chain (NoPE backend extension to SM121, FlashInfer 0.6.18 nightly for an FA2 NaN, NCCL/CUTLASS re-pins, PDL gate, indexer top-k init fix, fp8-KV smem tile cap). We built their v1→v8 patch chain on node 1, shipped the image to node 2 over the fabric, and launched with their launcher adapted to our IPs and HF-cache layout.

**Final serving config (both ranks):** TP=2 rayless mp over the 200G fabric, `--moe-backend marlin --enforce-eager` (sm_121 FP4-MoE kernel gap), MTP speculative decoding (n=4), `--kv-cache-dtype fp8_e4m3` with pinned KV budget, `--block-size 2304`, GMU 0.85, 262,144-token context, thinking left on.

**Observed serving performance:** ~22–26 tok/s sustained decode (matching the recipe's 21.8 tok/s), MTP mean acceptance length 2.4–3.7 (per-position acceptance up to [0.88, 0.78, 0.58, 0.49] under real load), 18–21 min cold boot, TTFT ~5–20 s on the 7.4k-token eval prompt. Model loaded: 91.2 GiB per node in ~14 min from NVMe.

### 2.4 Qwen3.8-Flash-Next

Served via SGLang (`lmsysorg/sglang:qwen38flashnext`, arm64) at TP=2 across both nodes. Three launch attempts failed before serving:

1. `--fp4-gemm-backend flashinfer_cutlass` (RadixArk card recipe, validated on GB300/B300) → CuTe-DSL `MLIRError` at warmup on sm_121
2. `marlin` FP4 fallback → the failure moved but persisted: the QSA sparse-decode resolver gates the FlashInfer TRT-LLM kernel behind `is_sm100_supported()`, and on SM121 it falls back to an FA4 CuTe path that dies in warmup — the true root cause of *both* failures
3. `--attention-backend triton` → unchanged, because the failing kernel is on the QSA path, not the main attention backend

The fix came from [tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark](https://github.com/tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark): a one-line guard extension (`is_sm120_supported()` accepted after a head-shape probe), plus their second patch bounding the fused mrope Triton kernel's temporal mask (qwen4_exp is the first M-RoPE model with partial rotary; unbounded `t_mask` read OOB on every image call). We rebuilt both as guarded apply-or-refuse string patches (`radixark/sglang-qwen38flashnext:sm121-qsa-mrope1`) and adopted their verified flag set: MTP (NEXTN, 3 steps / 4 draft tokens), `--ple-offload-embedding` (51B n-gram table to pinned host RAM), mamba state pool pinned at 97, cutlass FP4 GEMM restored, CUDA graphs on with prefill graphs off, KV pool pinned at 600K tokens.

**Observed serving performance:** ~8-min boot (vs 18–21 min for GLM/vLLM), TTFT ~1–2 s, ~30–40 tok/s sustained decode on the eval workload, MTP accept length ~1.77–2.0 on our prompts (below the recipe's 3.3 peak — our thinking-heavy long-form generations accept less than code). SGLang reports a proper `reasoning_tokens` split in usage, which vLLM did not.

**Known stack risk disclosed for this eval:** this day-0 SGLang stack has a documented token-0 (`!!!!`) degeneration loop at temp 1.0 with thinking on in tool-carrying sessions (sglang#36537 family). Our eval runs thinking-on at temp 1.0 (vendor-recommended) with no tools; outputs were checked for the loop signature (`!!!!` runs) — none occurred in any Qwen run.

## 3. Eval design

### 3.1 Protocol

- **Identical prompt** (verbatim) for both models: task spec + reference pack + 2 reference photos, single user message (`prompt.json`, 7,369 tokens)
- **N=3 generations per model**, thinking mode on, vendor-recommended sampling (GLM: temp 1.0 / top_p 0.95 / `reasoning_effort=max`; Qwen: + top_k 20, `reasoning_effort=xhigh`), `max_tokens=65536`
- **Phase 1: single-shot.** What lands is what gets scored.
- **Phase 2: repair loop.** Each failed artifact goes back to the model with the exact headless-QA error and symptom ("page shows the plaque, no scene renders"), up to 3 iterations. Scored as iterations-to-working; the phase-1 record stays untouched.
- **Blind scoring:** artifacts labeled A/B, scored from screenshots + interaction video; traces scored separately (planning, evidence use, engineering judgment, self-correction)

### 3.2 Objective gates (automated, headless Chromium via Playwright)

1. Loads with zero uncaught exceptions (`pageerror`) and zero console errors
2. A `<canvas>` exists and renders (screenshot is not plaque-only)
3. Click interaction fires without errors
4. No external requests beyond the Three.js CDN

### 3.3 Instrumentation

`harness.py` streams every run and captures, per run: full reasoning trace (`thinking.txt`), raw answer (`answer_raw.txt`), extracted artifact (`output.html`), fence-repair flag, TTFT / reasoning-phase / answer-phase timings, token accounting, and per-phase throughput. `qa_artifact.py` runs the headless gates and captures screenshots. `repair.py` drives phase 2.

**Harness lessons baked in after real failures:** uncaught JS exceptions surface via `pageerror`, not `console` (our first QA pass missed a dead page because of this); the glm45 reasoning parser buffers the entire thinking block until `</think>` before streaming, so a silent client ≠ a dead server; and raw artifacts must be written *before* any parsing — an extraction crash cost us one run's trace before that fix (the run was repeated).

### 3.4 The reference pack

`reference_pack/reference_pack.md` (~2,000 words), condensed from Thiruvaasal's evidence-tiered sources: shrine names and layout, vimana tier taper ratios (0.804/0.767/0.710/0.633 across 5 tiers), traced tower/eave/dome profiles as control points, Mayamata/Manasara proportion priors, and the image-derived granite palette. Every value carries its source confidence label (`source_backed` / `analog_hypothesis` / `schematic_context`) — the repo's core rule is that inferred values are never presented as measured fact, and the pack preserves that.

## 4. Phase 1 results: GLM-5.3-Flash

**Render gate: 0/3.** All three runs produced substantial, well-organized single-file code (~28–32 KB) with correct domain content — and all three died on a different JavaScript runtime/parse bug:

| Run | Completion tokens | Wall clock | Overall rate | Reasoning | Failure (headless QA) |
|---|---|---|---|---|---|
| 1 | 62,788 | 44 min | 23.6 tok/s | 151 KB trace | `SyntaxError: Unexpected token '}'` — one extra brace in 707 lines kills the entire script at parse |
| 2 | 63,086 | 46 min | 23.0 tok/s | 161 KB trace | TDZ: `Cannot access 'scene' before initialization` |
| 3 | 55,843 | 41 min | 22.5 tok/s | 141 KB trace | `null.width` dereference — no canvas created |

Prompt tokens: 7,369 (all runs). Fence-repair needed on all three (models wrapped the file in a ```html block despite instructions — cosmetic; the harness strips it).

**Qualitative read of the artifacts:** the code is *competent in structure* — deterministic value-noise terrain, a single `InstancedMesh` bake for all static voxels, separate instanced systems for water/foam/petals/birds, shader sky dome with sun glow, custom orbit controls, Web Audio bell with a try/catch. The domain material was genuinely used (correct shrine names Kshatriyasimha Pallavesvara / Jalashayana / Rajasimha Pallavesvara, granite palette from the pack, east-facing sunrise). The failures are one-line-class bugs in 30k-token outputs — a long-tail reliability problem, not an understanding problem. Whether the model can *find* its own bugs is what phase 2 measures.

### 4.1 Phase 2: repair loop

Same model, same server, feedback = exact QA error + symptom. Cap 3 iterations.

**A QA-gate bug of ours was caught here and corrected:** headless Chromium's
software-GL driver emits `GPU stall due to ReadPixels` *performance warnings*
during screenshotting, and the gate was counting console warnings as failures.
Verdicts were recomputed under the corrected gate (errors + pageerror only);
the raw artifacts were never modified. Marker files carry a correction note.

| Run | Single-shot bug | Repair outcome |
|---|---|---|
| 1 | SyntaxError (stray `}`) | **Repaired in 1 iteration** |
| 2 | TDZ (`scene` before init) | **Repaired in 1 iteration** |
| 3 | `null.width` deref | **Repaired in 1 iteration** |

**Phase 2 score: 3/3, all in exactly one feedback round.** GLM-5.3-Flash
localized and fixed each of its own distinct bugs from a one-line error
description plus the symptom, returning complete corrected files. The contrast
with phase 1 (0/3) is the sharpest finding of the eval so far: the model's
failure mode is not inability to write or debug the code — it is the absence
of self-verification in single-shot generation. Given a signal, it repairs
perfectly; left alone, it ships a parse error.

Repair traces (each iteration's full reasoning) are under
`runs/glm-5.3-flash/run_<N>/repair_<M>/`. Working artifacts for publication:
`run_<N>/repair_1/output.html`.

## 5. Qwen3.8-Flash-Next results

**Phase 1 render gate: 0/3** — the same single-shot long-file failure class as GLM, three distinct JS errors:

| Run | Completion tokens | Wall clock | Overall rate | Reasoning | Failure (headless QA) |
|---|---|---|---|---|---|
| 1 | 35,820 | 15 min | 39.2 tok/s | 66 KB trace | `T is not a constructor` |
| 2 | 29,073 | 12 min | 40.1 tok/s | 49 KB trace | `Missing initializer in const declaration` |
| 3 | 25,664 | 11 min | 40.0 tok/s | 41 KB trace | `Cannot access 't' before initialization` |

Notable contrast vs GLM *before* any quality judgment: Qwen used **less than half the tokens** (26–36k vs 56–63k) and **less than a third of the wall clock** per attempt — while hitting the same failure class. Speed and economy favor Qwen decisively; the gate says both models equally fail to self-verify single-shot.

### 5.1 Phase 2: repair loop *(in progress)*

Same protocol as GLM: exact error + symptom fed back, cap 3 iterations, corrected QA gate.

| Run | Single-shot bug | Repair outcome |
|---|---|---|
| 1 | `T is not a constructor` | **Repaired in 3 iterations.** Final scene: dominant vimana with domed sikhara, Nandi-row enclosure wall, lawn, birds, petal field, sea foam — strong composition. |
| 2 | `Missing initializer in const declaration` | **Repaired in 2 iterations.** Coherent scene: twin-towered compound, enclosure wall, shore foam, petal burst, sunrise palette; also carries the pack's provenance disclaimer on-page. |
| 3 | TDZ (`t` before init) | **Repaired in 1 iteration** at the gates — but the composition was visually broken (camera opens inside a giant wall; canvas rendered solid dark). After blind scoring flagged it "incomplete", it received **one extra visual-symptom repair iteration** (feedback: ranked list of likely causes — camera inside geometry, aimed away, no lighting, not added to scene). The model diagnosed framing/lighting and produced a working scene: twin vimanas, compound wall, trees, shore foam, well-framed initial view. Total for run 3: 1 gate iteration + 1 visual iteration. **Protocol asymmetry disclosed:** this run received a richer failure description than the other five; treat its outcome as less comparable. |

## 6. Head-to-head

**Single-shot (phase 1): GLM 0/3, Qwen 0/3.** Both models fail to produce a
parseable 30k-token single-file Three.js scene on the first attempt, every
time, each with a different one-line-class bug. The strongest shared finding:
*neither model self-verifies.* Both are reasoning models that spent 26k–63k
tokens thinking about the scene and none simulated "will this parse?"

**Repair (phase 2): GLM 3/3 in 1 iteration each; Qwen 3/3 in 3, 2, and 1
iterations.** Both models are fully capable self-debuggers when handed the
exact error. GLM was uniformly one-and-done; Qwen needed follow-ups on two of
three (its iteration-1 repairs introduced or retained secondary errors).

**Visual quality of the repaired artifacts** (see gallery/): GLM's three scenes
are all coherent, distinct, and evidence-faithful (twin vimanas, Nandi wall,
granite palette, shore orientation). Qwen produced one strong scene (run 1,
the best single composition of the six — blind 9), one adequate one (run 2 —
generic stepped towers, less Shore-Temple-specific; blind 5), and one that
reached the gate with a broken composition (run 3 — camera opened inside a
wall; after one extra visual-symptom repair it renders correctly but scored
a blind 6, still below every GLM scene). Qwen's scenes carry the reference
pack's provenance disclaimer on-page, unprompted — a notable
evidence-discipline signal.

**Economy (per attempt, phase 1):**

| Metric | GLM-5.3-Flash | Qwen3.8-Flash-Next |
|---|---|---|
| Completion tokens | 55.8k–63.1k | 25.7k–35.8k |
| Wall clock | 41–46 min | 11–15 min |
| Overall throughput | ~22.5–23.6 tok/s | ~39–40 tok/s |
| Reasoning trace size | 141–161 KB | 41–66 KB |
| Boot time (day-0 stack) | 18–21 min (vLLM) | ~8 min (SGLang) |

Qwen is roughly **2× more token-economical and ~1.7× faster** end-to-end on
this task, on this hardware. GLM thinks longer and its scenes show it.

**Deployment effort (day-0, 2×GB10):** comparable pain. GLM/vLLM needed a
7-patch image chain (community recipe); Qwen/SGLang needed a 2-patch image
(QSA guard + mrope mask). Both needed the sm_121 kernel-gap dance; both ended
on NCCL-over-RoCE across the 200G fabric. Neither vendor's stock path worked
out of the box on this hardware.

**Blind scoring (Ramchand, 2026-08-27, /10, labels unblinded after scoring):**

| Run | Label | Model | Score |
|---|---|---|---|
| 1 | A1 | GLM | 8 |
| 1 | B1 | Qwen | **9** |
| 2 | A2 | GLM | 7 |
| 2 | B2 | Qwen | 5 |
| 3 | A3 | GLM | 8 |
| 3 | B3 | Qwen | incomplete at first scoring → **6** after 1 extra visual-repair iteration |

Blind means: **GLM 7.7** (8, 7, 8 — consistent), **Qwen 6.7** (9, 5, 6 —
high variance: the single best scene of the six *and* the two weakest).
The blind scores track the qualitative read above almost exactly: B1 was
flagged as arguably the best composition, B2 as generic, B3 as broken. B3's
6 came after an extra visual-symptom repair iteration the other five
artifacts did not get (see §5.1), and it is still Qwen's second-weakest
scene — the extra help did not close the gap.

## 7. Caveats and threats to validity

- **Checkpoint ≠ model.** Both measurements are of third-party NVFP4 quants on patched day-0 serving stacks. The GLM quant has no published coding evals; a quant-related regression cannot be ruled out and is a plausible alternative explanation for the 0/3.
- **Day-0 serving stacks.** Patched vLLM (GLM) and a per-model SGLang image (Qwen) are not release-quality; engine bugs can masquerade as model bugs. We mitigated by capturing exact errors and keeping engine versions recorded per run.
- **N=3.** Single-shot variance is high for 30k-token generations; pass rates here are indicative, not statistically tight.
- **Token cap.** `max_tokens=65536` truncates some thinking-max runs (GLM run 3's first attempt died mid-HTML at the cap). Both models get the same cap.
- **The reference pagoda scene** that inspired this eval is itself model-generated (per its URL path), i.e., the genre target is a best-of, not a median.

## 8. Artifact index

```
shore-temple-eval/
  harness.py            # instrumented streaming runner
  repair.py             # phase-2 repair loop
  qa_artifact.py        # headless gates + screenshots
  build_prompt.py       # composes prompt.json (task + pack + photos)
  prompt.json           # the exact prompt both models got (7,369 tokens)
  params/               # per-model vendor-recommended sampling params
  reference_pack/       # condensed Thiruvaasal evidence brief
  RUBRIC.md             # gates + blind scoring rubric
  runs/<model>/run_<N>/ # thinking.txt, answer_raw.txt, output.html, metrics.json, QA screenshots
```