Lab Notes  /  Flash Teardown  /  Creative eval report

Lab Note · 2026-08-28 · Ramchand Kumaresan (Murai Labs)

Two Frontier MoEs on Two DGX Sparks: GLM-5.3-Flash vs Qwen3.8-Flash-Next Build the Shore Temple

Status: complete through blind scoring. All phases done: deployment, single-shot, repair, blind scoring (6/6 artifacts scored). Last updated: 2026-08-27 (dates in this doc are the days the work happened, 2026-08-26/27).

Authors/operators: Ramchand Kumaresan (Murai Labs), with Kimi Code CLI as the executing agent. Task corpus context: the eval prompt draws on Thiruvaasal, a research system for probabilistic reconstruction of the lost temples of Mahabalipuram.

1. What we set out to do

Two open-weight models launched within 48 hours of each other:

• GLM-5.3-Flash (Z.ai) — 320B total / 18B active MoE, hybrid sparse+linear attention, natively multimodal
• Qwen3.8-Flash-Next (Alibaba) — 180B total (125B LM, 6B active + 51B n-gram embedding + 4B MTP), hybrid Gated DeltaNet + Qwen Sparse Attention, multimodal

Both are too new to have community bake-offs. We wanted a work-quality comparison, not a benchmark-table comparison: give both models the same substantial creative-engineering task and compare what they actually produce, how they think, and what they cost in tokens and time.

The task: a single self-contained HTML file rendering an interactive 3D voxel-art scene of the Shore Temple at Mahabalipuram (8th-century Pallava, granite, three shrines, Nandi-topped enclosure wall, facing the Bay of Bengal at sunrise) — with orbit controls, a click-triggered temple-bell/petal-burst interaction, animated waves, and thousands of particles. Reference point: the voxel-art scene genre exemplified by wesche.com's Qwen pagoda scene (itself model-generated; we deliberately chose a different theme to avoid familiarity effects).

The twist that makes it more than a vibe test: both models received a curated reference evidence pack from the Thiruvaasal research repo — traced vimana/sikhara profile curves, parametric dimensions, a measured granite color palette (#786654 family), and two CC-licensed photos of the actual temple (both models are natively multimodal). So the eval also measures how well a model uses supplied domain evidence.

2. Hardware and deployment

2.1 Cluster

Two NVIDIA DGX Spark (GB10 Grace-Blackwell, sm_121) nodes, 128 GB unified LPDDR5X each, joined by a 200G ConnectX-7 QSFP DAC (measured 109 Gb/s RoCE at eval time; ~13.6 GB/s NCCL all-reduce). CUDA 13.0, driver 580.173.02, Ubuntu 24.04 (DGX OS), Docker 29 + NVIDIA Container Toolkit.

2.2 Checkpoint selection

Neither model's BF16 weights fit (GLM: 599 GiB; Qwen: 360 GB), and official FP8 (GLM ~306 GiB) also exceeds the 242 GB usable pair. We surveyed every published quant on Hugging Face on launch week:

Attribution caveat that applies to everything below: we measured these checkpoints, not the abstract models. The GLM quant card carries no downstream coding evals; quant-vs-model attribution is unresolved. See §7.

2.3 Getting GLM-5.3-Flash to serve: six walls, one recipe

The vendor-day-0 vLLM image (vllm/vllm-openai:glm53-flash-arm64-cu130) failed on GB10 in six distinct ways before serving:

1. Follower node must run --headless (rayless mp multi-node) — else AssertionError: collective_rpc should not be called on follower node
2. In-container hostname resolves to 127.0.0.1 → Gloo full-mesh failure; fix: VLLM_HOST_IP per node
3. NCCL "unhandled system error" on IB without --cap-add=IPC_LOCK
4. RoCE IBV_WC_RETRY_EXC_ERR mid-init (later solved properly by the recipe's NCCL env: NCCL_IB_ADDR_RANGE, NCCL_IB_ROCE_VERSION_NUM=2, memlock=-1, full /dev/infiniband passthrough)
5. Free memory (110.26 GiB) below the image's 0.92 default GMU target → --gpu-memory-utilization 0.88
6. The hard one: GLM-5.3's sparse MLA is NoPE (qk_rope_head_dim=0), and the only stock sm_12x sparse backend requires the packed fp8_ds_mla layout that hardcodes DeepSeek's pe_dim=64 → assert death in KV warmup. No config flag fixes a missing kernel.

At that point we found tonyd2wild's day-zero deployment repo — the same checkpoint on the same 2×GB10 topology, with seven day-zero bugs patched in a Docker image chain (NoPE backend extension to SM121, FlashInfer 0.6.18 nightly for an FA2 NaN, NCCL/CUTLASS re-pins, PDL gate, indexer top-k init fix, fp8-KV smem tile cap). We built their v1→v8 patch chain on node 1, shipped the image to node 2 over the fabric, and launched with their launcher adapted to our IPs and HF-cache layout.

Final serving config (both ranks): TP=2 rayless mp over the 200G fabric, --moe-backend marlin --enforce-eager (sm_121 FP4-MoE kernel gap), MTP speculative decoding (n=4), --kv-cache-dtype fp8_e4m3 with pinned KV budget, --block-size 2304, GMU 0.85, 262,144-token context, thinking left on.

Observed serving performance: ~22–26 tok/s sustained decode (matching the recipe's 21.8 tok/s), MTP mean acceptance length 2.4–3.7 (per-position acceptance up to [0.88, 0.78, 0.58, 0.49] under real load), 18–21 min cold boot, TTFT ~5–20 s on the 7.4k-token eval prompt. Model loaded: 91.2 GiB per node in ~14 min from NVMe.

2.4 Qwen3.8-Flash-Next

Served via SGLang (lmsysorg/sglang:qwen38flashnext, arm64) at TP=2 across both nodes. Three launch attempts failed before serving:

1. --fp4-gemm-backend flashinfer_cutlass (RadixArk card recipe, validated on GB300/B300) → CuTe-DSL MLIRError at warmup on sm_121
2. marlin FP4 fallback → the failure moved but persisted: the QSA sparse-decode resolver gates the FlashInfer TRT-LLM kernel behind is_sm100_supported(), and on SM121 it falls back to an FA4 CuTe path that dies in warmup — the true root cause of both failures
3. --attention-backend triton → unchanged, because the failing kernel is on the QSA path, not the main attention backend

The fix came from tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark: a one-line guard extension (is_sm120_supported() accepted after a head-shape probe), plus their second patch bounding the fused mrope Triton kernel's temporal mask (qwen4_exp is the first M-RoPE model with partial rotary; unbounded t_mask read OOB on every image call). We rebuilt both as guarded apply-or-refuse string patches (radixark/sglang-qwen38flashnext:sm121-qsa-mrope1) and adopted their verified flag set: MTP (NEXTN, 3 steps / 4 draft tokens), --ple-offload-embedding (51B n-gram table to pinned host RAM), mamba state pool pinned at 97, cutlass FP4 GEMM restored, CUDA graphs on with prefill graphs off, KV pool pinned at 600K tokens.

Observed serving performance: ~8-min boot (vs 18–21 min for GLM/vLLM), TTFT ~1–2 s, ~30–40 tok/s sustained decode on the eval workload, MTP accept length ~1.77–2.0 on our prompts (below the recipe's 3.3 peak — our thinking-heavy long-form generations accept less than code). SGLang reports a proper reasoning_tokens split in usage, which vLLM did not.

Known stack risk disclosed for this eval: this day-0 SGLang stack has a documented token-0 (!!!!) degeneration loop at temp 1.0 with thinking on in tool-carrying sessions (sglang#36537 family). Our eval runs thinking-on at temp 1.0 (vendor-recommended) with no tools; outputs were checked for the loop signature (!!!! runs) — none occurred in any Qwen run.

3. Eval design

3.1 Protocol

• Identical prompt (verbatim) for both models: task spec + reference pack + 2 reference photos, single user message (prompt.json, 7,369 tokens)
• N=3 generations per model, thinking mode on, vendor-recommended sampling (GLM: temp 1.0 / top_p 0.95 / reasoning_effort=max; Qwen: + top_k 20, reasoning_effort=xhigh), max_tokens=65536
• Phase 1: single-shot. What lands is what gets scored.
• Phase 2: repair loop. Each failed artifact goes back to the model with the exact headless-QA error and symptom ("page shows the plaque, no scene renders"), up to 3 iterations. Scored as iterations-to-working; the phase-1 record stays untouched.
• Blind scoring: artifacts labeled A/B, scored from screenshots + interaction video; traces scored separately (planning, evidence use, engineering judgment, self-correction)

3.2 Objective gates (automated, headless Chromium via Playwright)

1. Loads with zero uncaught exceptions (pageerror) and zero console errors
2. A <canvas> exists and renders (screenshot is not plaque-only)
3. Click interaction fires without errors
4. No external requests beyond the Three.js CDN

3.3 Instrumentation

harness.py streams every run and captures, per run: full reasoning trace (thinking.txt), raw answer (answer_raw.txt), extracted artifact (output.html), fence-repair flag, TTFT / reasoning-phase / answer-phase timings, token accounting, and per-phase throughput. qa_artifact.py runs the headless gates and captures screenshots. repair.py drives phase 2.

Harness lessons baked in after real failures: uncaught JS exceptions surface via pageerror, not console (our first QA pass missed a dead page because of this); the glm45 reasoning parser buffers the entire thinking block until </think> before streaming, so a silent client ≠ a dead server; and raw artifacts must be written before any parsing — an extraction crash cost us one run's trace before that fix (the run was repeated).

3.4 The reference pack

reference_pack/reference_pack.md (~2,000 words), condensed from Thiruvaasal's evidence-tiered sources: shrine names and layout, vimana tier taper ratios (0.804/0.767/0.710/0.633 across 5 tiers), traced tower/eave/dome profiles as control points, Mayamata/Manasara proportion priors, and the image-derived granite palette. Every value carries its source confidence label (source_backed / analog_hypothesis / schematic_context) — the repo's core rule is that inferred values are never presented as measured fact, and the pack preserves that.

4. Phase 1 results: GLM-5.3-Flash

Render gate: 0/3. All three runs produced substantial, well-organized single-file code (~28–32 KB) with correct domain content — and all three died on a different JavaScript runtime/parse bug:

Prompt tokens: 7,369 (all runs). Fence-repair needed on all three (models wrapped the file in a ```html block despite instructions — cosmetic; the harness strips it).

Qualitative read of the artifacts: the code is competent in structure — deterministic value-noise terrain, a single InstancedMesh bake for all static voxels, separate instanced systems for water/foam/petals/birds, shader sky dome with sun glow, custom orbit controls, Web Audio bell with a try/catch. The domain material was genuinely used (correct shrine names Kshatriyasimha Pallavesvara / Jalashayana / Rajasimha Pallavesvara, granite palette from the pack, east-facing sunrise). The failures are one-line-class bugs in 30k-token outputs — a long-tail reliability problem, not an understanding problem. Whether the model can find its own bugs is what phase 2 measures.

4.1 Phase 2: repair loop

Same model, same server, feedback = exact QA error + symptom. Cap 3 iterations.

A QA-gate bug of ours was caught here and corrected: headless Chromium's software-GL driver emits GPU stall due to ReadPixels performance warnings during screenshotting, and the gate was counting console warnings as failures. Verdicts were recomputed under the corrected gate (errors + pageerror only); the raw artifacts were never modified. Marker files carry a correction note.

Phase 2 score: 3/3, all in exactly one feedback round. GLM-5.3-Flash localized and fixed each of its own distinct bugs from a one-line error description plus the symptom, returning complete corrected files. The contrast with phase 1 (0/3) is the sharpest finding of the eval so far: the model's failure mode is not inability to write or debug the code — it is the absence of self-verification in single-shot generation. Given a signal, it repairs perfectly; left alone, it ships a parse error.

Repair traces (each iteration's full reasoning) are under runs/glm-5.3-flash/run_<N>/repair_<M>/ (raw run logs, 22 MB — not hosted; available on request). Working artifacts for publication: run_<N>/repair_1/output.html — rendered in the gallery.

5. Qwen3.8-Flash-Next results

Phase 1 render gate: 0/3 — the same single-shot long-file failure class as GLM, three distinct JS errors:

Notable contrast vs GLM before any quality judgment: Qwen used less than half the tokens (26–36k vs 56–63k) and less than a third of the wall clock per attempt — while hitting the same failure class. Speed and economy favor Qwen decisively; the gate says both models equally fail to self-verify single-shot.

5.1 Phase 2: repair loop (in progress)

Same protocol as GLM: exact error + symptom fed back, cap 3 iterations, corrected QA gate.

6. Head-to-head

Single-shot (phase 1): GLM 0/3, Qwen 0/3. Both models fail to produce a parseable 30k-token single-file Three.js scene on the first attempt, every time, each with a different one-line-class bug. The strongest shared finding: neither model self-verifies. Both are reasoning models that spent 26k–63k tokens thinking about the scene and none simulated "will this parse?"

Repair (phase 2): GLM 3/3 in 1 iteration each; Qwen 3/3 in 3, 2, and 1 iterations. Both models are fully capable self-debuggers when handed the exact error. GLM was uniformly one-and-done; Qwen needed follow-ups on two of three (its iteration-1 repairs introduced or retained secondary errors).

Visual quality of the repaired artifacts (see gallery/): GLM's three scenes are all coherent, distinct, and evidence-faithful (twin vimanas, Nandi wall, granite palette, shore orientation). Qwen produced one strong scene (run 1, the best single composition of the six — blind 9), one adequate one (run 2 — generic stepped towers, less Shore-Temple-specific; blind 5), and one that reached the gate with a broken composition (run 3 — camera opened inside a wall; after one extra visual-symptom repair it renders correctly but scored a blind 6, still below every GLM scene). Qwen's scenes carry the reference pack's provenance disclaimer on-page, unprompted — a notable evidence-discipline signal.

Economy (per attempt, phase 1):

Qwen is roughly 2× more token-economical and ~1.7× faster end-to-end on this task, on this hardware. GLM thinks longer and its scenes show it.

Deployment effort (day-0, 2×GB10): comparable pain. GLM/vLLM needed a 7-patch image chain (community recipe); Qwen/SGLang needed a 2-patch image (QSA guard + mrope mask). Both needed the sm_121 kernel-gap dance; both ended on NCCL-over-RoCE across the 200G fabric. Neither vendor's stock path worked out of the box on this hardware.

Blind scoring (Ramchand, 2026-08-27, /10, labels unblinded after scoring):

Blind means: GLM 7.7 (8, 7, 8 — consistent), Qwen 6.7 (9, 5, 6 — high variance: the single best scene of the six and the two weakest). The blind scores track the qualitative read above almost exactly: B1 was flagged as arguably the best composition, B2 as generic, B3 as broken. B3's 6 came after an extra visual-symptom repair iteration the other five artifacts did not get (see §5.1), and it is still Qwen's second-weakest scene — the extra help did not close the gap.

7. Caveats and threats to validity

• Checkpoint ≠ model. Both measurements are of third-party NVFP4 quants on patched day-0 serving stacks. The GLM quant has no published coding evals; a quant-related regression cannot be ruled out and is a plausible alternative explanation for the 0/3.
• Day-0 serving stacks. Patched vLLM (GLM) and a per-model SGLang image (Qwen) are not release-quality; engine bugs can masquerade as model bugs. We mitigated by capturing exact errors and keeping engine versions recorded per run.
• N=3. Single-shot variance is high for 30k-token generations; pass rates here are indicative, not statistically tight.
• Token cap. max_tokens=65536 truncates some thinking-max runs (GLM run 3's first attempt died mid-HTML at the cap). Both models get the same cap.
• The reference pagoda scene that inspired this eval is itself model-generated (per its URL path), i.e., the genre target is a best-of, not a median.

8. Artifact index

shore-temple-eval/
  harness.py            # instrumented streaming runner
  repair.py             # phase-2 repair loop
  qa_artifact.py        # headless gates + screenshots
  build_prompt.py       # composes prompt.json (task + pack + photos)
  prompt.json           # the exact prompt both models got (7,369 tokens)
  params/               # per-model vendor-recommended sampling params
  reference_pack/       # condensed Thiruvaasal evidence brief
  RUBRIC.md             # gates + blind scoring rubric
  runs/<model>/run_<N>/ # thinking.txt, answer_raw.txt, output.html, metrics.json, QA screenshots

Raw markdown (REPORT.md) · Teardown · Creative eval report · Blind-scored gallery · Battery transcripts