20 people, 20 GPUs, one model none could build alone

Four steps. Zero communication during training.

The entire KALAVAI protocol fits in a paragraph. A coordinator distributes a shared base checkpoint. Each contributor fine-tunes their copy on their own domain — independently, asynchronously, on whatever hardware they have. Nobody shares data, gradients, or activations. When everyone is done, they submit their checkpoints. A lightweight router (a single linear layer, trained for 500 steps on mixed data) learns which expert is best for which token. At inference, all specialists run in parallel and the router combines their outputs.

That's it. The mechanism is the protocol, not the infrastructure. Standard PyTorch. Standard HuggingFace. No custom CUDA kernels, no distributed training framework, no LoRA, no adapters.

# Everyone starts from the same model
base = load("pythia-410m", revision="step10000")

# Each person trains on their domain (independently, no communication)
specialist_code    = train(copy(base), code_data,    steps=2000)
specialist_science = train(copy(base), science_data, steps=2000)
specialist_fiction  = train(copy(base), fiction_data, steps=2000)

# A router learns who's good at what (1000 steps, one linear layer)
router = nn.Linear(hidden_size, 3, bias=False)
fused = MoE(specialists=[code, science, fiction], router=router)
train_router(fused, mixed_data, steps=500)

# Result: fused model outperforms every individual specialist
# +7.70% over best specialist (corrected per-domain eval)

← Prev 1 / 15 Next →

Consistent gains at every scale tested.

The fused model beats the best individual specialist at every tested scale. All results use the corrected per-domain equal-weight evaluation protocol — each domain evaluated separately at batch size 4, then averaged. The 6.9B result uses the step-budget sweep (2,000 steps, k=4 frozen layers), which recovers the full improvement seen at smaller scales.

Beats equal-compute monolithic training

The natural objection: just train one model on all the data for the same total compute. We tested this directly. A single model fine-tuned on mixed data for 6,000 steps (equal to 3 specialists × 2,000 steps) achieves +6.7% over base on mixed held-out loss. Under corrected per-domain evaluation, the MoE beats the monolithic by +0.47% on equal-weight aggregate and wins on every domain individually. The mechanism is not primarily about training efficiency — it's about cooperative specialisation. Note: the table below uses the original mixed-batch evaluation protocol (absolute losses are from that schema); the corrected evaluation yields lower absolute improvements but the ranking is unchanged.

1B results: replication holds

The 1B scale replicates the 410M result. Under the corrected per-domain equal-weight evaluation, the improvement is +7.49% over the best specialist (seed 42). The monolithic baseline at 1B also confirms: KALAVAI beats equal-compute monolithic training on per-domain evaluation.

6.9B: step-budget sweep recovers the improvement

Initial 6.9B results with 500 training steps showed reduced gains. The step-budget sweep (B1) confirmed the hypothesis: at 2,000 steps with k=4 frozen layers, the 6.9B improvement is +6.53% ± 0.024% (3 seeds, corrected equal-weight eval). The mechanism holds at 6.9B. Freeze depth is largely insensitive at 6.9B (<0.1pp difference across k=0 to k=4).

Before you train a single specialist, you can predict whether the cooperative is worth it.

The headline result of the paper is not that fusion works — it's that you can predict how well it will work from a single measurement taken before any training. Across all experimental conditions, fusion gain scales linearly with the mean divergence of the specialists from the base model:

Divergence is the mean per-token KL divergence of each specialist from the base model, averaged across all specialists. It is computable in minutes on a small validation set after training — no full evaluation required. If divergence is 15%, expect roughly +10% gain. If divergence is 25% (cross-lingual), expect roughly +18% — and likely more, since high-divergence settings exceed the linear prediction.

The four English-domain conditions cluster within ±2.4pp of the line. The cross-lingual condition exceeds the prediction because the base model is near-random on Yoruba and Welsh — when the base achieves near-random perplexity, specialists correct from a high-loss floor and the router routes with near-certainty. The 20-contributor out-of-sample point (+6.63pp residual) follows the same pattern: its heterogeneous mix of high-divergence language specialists pulls the cooperative above what English-domain regression would predict.

The formula also sets a divergence floor: below approximately 3.3% mean divergence, the predicted gain approaches zero. A cooperative of specialists that are too similar to the base model is unlikely to produce positive returns.

Cross-lingual cooperation: Yoruba perplexity 41.9 → 7.7.

Phase 2 tests the predictive model in the high-divergence regime — settings where mean specialist divergence exceeds 18%, far beyond the English-domain Phase 1 conditions. Three experiments, all returning GO verdicts.

The 20-contributor result is out-of-sample for the regression (the formula was fit on 6 conditions). It lands +6.63pp above the linear prediction — consistent with the cross-lingual pattern: a cooperative with high-divergence language specialists converts more efficiently than the English-domain regression predicts. Medical and chemistry specialists share routing (60/40 split), reflecting genuine semantic overlap rather than routing failure. Two domains (dialogue, instructions) show degradation due to data scarcity (<300 training chunks), not a mechanism failure — the router routes correctly (97–89% to own specialist), but the specialists themselves are undertrained.

When fusion works — and when it doesn't

The paper's core contribution isn't "MoE is good." It's an empirical characterisation of the conditions under which post-hoc fusion of independently trained specialists succeeds or fails. We identify three.

1. Shared initialisation is necessary

Specialists must start from the same checkpoint. The shared starting point preserves representational compatibility — specialists diverge in what they learn, but the geometry of their representations stays aligned enough for a router to combine them. Initialising from different checkpoints breaks this: their representational spaces are no longer aligned, and the router cannot learn coherent dispatch.

2. Frozen layers become necessary beyond ~5,000 steps

At short training horizons (≤2,000 steps), freezing layers is optional — more plastic representations allow specialists to diverge more freely. freeze=0 peaks at 2,000 steps (+8.12%). But beyond approximately 5,000 steps, unfrozen specialists over-specialise and become harder to fuse. Freezing the first K layers provides a structural anchor that preserves routing compatibility. The crossover from "freezing hurts" to "freezing helps" occurs at ~5,000 steps.

3. All specialists must run at inference

This is the paper's most surprising result. A domain classifier with 99.3% accuracy, routing each input to a single specialist, produces −21.1% degradation relative to base. The MoE running all three specialists and combining outputs produces +14.1% improvement. Same routing accuracy — opposite results. The 35 percentage point gap is the difference between a system that works and one that's worse than doing nothing.

The mechanism in one image

The cross-domain evaluation matrix shows why fusion works. Each specialist is best on its own domain (the diagonal) and worst on the others. The MoE router dispatches each token to the right diagonal entry, recovering all specialist gains simultaneously.

What doesn't matter (and what does)

Router architecture doesn't matter

Router ablation numbers use the original mixed-batch evaluation protocol (as noted in the paper). A uniform router (fixed 1/N weights, no training) achieves +6.7%. A trained linear router achieves +14.2%. A 2-layer MLP achieves +14.2%. The relative ordering holds under corrected evaluation: learned routing > uniform, linear ≈ MLP. The minimum bar is learnable suppression, not the function class.

Freeze depth sweep

We swept freeze depth from 0 (no frozen layers) to 12 (half the model). At 2,000 steps, freeze=0 wins. At 5,000+ steps, freeze=4 and freeze=8 both outperform freeze=0. Practical guideline: training under 5,000 steps — skip freezing. Over 5,000 steps — freeze 4–8 layers.

Specialist count scales gracefully

Three, four, and five specialists all achieve approximately +14.1% with near-zero variance. The mechanism doesn't degrade as you add more contributors. Two-specialist is slightly higher because the evaluation problem is narrower.

The mechanism survives base model maturity

Fusion improvement is consistent across Pythia checkpoints from 3.5% to 100% of pre-training at 410M (+7.03% to +8.81%) and 1B (+0.40% to +8.75%). Qwen-1.5B at full training shows a reduced gain of +1.06% ± 0.01% (corrected per-domain evaluation, 3 seeds) — near the formula's predicted floor (~3.3% divergence). Small divergence produces small gain, consistent with the predictive model.

Extra parameters don't explain the gains

Two baselines rule out "more parameters" as the explanation. A wider single model with 3.5× the parameters achieves only +5.9%. A multi-head baseline with identical parameter count to the MoE but hard single-expert routing achieves −21.1%. The gain comes from cooperative specialisation plus joint inference, not raw capacity.

Token-level routing, not document classification

On hybrid-domain prompts, the router switches experts mid-sentence. The prompt "Derive the equation for protein folding using Python pandas" forces a domain switch within a single sentence: science tokens ("derive," "equation," "protein," "folding") should activate the science specialist; code tokens ("Python," "pandas") should activate the code specialist. The router discovers this structure from the training signal alone — no supervision, no domain labels, no explicit boundaries.

The pattern is robust across multiple hybrid prompt types, including narrative/science, technical/narrative, and multi-domain sentences with three or more domain switches:

Router confidence distribution

In practice, the router operates as a near-hard switch: the highest-weight expert receives over 95% of the routing weight in more than 99.7% of tokens. Crucially, hard routing (argmax dispatch) and soft routing (learned softmax weights) produce identical perplexity — confirming that the value is selection and suppression, not the specific weighting scheme.

Downstream benchmarks

Perplexity improvements are clear and consistent. Downstream benchmark accuracy is more modest — less than 1pp on standard tasks at 1B. This is expected at these scales: perplexity and benchmark accuracy don't reliably track below 7B parameters. The KALAVAI paper is explicit about this gap.

How specialists diverge during training

Each specialist rapidly improves on its own domain while degrading on out-of-domain content. This divergence is exactly what makes the MoE valuable — if specialists didn't degrade on other domains, the router would have nothing useful to select between. The code specialist's catastrophic forgetting of science is a feature, not a bug.

What we don't claim

The paper is explicit about five things it does not claim.

Limited architecture generality so far. All primary results use Pythia. Qwen-1.5B shows +1.06% at 3.16% mean divergence — near the formula's predicted floor (~3.1%). Small divergence produces small gain, consistent with the predictive model. The mechanism works; divergence magnitude governs gain magnitude.

No guaranteed downstream gains. Perplexity improvements are clear; benchmark accuracy improvements are modest (<1pp at 1B). Perplexity and downstream accuracy don't reliably track at these scales.

No real cooperative demonstrated. All experiments are simulated cooperatives on single machines. Heterogeneous hardware, asynchronous submission, and contributor reliability are open engineering problems.

No frontier-scale evidence. Results reach 6.9B parameters (+6.53% ±0.024%, 3 seeds, step-budget sweep). Evidence above 7B is absent.

What's next if NeurIPS 2026 accepts.

The paper is complete for submission. These experiments are planned for the camera-ready version to preempt likely reviewer objections and extend the predictive model.

Five problems cooperative training solves

The protocol's zero-communication property — contributors share only a starting checkpoint and a final trained checkpoint, never data — opens applications that are structurally impossible with synchronous training or federated learning.

30 minutes. One GPU. The whole protocol.

git clone https://github.com/mechramc/Kalavai.git
cd Kalavai
pip install transformers datasets torch accelerate
python experiments/kalavai_pythia_experiment.py

Requires any GPU with 24GB+ VRAM (RTX 3090, 4090, 5090, A100, or equivalent). Produces trained specialists, fused MoE, all evaluation numbers, and figures. All results use the corrected per-domain equal-weight evaluation protocol.

Every experiment is a self-contained Python file. No config files, no YAML. Read the script, understand the experiment, run it. 322 automated audit checks verify every result before any paper-ready number is reported.