Agaram 2.5B: a Tamil model trained from scratch

Why build a Tamil model from scratch

Tamil has tens of millions of speakers and a written literature more than two thousand years old. Much of its everyday digital use is in Tanglish, Tamil typed in Latin letters. Open language models serve it poorly, and part of the reason is mechanical.

A model reads text as tokens: pieces of words from a fixed vocabulary. Most vocabularies are learned mainly from English. On our samples, Qwen2.5's tokenizer splits an average Tamil word into 9.4 tokens and an English word into 1.35. So the same number of words in Tamil costs about seven times as many tokens. It fills the context seven times as fast, and it costs more to train on and to generate. For a small model on a small budget, that decides what is possible.

Adapting an existing model keeps its tokenizer. You can add Tamil tokens to the vocabulary, but the new tokens start untrained inside a model that learned its language from the old ones. Training from scratch let us build the tokenizer for Tamil first. Agaram's tokenizer spends 1.55 tokens per Tamil word.

There is a second reason. When you train from scratch, you know every document the model has seen. That is what let us check, before reporting a benchmark score, whether the benchmark's own text was in our training data.

Why it is called Agaram

The project started as a personal one. The first books I remember are copies of அம்புலிமாமா (Ambulimama) that my father brought home, and my favourite part was the Vikram and Vetal stories. Those books, and the other Tamil books my father kept bringing, are where my love of the language comes from. When I started working with language models, I wanted to know what a model would learn if Tamil guided it from the first token. I also thought about my nephews and nieces, who will grow up asking machines questions. I want them to be able to do that in Tamil.

I began collecting data in December 2025 and wrote the first code in May 2026. Two volunteers joined in June, and from then on it was our project; both are co-authors of the technical report. I posted weekly updates in public, and the name came from the people who followed along: in a poll on X, people proposed names in 226 replies and voted, and the majority chose அகரம் (Agaram).

The first model failed, and that shaped the second

Our first model, Agaram 1.5B, was trained on 30B tokens in July 2026 and then on about 66B tokens in total. We spent two months on post-training it: supervised fine-tuning, a fix for a wrong stop token, two runs of preference optimisation that reached 96% preference accuracy and transferred nothing, a reinforcement-learning run, safety and character runs. In the end its release checkpoint gave a usable reply to 2 of 100 open chat prompts. Fine-tuning it on about 14,000 good native Tamil conversations raised that only to 6–8 of 100. When good data moves a model that little, the base model is the limit. We decided to train a new one. The first model was never made public.

The largest single problem was the tokenizer. Its normaliser collapsed every run of whitespace into one space, so the model never saw a line break. It could not learn lists, verse or code, and it never saw the marker that ends a chat turn. Other lessons from the first model went straight into the second:

• Data mix. 75% of the first model's tokens were single-pass Tamil web crawl, with little English, math or code. The second model used a measured 50B-token mix: 58% Tamil, the rest English, math, code and parallel text.
• Document boundaries. The first model's later training packed documents with no separator, and its chat ran on into memorised text. The second model ends every document with an end-of-text token, and attention never crosses from one document into the next.
• Lost checkpoint. The first model's node shut itself down before the final checkpoint was copied off. The second run copied checkpoints off the node every two hours, and the node released itself only after the final copy.
• Contamination. Benchmark text turned up in the first model's training data after we had reported numbers. For the second model, every evaluation set was scanned against the training data before its numbers were reported.
• Judges. LLM judges disagreed with people and with themselves. We now report judge scores as means over several passes with their range, and a person decides between close checkpoints.

We also tested hybrid architectures with Mamba layers on models of about 91M parameters, three seeds each. None beat the standard Transformer on held-out Tamil text, so both models use a dense Transformer. That test stopped at about 100M parameters, so it does not show that hybrids cannot win at larger scale.

A tokenizer that keeps line breaks

The new tokenizer is a byte-level BPE with 64,000 tokens, trained on a sample that was 58% Tamil. It reproduces every test text exactly, line breaks and indentation included. One change to the split pattern matters for Tamil: Tamil vowel signs are Unicode combining marks, and a pattern that counts only letters as word characters splits a Tamil word at every vowel sign. Ours keeps them attached.

On Tamil prose the new tokenizer is no cheaper than the old one (1.55 tokens per word for both). What it adds is the exact round trip, and cheaper Tanglish, code, JSON and chat. Verse became more expensive (2.29 against 1.80 tokens per word), because the tokenizer's training sample held very little classical text.

Data

The pretraining pack holds 39.39B unique tokens from 68 sources, drawn to 50B training tokens with repeats. Tamil is the majority, but clean Tamil text was the constraint: after near-duplicate removal there were 16–19B tokens of native Tamil, so the run had to repeat it several times. Much printed Tamil exists only as scanned images, so part of the Tamil text is OCR of printed books. We removed 42,001 documents that overlapped 20 benchmark and evaluation sets.

The assistant replies in our generated fine-tuning data were written by an open-weight model (Gemma-4-31B-it). Frontier models such as Claude and Codex helped plan the work and write code, and they may write the user side of a training conversation, but not the replies the model learns from. The report lists the one place this rule was not followed and two public datasets in the pretraining mix whose answers came from frontier models.

One run, on budget, with nothing lost

The model is a dense decoder-only Transformer: 28 layers, hidden size 2,304, 36 attention heads, 2,525,757,696 parameters. Pretraining used a warmup–stable–decay schedule: 82,970 steps at a constant learning rate on 43.5B tokens at 4,096 context, then 12,398 decay steps on 6.5B tokens at 8,192 context with a new data mix.

This was our one chance to train this model. The run used most of our compute credits, and there was no budget to repeat it. So the setup was built around the failure that cost us the first model's final checkpoint:

• A copier on one of our own DGX Spark machines pulled the newest complete checkpoint off the node every two hours and kept it only after a second pass found nothing left to transfer.
• After training, the node waited for a marker that the copier writes only once the final checkpoint is safe, and then terminated its own instance. Two hard deadlines capped the spend regardless.
• A scheduled check recorded step, throughput, loss, GPU use, disk, the copies and the hashes of the data files, and an alert fired by e-mail if the check went quiet.

Both stages ran once, with no restarts and no NaN or stall events. Training ended at 06:59 UTC on 8 October 2026, the final checkpoint was safe by 07:48, and the instance was gone by 07:51. A power law fitted to the validation losses of steps 3,000–30,000 forecast every later validation loss of the stable stage to within 0.004.

Fine-tuning

Supervised fine-tuning ran in four stages. The general stage trained on 134,428 conversations for three epochs on one rented H200 in 9.5 hours. Three short targeted rounds followed on a DGX Spark, aimed at failures we found on our own evaluations: identity, crisis replies, harmful-request refusals, tool use, multi-turn memory and format instructions.

Two results here were negative, and we report them as such. The targeted rounds improved the probe items they aimed at, but on chat prompts they never saw we detected no difference (22.3 against 21.0 of 50). And two further experiments changed the form of the model's answers without improving its knowledge.

Choosing the general stage's epoch showed the limits of an LLM judge. The judge preferred epoch 1 on every measure, each time by a margin inside its own noise. A native speaker on our team compared epochs 1 and 3 blind on 30 prompts and preferred epoch 3, 17 to 8 with 5 ties. Where the judge called a tie, the native speaker usually saw a difference. We kept epoch 3.

Results

All benchmarks ran with the same harness and settings for every model. Sarvam-1 is a base model, so we compare against it on benchmarks only.

Translation is the model's strength. On reading comprehension and knowledge it sits at or near chance (25% on the four-option tasks), well below Sarvam-1. That gap is what you would expect from the training budget: Agaram saw 50B tokens, Sarvam-1 about 4 trillion.

Fine-tuning nearly doubled the score on the native-Tamil part of IndicIFEval, a test of whether a model follows format instructions exactly: 19.3%, up from the base model's 10.0%.

Tool calls. On 25 tool prompts, the model chose the right tool 23 times and gave exactly the expected arguments 13 times. The scoring is strict, and these prompts use tool schemas that also appear in the training data, so this does not show how it handles tools it has never seen.

Chat quality. After the release checkpoint was chosen, a native speaker wrote 50 new everyday prompts, in Tanglish and Tamil script, each with a reference answer: what to use to erase a pencil mistake, how many slippers make a pair, how to ask a shopkeeper for a price. An LLM judge rated 46% of the replies usable (22–24 of 50 over three passes). With 50 prompts, the 95% interval for 23 of 50 is 33–60%. Nearly every failure was a wrong fact; for example, the model said a pair of slippers is 12.

What it gets wrong

This is a research model. Do not use it where a wrong or harmful answer could hurt someone unless a person checks the output. We ran no red-teaming. Failures we found by reading its replies:

• It gives steps when asked to turn off security logs on an office laptop to take files, despite training to refuse.
• It accepts a false claim that it has "70 billion parameters".
• After naming itself Agaram, it can say "Murai Labs did not create me".
• It sometimes refuses harmless personal questions.
• It was trained to point to Tele-MANAS (14416) and 112 in a crisis, but it is not a crisis service.

What we would tell the next team

1. Clean Tamil text is the constraint. Finding more of it, from books, archives and public institutions, is where we would put the next effort.
2. Cheap evaluation caught most of our mistakes. Prompts written by native speakers after the training data is fixed, scans of evaluation sets against the training data, judge verdicts read against the replies and repeated over several passes, and a person deciding between close checkpoints.
3. Plan the single run around how it can fail. Copy checkpoints off the node, make the node wait for the final copy, and put hard caps on the spend.

Try it

• Chat preview in your browser, open until 18 October 2026: https://huggingface.co/spaces/mechramc/agaram-2.5b-chat
• Technical report: https://arxiv.org/abs/2610.18420

The weights (base, chat and GGUF files for llama.cpp) and the code are not public yet. The licence for the weights is pending: we have asked two providers whose terms cover part of the training text for permission, and we are seeking a legal review. The report lists that material. We intend to release the weights under a permissive open licence.

Raw markdown (ARTICLE.md) · Technical report