01 Latest · quipu-moe-1B-A149M
A 1B mixture of experts that uses 149M parameters per token.
Pretrained from scratch on one rented RTX 5090. Cost from the spend ledger, including the A/B tests, the chat fine-tune and evaluation.
What changed from 114M
16 layers, each with one shared expert and 64 small routed experts, of which every token uses 4. Routing is balanced without an auxiliary loss (Quantile Balancing). A new 49,152-token tokenizer, trained for this model, keeps code indentation intact and covers ten languages.
Every design switch was A/B tested at 200M tokens before the full run, and the losers are published too. Muon beat AdamW clearly: 3.32 vs 4.06 loss, against seed noise of 0.03.
Data: 60% permissively licensed code, 28% English, 12% Indonesian, Malay, Chinese, Japanese, Korean, Tamil, Hindi and Urdu, each document checked with my own language-ID model. HumanEval and MBPP were filtered out of the training data.
Bits per byte at the end of pretraining
Lower is better, comparable across tokenizers. Held-out text per language, and held-out code.
02 Distillation
Keep only the answers that pass their tests.
The chat model could barely write code: greedy decoding solved 5 of MBPP's 257 problems. So an open-weight teacher, Qwen2.5-Coder-1.5B-Instruct (Apache-2.0), answered 5,000 coding problems, and every answer was run against the problem's unit tests in a sandbox. Only the 3,277 that passed were kept. The chat fine-tune was then repeated with those added.
About 4x better, and still low.
SantaCoder (1.1B) reports around 35% on MBPP and Qwen2.5-Coder-1.5B about 69%, with different settings. Quipu uses 149M parameters per token and saw 4.26B tokens. It is not competing with them yet.
The same idea, at a much bigger scale.
A 7B teacher, Qwen2.5-Coder-7B-Instruct (Apache-2.0), generated 638,342 answers, and every one was run against its unit tests in a sandbox. 498,813 passed and were kept: 78.1%.
Generation ran on a rented RTX 3090, the laptop and Modal H100s; verification ran on Modal CPUs.
The demo runs on a small CPU machine that sleeps when idle: the first visit can take about 15 seconds to wake it.
03 Data v3
Ten billion tokens of pretraining data, built for $2.51.
The next pretraining batch: 10,000,003,101 tokens from 42 open, licence-checked sources, built on Modal CPUs. Benchmark problems (HumanEval and MBPP) were filtered out, as before.
What's in it
Mostly code and educational text, with maths, reference text, nine languages from FineWeb-2, and a small slice of the verified teacher answers from distillation v2.
04 Training code
Ready for more than one GPU.
Everything so far was trained on one GPU. These two changes are for the runs after that.
Multi-GPU data-parallel training.
Verified on real GPUs: the 2-GPU test passed. The same training code runs on one GPU or several.
Expert dispatch rewritten with grouped matmuls: 3.24x faster.
Measured on a B200, in tokens per second.
05 First model · quipu-114m
A 114M-parameter decoder, sized for one laptop.
Every number here was measured on the machine that trained it, an RTX 5060 Laptop GPU with 8 GB.
Architecture
Decoder-only transformer in the modern style: RMSNorm, rotary position embeddings, SwiGLU feed-forward, and grouped-query attention with 12 query heads sharing 4 key/value heads. Input and output embeddings are tied, and the context is 1,024 tokens.
Short context by design. Long-range recall is meant to come from memory rather than a longer attention window.
06 Data · quipu-114m
Three billion tokens: mostly text, one-fifth code.
80% FineWeb-Edu, 20% code from github-code-clean. Only permissive licences are kept: MIT, Apache-2.0, BSD, ISC, CC0 and Unlicense. Minified files, vendored directories and files over 16k tokens are filtered out. That filter alone removed as many tokens as the entire code budget.
HTML is capped at 10% so markup doesn't crowd out the languages the model should actually learn to write.
07 Engineering
The failures that don't announce themselves.
Most of the work in a long unattended run is catching things that fail silently. Each of these was found in review, before the run.
Attention was quietly using a slow kernel.
Fixing the fallback took the projected run for the original budget from ~65 hours to ~39.
About 1 in 130 file renames fails transiently.
That is enough to kill a 50-hour run over a log write. Every atomic write now retries.
A resume could restart from zero, silently.
After days of training, with no error. It now refuses unless nothing could have been saved.
One bad batch could have deleted every clean checkpoint.
Non-finite steps are now skipped, and three in a row stop the run before retention prunes the good ones.
08 Results · quipu-114m
It learned the shape of language in 45 hours.
All 5,722 steps ran unattended over a weekend: no restarts, no skipped steps, no data repeated. Validation loss on held-out FineWeb-Edu text, measured every 100 steps:
Watch it learn
Same prompt, greedy decoding, at each saved milestone.
Fluent, on topic, and often wrong.
The final model says the printing press was invented "in 1848 by the French printer Pierre-Louis Leclerc". That is the honest result for a 114M base model: it has learned the form of knowledge without reliable facts, and it still repeats itself under greedy decoding.
The code loss of 0.97 flatters it.
GPT-2's tokenizer splits indentation into single spaces: 36.5% of the code validation tokens are pure whitespace, against 2.0% for text. They are easy to predict, and greedy code output collapses into runs of spaces. That measurement is why the next model gets a code-aware tokenizer.
09 Next
Where it goes from here.
- donePretraining at 114M3B tokens in 45 hours on one laptop GPU. Weights are public.
- done1B mixture of experts with a code-aware tokenizer4.26B tokens, ten languages and code, $18.75 in total. The whitespace result above is why the tokenizer changed.
- doneChat fine-tune and execution-verified distillationHuman-written conversations, then teacher answers that pass their tests: MBPP 1.9% → 7.4%.
- doneDistillation v2A 7B teacher, 638,342 answers, every one tested; 498,813 verified answers kept (78.1%).
- doneData v3 and multi-GPU training10B tokens from 42 open sources for $2.51; data-parallel training verified on 2 GPUs; MoE dispatch 3.24x faster.
- nextContinued pretraining on data v3Monthly runs on free compute credits, with a decayed checkpoint released every two months.
- nextChat fine-tune on the verified answersAll 498k execution-verified answers from distillation v2.
- laterA ~1B-active modelWhen bigger compute is available.
10 Contact
Get in touch.
Questions about the models or the data, bug reports, research collaboration, or compute for bigger runs.
hello@quipulm.devIssues and pull requests are welcome on GitHub.