Cartesia, the voice AI company, says it has built a language model that no longer needs a tokenizer, and that it beats a conventional design at every size the company tested. The claim comes from a two-part research post the company published on 28 October 2026, announced on X by chief scientist Albert Gu, who also teaches at Carnegie Mellon.
First, the term. A language model does not read letters. Before it sees your text, a separate program called a tokenizer cuts the text into pieces from a fixed menu, a bit like snapping a sentence into Lego bricks of set shapes. Llama 3, Meta’s open model, uses a menu of 128,000 pieces, and each one covers about 4.5 bytes of text on average. Humans designed the rules that build that menu, and the model is stuck with them.
Cartesia calls the tokenizer one of the last hand-built parts of the pipeline. Its alternative is the hierarchical network, or H-Net, introduced in 2025. It takes raw bytes and learns where to draw the boundaries itself, inside the model, as part of training. Gu framed it as a bet on the “bitter lesson”: that simple, general building blocks learning from raw data tend to win.
The first test, according to Cartesia Research, was a controlled swap. The team built a standard tokenized Transformer at several sizes, then replaced only the tokenizer with a one-stage H-Net matched on parameters, computing budget and compression. The H-Net reached lower error at every size. On Cartesia’s validation data it was 1.28 times more efficient with computing power. On two standard evaluation suites the gain was about 2x, and on code it ranged from 1.49x for Markdown to 2.40x for Go. On ten programming languages absent from training, including Prolog, Lean and Fortran, the median gain was 3.10x, peaking at 5.37x on Prolog.
The second test used a larger two-stage model with 3.67 billion parameters, trained on the open recipe behind Hugging Face’s SmolLM3-3B. Cartesia says it matched the closing benchmark average posted by SmolLM3 after roughly 1 trillion training tokens, against roughly 11 trillion for SmolLM3. The company reads that as more than a tenfold saving in compute and data, while admitting that matching data and compute fairly between byte-level and tokenized models is an open research problem.
The behavior is the interesting part. The model groups common multi-word phrases into single units and handles Hindi far more compactly than standard tokenizers do. It also adapts mid-document: an invented word gets split into small pieces on first appearance, then folded into fewer chunks as it keeps recurring. When reading code, it picks up fresh variable names as single units mid-stream. A fixed tokenizer cannot do any of this, because its menu is frozen before training starts.
Cartesia says the idea travels beyond text. Trained on nothing but bacterial DNA, an H-Net was 6x to 7x more efficient with compute than a Transformer, and the chunks it drew matched codons, genes and repeated stretches in human DNA it had never encountered. Early image work showed it grouping regions of a picture without labels. Cartesia, best known for voice, says raw audio was the original motivation.
Every figure here is Cartesia’s own, and none has been independently reproduced. The company’s earlier 2025 H-Net paper drew at least one outside replication that fell short of a Llama 2 baseline on efficiency, so the gap between a company blog and a replicated result is real. Cartesia also says it plans to release the improved internal version, which would let others check the numbers.
For anyone training or fine-tuning models in non-English languages, code or biological sequences, the release is worth queuing up as a benchmark candidate: those are the areas where the claimed gains are largest and where a fixed tokenizer hurts most.
Reported by Cartesia Research in its blog post “From Tokens to Bytes I: Scaling Hierarchical Networks” on 28 October 2026.
