


11 Oct 2026 · 24m · 14 chapters

The episode explains “bitification,” a research method to retrofit existing subword language models so they operate over raw UTF-8 bytes instead of human-like subword tokens, improving character-level precision and reducing tokenization artifacts.
No guests are named in the transcript; it appears to be a host-led discussion.
Subword tokenization causes failures on literal character tasks (e.g., spelling backwards, counting letters in “strawberry”), introduces spacing-based logic biases (adding a space can map to different tokens), and limits non-Latin scripts due to a fixed token dictionary. Byte-level models previously lagged because of compute constraints, but bitification achieves it with <1% of original training compute by distilling and retrofitting.
“One Byte Future Context” (boundary decisions using a one-byte lookahead via a special boundary symbol); CUTE benchmark for character-level understanding; improved code generation (semicolon/case sensitivity); faster generation via the “softmax bottleneck” avoidance (512-byte output menu).
Written by AI. May contain mistakes.
Exploration of the concept of biteification and its implications for AI.
“We are exploring a new method detailed in the recent research called biteification.”
Discussion of the limitations of subword tokenization in AI models.
“So let's figure out why this barrier exists in the first place, because to really appreciate why bitification is so important, we kind of have to look at the rather clunky way AI is currently forced to read.”
Explaining why AI should process raw bytes instead of subword tokens.
“I imagine computer programming is like the biggest victim here.”
Insights into the economic and performance challenges faced by byte-level models.
“So hearing all this, the solution feels like it's just staring us in the face.”
Detailed explanation of the breakthrough in retrofitting AI models to read bytes.
“I mean, why hasn't the entire industry already shifted to this?”
Understanding the flow of the new latent tokenizer language model and its components.
“They achieved this retrofitting using less than 1 % of the compute budget it took to originally train the model.”
Explaining the boundary predictor's role in aiding the new model's performance.
“Okay, there is a specific detail about that boundary predictor, you know, the part chopping up the patches, that I find absolutely fascinating.”
Overview of the two-stage training process to integrate new reading mechanisms.
“It completely removes the need to look into the future during real-time generation.”
Learn about the two-stage training process used to integrate new components into existing models.
“So we have this beautifully complex architecture.”
Discover how task arithmetic allows for efficient updates in AI models without retraining.
“Task arithmetic is arguably one of the most powerful discoveries in this research regarding the whole ecosystem of AI development.”
Examine the benchmark results showcasing the capabilities of retrofitted models compared to traditional models.
“The implications for you as an end user or developer are massive.”
Learn how bitified models overcome traditional speed bottlenecks to enhance processing efficiency.
“But I do have one major lingering doubt.”
Explore how byte-level models reduce computational costs and enhance AI capabilities.
“Because a bitified model's final output vocabulary is just the 256 raw bytes, or 512 with our boundary trick, that softmax calculation is practically instantaneous.”
Reflect on the implications of AI learning directly from raw binary data instead of tokenized language.
“or simply hoping for an AI landscape that just isn't structurally biased towards English users, This shift away from artificial human chunks to raw binary bytes is essential.”

Get the gist, explore an idea or ask about something said.
On Best AI papers explained, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00You know, you've probably had this exact experience recently. You sit down at your computer, you fire up one of those incredibly advanced AI models, and you ask it to do something highly complex. Right, like synthesizing a 20-page financial report or something. Exactly. Or writing a college-level essay on some super niche topic. And boom, three seconds later, it spits out something brilliant. It feels like magic. But then you ask that same supposedly genius AI to do something a second grader could do. Like spelling a word backwards. Yes, or you ask it, you know, tell me how many times the letter R appears in the word strawberry.
0:39And it fails. It just completely and utterly fails. It's such a baffling contradiction to witness, isn't it? I mean, you have a system capable of passing the bar exam, yet it seems to lose its mind when you ask it basic questions about the literal letters forming its own sentences. Which is exactly why we are jumping right into today's deep dive. Welcome, by the way. We're glad you're here. Our mission today is to unpack a major fundamental breakthrough in how artificial intelligence processes information. We are exploring a new method detailed in the recent research called biteification. Biteification.
1:15And the goal here isn't just to fix the strawberry spelling problem, right? It is about fundamentally changing the basic diet of an AI, taking it away from human words and feeding it raw native computer bites. We really should establish the stakes here early on because what the research outlines, it isn't just a minor technical tweak. It is a structural shift that addresses a massive performance barrier. Right. If you are using AI for highly detailed work, whether that is drafting Python code, analyzing biological data, or just anything where absolute character level precision is required, this shift to bytes is what finally unlocks that capability.
1:55So let's figure out why this barrier exists in the first place, because to really appreciate why bitification is so important, we kind of have to look at the rather clunky way AI is currently forced to read. Yes, the industry term for this is subword tokenization. Right, subword tokenization. And the easiest way I can wrap my head around it is by comparing it to speed reading. Oh, that's actually a highly effective way to visualize it. Yeah, because if you take a speed reading course, the very first thing they teach you is to stop looking at individual letters. You know, you are trained to scan your eyes over whole blocks of text, recognizing the overall shape of words and phrases.
2:32Right, so you aren't sounding anything out. Exactly. Right. It lets you burn through a book in an hour and grasp the plot perfectly. But if I interrupt you and ask, hey, in that paragraph you just speed read, what was the third letter of the seventh word, you'd be totally lost. Because you weren't looking at letters at all. You were looking at chunks. Right. And that is basically what current AI models are doing with these tokens, right? The mechanics are very similar, yeah. The AI's vocabulary is not made of individual letters. It's actually made of tens of thousands of predefined chunks. So some are whole words, some are syllables, some are just common prefixes.
3:08Okay. When you type a prompt, the system slices your sentence into these prepackaged subword tokens before the AI ever even sees it. Which totally explains the spelling problem. It doesn't see S-T-R-A-W-B-E-R-R-Y. It just sees a single abstract concept block labeled strawberry. Exactly. And while that chunking method is, you know, highly efficient for generating polite emails or summarizing articles, it is actively detrimental in domains where the fine-grained details are the entire point. I imagine computer programming is like the biggest victim here. Oh, programming is a prime example. If you've ever tried to get an AI to write complex scripts for you, you know that missing a single semicolon or using a capitalized letter instead of lowercase will break the entire thing.
3:55Right, and you cannot just get the gist of computer code. No, not at all. And the same applies to analyzing DNA sequences in biology. A biological sequence doesn't have words or syllables. It is a continuous string of individual characters representing nucleotides. Oh, wow. Yeah, so subword tokenization tries to force human language structures onto data that simply doesn't operate that way. Exactly. It's a complete mismatch. The research also highlights something that caught me totally off guard about this chunking method. It apparently introduces logic biases just based on how things are spaced out.
4:30The spacing issue is fascinating and, yeah, highly problematic. Because of how the predefined dictionary of tokens is built, the word hello, written at the start of a sentence, might be mapped to, say, token number 400. Okay. But if you put a space before it, like space hello, the system might map that to an entirely different token, let's say token number 8052. Wait, really? Hmm. So the AI literally interprets it as a fundamentally different mathematical concept just because I hit the space bar? It does. The data demonstrates that models will generate entirely different, sometimes completely contradictory responses based purely on the invisible spacing artifacts introduced by tokenization.
5:11That is wild. Furthermore, there's a strict structural limitation regarding vocabulary size. Because the model can only hold a few tens of thousands of tokens, developers naturally fill that dictionary with the most common text available in their training data. Which I'm guessing is overwhelmingly English. Overwhelmingly, yes. This means that if you are prompting the AI in a non-Latin script, the model doesn't have neat, prepackaged word chunks for your language. So what does it do? It has to awkwardly chop your prompt into tiny, inefficient fragments. It makes the model slower, more expensive to run, and generally just less capable for non-English users.
5:48So hearing all this, the solution feels like it's just staring us in the face. If reading in subword chunks causes all these blind spots, just have the AI read the foundational language of computers. Have it read raw UTF-8 bytes. Just for clarity for everyone, a UTF-8 byte is the standard digital encoding for text. It translates the basic ones and zeros of a computer into the 256 basic characters, letters, numbers, and symbols, that make up our digital world. Right. So just feed it those raw bytes. Let it read letter by letter, character by character. But if that's the obvious fix, why are we still stuck in the speed reading token trap?
6:29I mean, why hasn't the entire industry already shifted to this? Well, because whenever researchers have tried to build models that read byte by byte, those models have severely lagged behind the performance of subword models. Yeah. But the critical context here is that this wasn't a theoretical failure. It was an economic mismatch. Okay, walk me through that mismatch. Why does the economy of this dictate the architecture? Think about the state-of-the-art models you hear about in the news. You know, the massive systems with billions of parameters. Tech giants pour hundreds of millions, sometimes billions of dollars into training them.
7:03Right. Massive server farms running for months. Exactly. They run on vast warehouses of supercomputers. Now, historically, if a researcher wanted to prove that reading raw bytes was better, they had to build their new byte-level model from absolute zero. Oh, I see. They start with an empty brain. An entirely blank slate with random initialization. And then they try to compare the performance of that brand new, poorly funded model against a subword titan that has received billions of dollars of computational education. So it's not a fair fight at all. Not even close. The compute budget simply wasn't there for the byte models to catch up and prove they could scale competitively.
7:40Which brings us to the actual breakthrough outlined in the paper. They call it bitification. And they found a workaround so they didn't have to start from an empty brain. Yes. They engineered a method to retrofit existing, highly advanced subword models into byte-level models. And the kicker is the cost. This is the crazy part. They achieved this retrofitting using less than 1 % of the compute budget it took to originally train the model. Less than 1%. So they take these existing heavyweight models, like the research specifically mentions models like LAMA-3, QUEN-3, and ALMO-3. And they essentially teach these established brains a completely new way to read.
8:16They established a direct bridge. Yeah. Okay, I really need help visualizing how this operates mechanically because in the paper, they call this new retrofitted architecture an LTLM, a latent tokenizer language model. How do you actually bolt a new reading mechanism onto an existing neural network? We can map out the flow. It's actually an incredibly elegant pipeline. It starts at the very edge with a local encoder. When you type your raw bytes into the system, they hit this encoder first. In the research, they use something called MLSTM layers for this. M-L-S-T-M. Okay, I know we promised to unpack the jargon today, so what does that actually mean for the data?
8:55Well, without getting bogged down in the deep math, an M-L-S-T-M is a specialized neural network layer equipped with a very robust memory mechanism. Okay. It is uniquely designed to process long sequences of data, like a massive continuous stream of individual bytes, without losing track of the context. So this local encoder takes your raw bytes and gives them some initial contextual meaning. So the letters aren't just raw data anymore. The system starts recognizing that, hey, these specific letters are hanging out together. What happens next? Those contextualized representations are passed to a neural module called the boundary predictor.
9:31This piece has one crucial job. It has to decide where to chop this continuous stream of bytes into a patch. And a patch is essentially the bitified version of a token, right? It's a dynamically created chunk. Correct. Once the predictor decides where the boundaries are and groups the bytes into these patches, those patches are fed into the massive global model. Wait, the massive global model? You mean the recycled brain? Yes. This is the giant billion-dollar brain they recycled from the original subword AI. Oh, wow. So the heavy lifting, like the actual thinking, is still being done by the massive original model.
10:08It's just being fed these custom chopped patches instead of those old rigid dictionary tokens. Exactly. The massive brain processes the concepts, does the heavy computational work, and then outputs its response to a depooling module. A depooling module. Yeah. This module basically unpacks the patches back into individual bytes. And finally, a local decoder translates that unpacked data into a prediction for the very next raw character to output to your screen. Okay, there is a specific detail about that boundary predictor, you know, the part chopping up the patches, that I find absolutely fascinating.
10:46The authors highlight a tweet called the One Byte Future Context. Oh, this is one of the most clever technical implementations in the entire architecture. Let's use the exact example provided in the research because it just makes it so clear. Imagine the AI is reading the phrase, the flower bed. The system is processing it letter by letter. F-L-O-W-E-R. Now it hits the R. The boundary predictor has to decide, do I chop this right now and make a patch out of the word flower? Right, because if it's just guessing based on the past, flower makes perfect sense. Exactly. But the researchers gave the predictor the ability to peek one single bite into the future.
11:27And by peeking ahead, it sees the letter B. Right. And seeing that B changes everything. The AI realizes, oh, this isn't the word flower. This is the beginning of the compound word flower bed. So instead of chopping it early, it makes a much smarter, more semantically logical decision to place the boundary after the bee, creating the patch flower bed. That one-bite look ahead drastically improves the coherence of the patches being sent to the global brain. But I have to push back here because when I was reviewing this, I hit a massive logical wall. It sounds like a time travel paradox. Time travel paradox.
12:06Yeah, I mean, I get how peeking ahead works when the AI is reading my prompt because the text is already there on the screen. But what about when the AI is generating a brand new response in real time? How can it look one bite into the future to predict a boundary when it hasn't even generated that future bite yet? That is such a great point. You've identified the exact engineering hurdle the researchers face during the decoding phase. You cannot observe a future that hasn't been computed. Right. But to neutralize this paradox, they introduced a brilliantly simple workaround involving the system's vocabulary.
12:40Okay, lay it on me. How do you bypass time travel? They added a special mathematical symbol to the AI's vocabulary, which represents a boundary. In the documentation, it's denoted by a bracket B bracket. Okay, bracket B bracket. Now, a standard byte-level model has a vocabulary of 256, representing all possible byte values. The researchers simply doubled that vocabulary size for the local decoder to 512. Wait, I'm lost. What are the extra 256 characters if the foundational computer alphabet only has 256 slots? The extra 256 items are the exact same raw characters, but mathematically fused with that special boundary symbol.
13:23Oh, I see. So instead of predicting a letter and then separately trying to guess if a boundary should come next, it predicts them as a single package deal. Exactly. The decoder might output a standard letter R, meaning here's an R and keep the patch going. Or it might output the fused version, which tells the system here's the letter R and a boundary goes immediately after it. So it predicts the character and the future boundary state simultaneously. Yes, in a single computational step. It completely removes the need to look into the future during real-time generation. That is incredibly slick engineering.
14:02So we have this beautifully complex architecture. We have new local encoders acting like high-definition nerve endings attached to a massive, recycled, global brain. But that brings us to the training process. I mean, how do you actually attach these new bite-reading nerves to an old brain without completely scrambling the billions of parameters already inside it? They used a two-stage training process called distillation. Distillation. How does that work in practice? Well, let's imagine attaching a brand new, highly advanced digital lens to an older professional camera body. If you just jam them together and start changing the internal settings of the camera body right away, you'll ruin the pictures.
14:41Right. So in stage one, the massive global brain, the camera body, is completely frozen. None of its parameters are allowed to change. The training focuses exclusively on calibrating the new lens, which is the local encoder and decoder. So you are basically forcing the new bite-level nerves to translate their signals so they look exactly like the old subword signals the brain is used to receiving? You are. Forcing synchronization, exactly. The new components learn to mimic the behavior of the original tokenizer. Once that calibration is stable, and the brain is receiving signals it understands, you transition to stage 2.
15:19And what's stage two? In stage two, you unfreeze the global brain, and the entire model is trained end-to-end. Ah, so now the system is actually learning the nuances of its new high-resolution input, but it doesn't panic because the baseline connection is already solid. The transition is seamless. And because they established that mimicking phase first, stage two requires an incredibly small fraction of the data and compute power it would normally take to train a model of that size. Which is how they kept the budget under 1%. That makes so much sense. Now, while we are talking about training, I really need you to explain task arithmetic because this concept genuinely blew my mind.
16:00Task arithmetic is arguably one of the most powerful discoveries in this research regarding the whole ecosystem of AI development. So if I understand it right, because the core global brain in the Bitified model is largely the same mathematical structure as the original subword model, the researchers found a way to just share upgrades? Yes. Consider how frequently AI companies release specialized updates. Let's say the creators of the original Ulmo 3 model release a new, highly refined version that has been explicitly trained to follow highly complex, multi-step instructions. Okay, a new shiny version.
16:40Normally, if you had built your own custom byte-level model, you would have to spend millions of dollars completely retraining it to learn those new instruction-following skills. But with task arithmetic, they don't retrain anything. They literally just use math. They take the mathematical weights, you know, the billions of numbers that make up the upgraded subword model, and they subtract the weights of the original base subword model. They just subtract them. Like taking the coordinates of a destination and subtracting your starting point to find the exact vector of the journey. That vector is the pure mathematical representation of the upgrade itself.
17:17They take that difference and they mathematically paste it directly onto their new Bytefied model. And it just absorbs the new skills. Zero extra training required. Zero extra training. The Bytefied model instantly gained the new instruction following capabilities. The implications for you as an end user or developer are massive. Why is that? Because it means shifting to byte level models doesn't isolate you. You can still instantly benefit from the massive ecosystem of upgrades, safety patches, and refinements being built for the original subword models. Okay, so the architecture is brilliant, the math is elegant, but rubber meets the road here.
17:54When they actually tested this Frankenstein creation in the wild, did it hold up? The benchmark data confirms that the shortcut works beautifully. The researchers applied this retrofitting to three major models, resulting in Bulmo 7b, Buen 8b, and Blama 8b. Catchy names. Right. And across the board, these retrofitted models vastly outperformed earlier byte-level models, like Evobyte and BLT, that had been trained from scratch. What specific tasks were they testing, though? They focused heavily on the CUTE benchmark, which specifically measures character-level understanding. Ah, the spelling test.
18:26Exactly. This tests the model's ability to manipulate letters within words, detect spelling errors, and handle complex formatting. The Bytefied models demonstrated a massive leap in capability over their subword counterparts. Furthermore, they proved far superior at generating diverse, syntactically complex computer code. So it passes the strawberry test, and it writes better code. I love it. But I do have one major lingering doubt. Speed. Okay, speed. If an AI is suddenly forced to read and generate text letter by letter, bite by bite, instead of in large chunks, shouldn't it be incredibly painfully slow?
19:04You would naturally assume so, but this brings us to a phenomenon known as the softmax bottleneck, which bitification actually solves. Okay, I hear the term softmax bottleneck thrown around a lot when people talk about expanding AI vocabulary, but I've never fully grasped the mechanics of why it slows everything down. Walk me through the math on this. Okay, in a traditional subword model, every single time the AI wants to output a chunk of text, it has to calculate a probability score to decide which chunk is the best choice. That mathematical operation is called a softmax. Got it. Now, if your vocabulary has 100 ,000 subword tokens in it, the model has to calculate 100 ,000 separate probabilities in real time before it can print a single word to your screen.
19:49That sounds computationally heavy. It is. Now, developers often try to make models faster by putting larger phrases into the vocabulary so the model can output more text in a single step. Makes sense. Bigger chunks, fewer steps. But if you expand the vocabulary to, say, 400 ,000 tokens to include those larger phrases, the softmax calculation becomes so massive that the mathematical burden outweighs the benefit of outputting larger chunks. Oh. That is the bottleneck. You hit a physical wall where expanding the vocabulary makes the model run slower, not faster. Oh, okay. So how do the bitified models escape this bottleneck?
20:29By fundamentally changing the size of the menu. Because a bitified model's final output vocabulary is just the 256 raw bytes, or 512 with our boundary trick, that softmax calculation is practically instantaneous. Wow. Calculating 512 probabilities takes a fraction of a fraction of the compute power required for 100 ,000. But wait, the output is still just a tiny letter. How do they actually speed up the overall generation? They force a higher compression ratio earlier in the pipeline. They use a technique called byte-pair encoding merges. Essentially, they tune the boundary predictor we talked about earlier and tell it, hey, pack more bytes into each patch before you send it to the massive global brain.
21:13Oh, so the system is taking bigger bytes of the data. Exactly. The patch moving through the heavy neural network gets physically longer, holding more characters. But when it finally reaches the end and has to output a prediction, it's still only choosing from that tiny lightning fast menu of 512 bytes. You have grasped it perfectly. They can arbitrarily speed up the model's processing efficiency by adjusting how densely they pack those patches without ever hitting the computational wall of the softmax bottleneck. That is brilliant. It is a profound structural advantage that proves byte-level models aren't just more precise.
21:48They offer a highly promising path toward reducing computational energy and lowering the massive costs of deploying these models. So bringing this all together, you know, for everyone listening right now, the implications of this deep dive are intensely practical. Absolutely. We've explored how the traditional method, forcing an AI to speed read via tokenization, creates invisible blind spots that ruin its ability to spell, write precise code, or process raw scientific data like DNA. Right. And we've seen how bitification bridges that gap, utilizing an elegant latent tokenizer architecture to retrofit brilliant AI brains with a new high resolution nervous system, all without spending billions of dollars.
22:30Which is huge. Yeah, and we unraveled the clever engineering of the one-byte future context and the vocabulary doubling trick, and we saw how relying on raw bytes completely shatters the softmax computational wall. The practical reality of all this data is that it lays the foundation for AI that is fundamentally more capable. Whether you are relying on AI to write flawless software or analyzing massive data sets, or simply hoping for an AI landscape that just isn't structurally biased towards English users, This shift away from artificial human chunks to raw binary bytes is essential. It really is.
23:08And it allows the models to finally perceive the world's digital data in true, uncorrupted high definition. And wrapping up, it leaves me with one final, somewhat mind-bending thought for you to mull over. We've spent the last decade forcing AI to view the world through the blurry, arbitrary lens of human language fragments. Now we are finally taking the training wheels off. We are. If these incredibly powerful neural networks start learning directly from the raw binary pulse of the digital universe, unconstrained by our dictionaries, what hidden patterns in human language, or even in the universe's data, might they soon discover that our human brains are literally incapable of perceiving?
23:49Because for the first time, the machine isn't just looking at the shape of our words. It's looking at the atoms. It's an exciting frontier to consider, for sure. Thanks for joining us on this deep dive. We'll catch you next time.
This Nature paper discusses Byteification, which offers a novel method to convert traditional subword-based large language models into efficient byte-level models that handle raw text encodings natively. By introducing a two-stage training procedure and a latent tokenizer architecture with non-causal boundary prediction, researchers can bypass the limitations of traditional tokenization—such as English-centric biases and poor character understanding—while preserving the core capabilities of established models. These byteified models achieve competitive performance on par with state-of-the-art systems while enabling high-throughput scientific applications and efficient task arithmetic post-training. Ultimately, this approach removes long-standing performance barriers and significantly reduces the computational investments typically required to build byte-level language models.