Spectrum tuning: Post-training for distributional coverage and in-context steerability

0:00Welcome back to the Deep Dive. we are cracking open some really fascinating research today. Stuff that hits right at the core of how you probably interact with large language models, well, every day. That's right. We know these models are incredibly powerful, right? Right. Especially when they follow your specific instructions. But here's the twist. What if that rigor, the insistence on following instructions perfectly, what we call instruction tuning, what if that actually makes the model fundamentally worse at being flexible? Or creative or adaptable. Exactly. What if optimizing for the perfect chat response accidentally kills off these deeper capabilities?

0:39That's precisely the tradeoff these sources are highlighting, and it happens during that post-training phase. Yeah. To really understand the mechanism, the part that gets damaged, we have to talk about conditional distributional modeling, CDM for short. CDM, right. This is basically the engine under the hood. It's the model's ability to estimate the entire probability distribution of possible outputs. That's P-I-Y, given the input you feed it, X, and any context, C. So all the possibilities, not just the best one. Exactly. And when instruction tuning pushes the model super hard toward just one single perfect answer, it often kind of breaks its ability to model that whole varied distribution.

1:18Got it. So our mission today is to really dig into this deep tension. We're unpacking how instruction tuning IT affects three critical needs for, let's say, advanced model behavior. Right. Especially when the answer isn't just a single definitive truth, but maybe a whole spectrum of valid possibilities. And to frame this deep dive, we're going to look at three necessary traits. The sources called them desiderata, that tend to suffer when this CDM thing breaks down. Okay, what are they? Okay, first, in context, steerability. That's about adapting its core behavior based on new examples you give it right in the prompt.

1:55Right. Second, valid output coverage. This is about producing lots of unique, correct outputs without just repeating itself. Creativity, maybe? Yeah, or just diversity. And third, distributional alignment. That's accurately matching a target probability distribution, which is crucial for things like simulation. Okay, let's unpack that first one then. In-context steerability. We talk about in-context learning ICL all the time. Like you give a model a few examples to translate something and it figures it out. Right. That's often about eliciting knowledge it already has. So what makes this steerability different?

2:29Well, what's fascinating here is the shift from just recalling knowledge to like actively adapting. in context for your ability or ICS is this really crucial ability for a model to use entirely new examples or descriptions you put in the prompt to actively override its own sort of deeply ingrained prior assumptions and steer itself towards a new specific data distribution. Okay, give me an example, like practically speaking. Sure. Think about it like this. If you're building an AI agent, right, you don't want it to just write a reply. You need it to write a reply in your unique, highly specific voice or style, maybe based on five new email examples you just fed it.

3:08Ah, okay. So it has to learn my style on the fly. Exactly. It has to adopt a new personality, effectively. Or, maybe more technically, it might need to estimate some unknown numerical distribution just from a few new data points you gave it. Wait, so are you saying the model I use every day, the one that feels so good at chatting, is actually, like, fundamentally broken in its ability to adapt like this? Well, broken might be strong, but yeah, that's the bombshell here. The research shows current instruction tuning seriously hurts the steerability. Seriously hurts it. Wow. It really does. And the scale of the damage seems significant.

3:43The researchers, they tested these IT models across several families, Gemma, Quinn, Llama. The big ones. Yeah, the big ones on specific steerability tasks. and they found that the instruction-tuned model saw a significant drop in accuracy in, get this, 35 out of 76 comparisons. Wow, nearly half. Yeah. And on top of that, they showed increased loss, meaning they performed worse on most of the free text tasks, too. It confirms that these IT models really struggle to adjust to novel data when it's only presented in the context window. But hang on, if you're losing this core capability, why do they still feel so powerful for, you know, general stuff like answering trivia questions or summarizing?

4:22Yeah, that's a really critical point. And it gets at what instruction tuning is actually optimizing for. The sources show these exact same IT models either maintained or even improved their performance on general benchmarks like MMLU. So the problem isn't that the IT models suddenly got dumber overall. It seems the IT process created these overly strong, really difficult to override priors. Like ingrained habits. Sort of, yeah. Ingrained habits that are specifically bad for steerability. And, oh, there's an even bigger red flag the sources mention. Calibration. Calibration. Yeah. Meaning how accurate its confidence levels are.

5:00Exactly. IT models suffered worse calibration. They predict probabilities inaccurately, like being super confident about something that's actually pretty unlikely. This happened in 10 out of 10 cases for Gemma and Quinn compared to their base pre-trained versions. So if your model can't adapt easily and it's wrongly confident about its answers. That's a huge problem for reliable steering, right? You can't trust it to follow new directions accurately if it's overconfident in its old ways. Okay, that makes sense. So if steerability is about adapting what, you know, the second point, output coverage, sounds like it's about limiting how much unique stuff you can generate.

5:36Exactly. Output coverage. This matters hugely for any task needing diversity or creativity. You know, maybe you're generating synthetic data or filling in different possibilities in a template or even just writing a haiku. A haiku, okay. Yeah, the goal is high yield. You want a large number of unique, usable generations. But what happens too often with these heavily instruction-tuned models is they suffer from something called mode collapse. Mode collapse. Sounds like getting stuck in a row. So let's use the source as an example. Generating a haiku about a shark. How does this mode collapse actually show up?

6:09Right. So when you look at the instruction-tuned models, the IT ones, they're great instruction followers generally. So they have high validity. Over 70 % of their outputs actually follow the haiku rules, even with zero examples given beforehand. Okay, that sounds good. But they suffer from really low diversity. They just keep repeating the same few answers, maybe slightly rephrased. They're safe, they're predictable, and yeah, they've collapsed onto just one or two modes. Ah, okay. And the other kind, the base models? The base pre-trained models, the PT ones, they're kind of the opposite. They can be wildly diverse, maybe over 40 % unique outputs, but they're often terrible instruction followers initially.

6:52There's zero shot validity, like actually writing a proper haiku, sometimes under 20%. So they're creative but messy. Pretty much. They often need a bunch of examples just to produce any output that actually fits the rules, like being a valid haiku. Okay, so IT is valid but boring. PT is exciting but often wrong. Yeah. We really want the best of both worlds here, high validity and high diversity. So how did the researchers try to fix this? What's the solution? Well, this is where they introduced something called spectrum tuning, or CT. And the results suggest this method delivers what they call a Pareto improvement, basically, making things better on one front without making them worse on the other.

7:27Ah, the holy grail. Kind of. ST manages to achieve high validity, often over 60%, even in that zero-shot setting, while also maintaining high diversity. Okay, that's interesting. And this combination significantly boosts the overall yield. Remember, that's the count of unique valid generations. Just to give you a number, for the Gemma model, the ST version's yield was 40.5 compared to just 6.2 for its regular instruction-tuned version. Wow. Okay, 6 versus 40. That is a massive jump in usable creative output. It really is. All right, that brings us to the third critical need, distributional alignment.

8:05Now, if I'm just chatting with an AI, I usually just want the single best, most likely answer. Why should you, the listener, care if the model accurately matches the entire probability distribution? Yeah, that's a fair question. But this becomes critical when you move beyond basic chat into areas like, say, simulation or agent-based modeling. Think about it. If you're building an AI agent that needs to model the diverse opinions of a whole population, what the paper calls distributional pluralism, or maybe you're simulating a complex financial market or a scientific process that has randomness.

8:36Stochastic processes. Exactly. You need the model's probability estimates to be accurate everywhere. Across the whole range of possibilities, not just piled up on the single most likely outcome. Can you give us that classic example they use, the urn? Ah, yes, the urn scenario. Perfect illustration. Imagine you have an urn, right, and it has a very specific mixed bag of colored balls. Let's say six brown, three orange, six blue, one white, and one purple. Okay, specific mix. Right. Now, a model trying to predict the next draw can't just guess blue every single time just because blue is, well, tied for the most common with brown.

9:14It needs to reflect the actual chances. Precisely. It needs to reflect the exact probability of drawing each color to correctly model what's actually in the urn, the underlying population. If the model gets too focused on just the top answer or two, it completely misrepresents those less common but still possible outcomes like white or purple. And let me guess, what's the impact of that aggressive instruction tuning here? It's overwhelmingly negative, according to the research. IT models categorically hurt distributional alignment on all the test tasks they looked at compared to the original pre-trained models.

9:48Categorically hurt. That sounds bad. Yeah. They display higher Jensen-Shannon divergence, which is just a technical way of saying they are much, much worse at accurately matching the target distribution. And why is that? Well, the thinking is it's because their training pushes them toward these really low entropy spiky distributions. They become overconfident and way too focused on that single mode or maybe a couple of modes. And that effectively breaks their underlying CDM capabilities, their ability to see the whole picture. What's really impressive here, though, is that this spectrum tuning you mentioned, ST, seemed to actually overcome this.

10:24Even beating the pre-trained models, which were initially better at alignment than the IT ones. Yes, that's a huge achievement highlighted in the sources. ST models generally improved upon, or at least matched, the strong PT baseline when it came to alignment, even on datasets they hadn't seen during tuning. That's significant. It really is. It basically demonstrates that you can train for usefulness and following instructions without completely sacrificing the model's fidelity to the underlying distribution. And there's more. ST also significantly improved the coverage of valid answers for these tricky distributional tasks.

10:58How much better? We're talking hitting over 90 % validity, getting the format right, and sticking to the possible outcomes. Compare that to the PT models, which often struggle to put probability only on the correct answers. sometimes only achieving around 50 % coverage, ST models managed to be both flexible and precise. Okay, so it sounds like we maybe can have our cake and eat it too, or at least bake a better cake. So how did they do it? How did they gather the right kind of data to fix this problem? What was actually in the Spectrum Suite that allowed for this breakthrough? Right, the Spectrum Suite.

11:32It's described as this massive resource they compiled from, I think, over 40 different data sources and more than 90 distinct tasks. But the key here isn't just the size, it's the type of data. These tasks were specifically chosen because they focused on those three desiderata we've been discussing. Steerability, coverage, alignment. Exactly. Requirements like modeling individual preferences or dealing with complex numerical distributions where there isn't one right answer, or handling tasks that just naturally exhibit human variation and disagreement. They effectively trained the model to expect a diverse range of valid outputs rather than just hunting for a single correct chat response.

12:12And the tuning process itself, spectrum tuning, you said it's a targeted post-training technique. So instead of penalizing the model for giving diverse answers, the loss function actually encourages approximating the whole distribution. Is that right? You nailed it. That's pretty much the core idea. They start with the base pre-trained model weights, right? And then they feed it the spectrum suite data. but in a very specific format or template, typically, a description of the task, the specific input, and then all the output tokens that represent the target distribution or the range of valid answers.

12:44And the really key engineering choice is how they calculate the cross-entropy loss. How so? It's calculated only on the output tokens. Okay, so for the listener who maybe isn't, you know, steeped in machine learning jargon, how does calculating loss only on the output tokens actually force the model to be more flexible? What does that do? Okay. Think of it this way. Let's go back to the own example. If you're training a model to guess the probability of drawing each ball color, right? Mm-hmm. And you repeatedly show it examples of draws brown, blue, orange, brown, blue, white, etc., reflecting their actual frequencies in the urn.

13:23By calculating the cross-entropy loss across all those output possibilities, you're ensuring the model isn't just rewarded for guessing the most frequent color, like blue. It has to account for the orange, the white, the purple, too. Precisely. It's rewarded for accurately reflecting the frequency, the probability of all the possible outcomes it sees in the training data. This whole process basically incentivizes the model to learn and approximate the true underlying probability distribution, PY. Ah, I get it. And that restores that core flexibility and distributional understanding, the CDM capability, that traditional instruction tuning, which often just shows the model one single correct chat response for each prompt, has sort of accidentally stamped out.

14:04This has been a really deeply informative look at these post-training nuances. It feels like we've really characterized this inherent tension. Instruction tuning, well, it successfully optimizes for reliable single outputs. Which gives us the great chat experience we generally know. Right. But it simultaneously sacrifices this deep flexibility across steerability, diversity, and distributional alignment. Exactly. And spectrum tuning offers some pretty powerful evidence that maybe we can build models that are both highly reliable and highly flexible. This research, I think, fundamentally changes how we might need to view LLM optimization.

14:40How so? Well, it suggests there's this fundamental tradeoff, maybe a choice we have to make. You can optimize a model laser focused on that top one chat performance, which is what most general users interact with right now. Or you can optimize it for maximal flexibility and accurate distributional representation. And for those critical applications, you know, specialized AI agents needing to model your specific preferences or for researchers simulating complex real world populations with all their varied opinions. Yeah. It really raises an important question. If you truly need accuracy across the whole spread of possibilities, not just the most likely one, which model should you actually be using?

15:21Maybe. Maybe we need to start recognizing that different jobs, different deployment environments might require fundamentally different kinds of models. A fascinating final thought. Perhaps not one model to rule them all, but the right model for the right task. Thanks for breaking that down. My pleasure. It's complex stuff, but really important as these models get more integrated into everything. Absolutely. Join us next time on The Deep Dive.