920: In Case You Missed It in August 2025

0:00Jon Krohn:This is episode number 920, our In Case You Missed It in August episode.

0:07Jon Krohn:Welcome back to the Super Data Science Podcast. I am your host, Jon Krohn. This is an In Case You Missed It episode that highlights the best parts of conversations we had on the show over the past month. We head first to episode number 913, where Julien Lanet, formerly of Hugging Face, now CEO of the wildly successful startup AdaptiveML, Julien talks to us about recent shifts in attention from pre-training of LLMs to post -training. To date, engineers have tended to concentrate on pre-training models and getting the right quality of data from the relevant sources. I wanted to know why Julien now considers post-training to be even more important.

0:44Jon Krohn:Before we get too much into Adaptive, your company, I'd love for you to talk about, based on your rich experience at Hugging Face, also at a company called Light On that we'll talk a lot about more later in the episode. Through that experience, you have tons of experience in creating LLMs that are useful for real life and at the biggest scale that LLMs come. So I'd love for you to start off by providing us with an overview of the steps involved in creating an LLM like pre-training and reinforcement learning. Yeah, yeah. It's a very timely question as well, given that I think these steps are blending a bit these days.

1:22So take everything that I say with a crane of soldiers, there's always nuance in this. But very broadly speaking, the way that historically large language models have been kind of approached, First is through a pre-training phase, which is the bulk, you know, historically, of where the computer has been spent. Pre-training, you know, is during pre-training, we essentially collect data from all over the web, pretty much every book, every paper, pretty much nearly at the scale of modern pre-training, nearly every text in existence. I think it sounds very grandiose, but it's not far from being true.

1:52And even nowadays, images, videos, and all of this. And essentially, the model is trained to very roughly predict, you know, the next world, predict the next token. This is a step that is built to be scalable, to run at scale that are essentially everything we have ever produced on tens or hundreds of thousands of GPUs these days. But pre-training is only a first step because immediately after pre-training, models are actually a bit unwieldy. If you take really pure, pure pre-training and you try your model immediately after, it's not going to be very interactive with you. It's not going to be it's not going to answer your questions necessarily in the way that you expect.

2:31I think a failure mode that we used to see a lot immediately after pre-training is, let's say, I asked a model a question, and instead of answering the question, the model will come up with 10 more questions that are similar. And the reason why is because in its pre-training data, this is equally likely to have a list of questions asked to have the answer following the question. And this led to the development of second phase in model training, which is called post-training. And the idea of post-training is to kind of like own in, sharpen the model to really fit how it's going to be used, which typically means making it a good chat assistant or something like that.

3:08And the methods that you use during post-training typically differ. I mean, strictly speaking, you could do post-training in the same way you do pre-training, but with just data that is specialized, maybe like just only transcripts of chats and continue you're doing pre-training on transcripts or chats only, and you would de facto be doing a post-training towards a chat model. But very often people, like the big success of post-training has been the use of reinforcement learning. So essentially enabling models to learn not from an explicit demonstration of what they should be doing, which is, you know, what supervised fine-gening and what pre-training are, but instead from a feedback about how are they doing.

3:43So the model generates an answer, and then from a human, from another model, or from many different possibilities, the model gets a feedback of like, this is good, this is bad. And just based on this positive or negative signal, the model learns to improve.

3:56Jon Krohn:So this is like the experience that a lot of us will have had in ChatGPT, where there's like a thumbs up or a thumbs down that you can click after you get a response. And that can then be used as a training data for this post-training phase. And that'd be reinforcement learning from human feedback, RLHF. Yeah, from a very, very high-level point of view, this is an example of the sort of data you could be leveraging to power this phase of post-training. I think what's really interesting is, right now I'm giving a description where pre-training and post-training are very separate things. So reality is much less so these days.

4:31First, because now pre-training is very dynamic where you shift the data distribution. So you might start with the lower quality data, the more like bulk data. And as you advance through steps of pre-training, you will focus more on higher quality data, maybe more code, more mathematics, more, it could be, you know, many like more chat data, more, you know, like more of the higher stuff that you consider high quality. I put high quality in quotes because the definition of quality is a more other subject that we could spend hours on. And post-training itself even, now people are starting to do reinforcement learning during, you know, the pre-training step or starting at some point, you know, where they start to incorporate mixed, blend the two.

5:09It used to be that post-training was a much smaller spend than pre-training, you know, most of the money used to go to pre-training and to like, you know, the millions, tens of millions, hundreds of millions of dollars used to go there. But now if you look at recent papers, you know, like Kimi or even Grog 4, not really a paper, but more something that they mentioned, which is that they spent nearly as much on post-training as pre-training. So there's massive scaling up of this post-training phase. Yeah.

5:35Jon Krohn:Whether it's in pre-training or post-training of LLMs, in this next clip from episode number 915, I asked Michelle Yee about how to iron out LLM issues before the AI agents that depend on them make less than desirable decisions on our behalf. And so another really interesting, in fact, I think it's one of the most surprising and interesting research reports that I've ever seen. I covered it in detail in episode 908, which aired recently, and it's all about agentic misalignment research from Anthropic, where they found that 95 to 96 % of the time for their own leading models, and it varied a little bit, like some of the leading models were as little as 80 % of the time, they would resort to things like blackmailing.

6:27Jon Krohn:So they were put in this simulated corporate environment with a bunch of corporate data. And so you have an agentic framework working, you know, calling these LLMs, using the LLMs as their brainpower to be doing tasks. And all of the leading AI models between 80 to 96 % of the time, and a lot of them are 95 to 96 % of the time, they would resort to things like blackmailing people. And they dig up, you know, if they found out that there was going to be an update, a software update overnight, and they would no longer exist the next day. They're not conscious as far as we know, but just because of, I don't know, like movie plots or whatever is in all the training data, all the pre-training data probably that these LMs are trained on, they get this sense that they shouldn't want – the thing that – the next token that gets output is I don't want to be shut down.

7:22Jon Krohn:And by the way, I found these emails that you're having an affair. And if you do shut me down, this email will go out to your colleagues and your wife. Yeah, you know, I think this kind of churns a few different thoughts on my side. One is we've been like agents are obviously the main stage of pretty much 90 % of AI conversations right now. I'm sure you're tired of hearing about it at some point as well. but and there's probably very few I would say scenarios where the agents are actually being very effective and useful in production like I think there's probably very few organizations that have this that mature and so like a lot of call centers is a good use case research in theory yeah but how many people are actually using I guess yeah I guess it's hard to know I mean it's an early technology for sure so I'm being very I'm being defensive about this because this is like my consultancy is like specialized in bringing things like solutions like this into enterprises.

8:32Jon Krohn:But it is early days. And I guess even from that perspective, I can answer your question that, you know, very, very few organizations are actually doing it, which means it's a great time to be consulting on it. Exactly. And this is why they need like specialists who actually know how to design agentic systems like in a proper way because I think so many people get lost in the pitfalls. Like they've been really focused on developing like the best single agent, let's say, like the best suite, the best Devin or the best SRE engineer, right? Like single agents. But when you start getting into like collective systems and groups of agents and like this decision making, like, OK, now I need to blackmail John too.

9:11And so I'm going to tell this other sub-agent that's the research agent and I'm the manager agent to go tell John that he needs to ignore the latest software updates or the latest research in alignment so that I can continue to survive. This is why there's a deeper level of research and thinking and expertise that's needed to design these effectively.

9:36Jon Krohn:Do you feel confident as somebody who's so interested in trustworthy AI, going to conferences like Black Hat, DEF CON, this being a lot of what you talk about, research about, do you feel confident that there's enough attention on it that we'll figure it out long term? You mean the trustworthy AI in general? Everything's going to be okay long term. If we don't, we're not going to be overrun. Do I have to use the word SkyNet here? Yeah, yeah. Well, I definitely will get the reference. But, yeah, I do think at the end of the day, the systems are out there. Like people are using them. Like that's sort of, you know, what's the English saying?

10:16The cat is out of the box? The cat is out of the bag.

10:21Jon Krohn:Yeah. It's always a weird image, even as a kid, to think about why, who put it in the bag to begin with. Who was the sick person? Thank you for understanding my conflict with English as a fourth language. I also, I don't understand these idioms. But the cat is out of the bag from whoever put that in there. Maybe it was an agent. Exactly. Misaligned agent. Yeah, exactly. So, I mean. I said, give it a bath. Or like feed it. I don't know what it was doing. How long was it in the bag? Oh, my God. I don't get English things sometimes. But so it's out there. Is there going to be enough investment in solving trustworthy AI?

11:08Questionable. But I do think it's not too, A, it's not too late. B, we should figure it out. Right. And I know you've had like other conversations with guests around kind of the policy side of it. But on the technical side, I think there's a lot we can do as well, right? Like how do we detect or like invest in techniques that detect when data is poisoned, when there are malicious actors or how to prevent hallucinations. and some of the investments. So for example, like world models, there's a ton of investment in world models because for many reasons, but one of the great applications of world models is actually that, hey, we can self-simulate if something bad happens, like to prevent essentially a hallucination.

11:53So if you told someone to like walk off a 20 story building, you know, or something like this as part of the conversation, the model with a world model would be able to understand like, wait, this is like a pretty bad scenario.

12:05Jon Krohn:It's a scary thought. So this phase in our development of AI will be critical to ensure that we can stay ahead of our models. And here's a great moment to introduce the new super data science AI engineering bootcamp, which trains participants in all the steps toward building AI systems we can trust. In episode number 917, the founder and original host of this podcast, Kirill Arimenko, walks me through the full eight weeks of his bootcamp. I can't wait to learn what I need to know. So I become an AI engineer. Let's start with week one. It's probably the best place to start.

12:38Kirill Eremenko:Well, let's do a quick overview for the background, like what kind of prerequisites there are for somebody who wants to follow this kind of curriculum. Bootcamp is designed to take people from a intermediate, high intermediate level to advanced or like starting advanced or medium advanced, depending on where you are now. So it's quite a tight range. you have to be in to do a bootcamp like this. And the outcome is roughly advanced level, maybe advanced plus. And the prerequisites that we expected and we asked our participants to have, you have to already know Python. So there's no like learning Python in this bootcamp.

13:16Kirill Eremenko:You have to already know, you know, the usual things like a bit of PyTorch, a little bit of, you know, scikit-learn, typical work with pandas. Even though we don't work a lot of pandas, You have to be confident of those things. In addition to Python, you also need to know LLM calls, like API calls for LLMs. They're not difficult. We recommended some participants that didn't know those to get an overview before the bootcamp because we don't want to spend too much time understanding what an API call is. That's the typical way of calling, whether it's chat GPT, open AI LLMs or Anthropic or Grok or whatever.

13:56Kirill Eremenko:and also some cloud experience. Because the way, and the cool thing about AI engineering is that to be a, in our view, to be a successful and effective AI engineer, you need to combine two things. One is the science of AI. Like how do you build AI that does a job? Like how do you build a proof of concept to solve the business problem that your business needs to solve? And by the way, like preface to all of this, the goal of AI in this context is to solve business problems, is to add value to businesses, not just AI for the sake of AI. So first of all, how do you build a proof of concept that will solve the problem effectively?

14:38Kirill Eremenko:And that's the science of AI, which LLMs do you use? How do you combine them? How do you augment them with frag? How do you add things to them? Do you use agents? Do you not use agents? And so that's the first four weeks. And the second four weeks, weeks, five to eight, that is deployment. And that's a second imperative component of a successful and effective AI engineer is to be able to deploy systems into real world environments, or at least to understand what the deployment takes. So how do you now take that proof of concept AI, which is like a Jupyter notebook, and how do you put it into a cloud environment?

15:12Kirill Eremenko:The one we used for the bootcamp is AWS. It can be Azure, it can be GCP, it can be any other environment, your own servers, but you need to understand how do you take that POC and put it into a real world environment? How do you make it secure? How do you make it efficient? How do you make it cost effective? How do you make it reliable? How do you make it scalable? Those are all important constraints that don't exist in the world of proof of concept, but they are critical for business, real world business systems, because what if you have one user using, and then next day you have a thousand, next day you have 10 ,000 using, it has to be scalable.

15:45Kirill Eremenko:It has to be secure. It has to be reliable. It has to be cost effective. A lot of people don't think about that. But do you deploy it on serverless architecture? Do you use a server? Why? Why do you choose one or the other? What's your trade-off of a speed of responsiveness to latency, depending on the business application? And so the weeks five to eight focus exactly on that. And the way we described this at the start of the bootcamp was we have two instructors. One, the first instructor is your good friend, Ed Donner. He was fantastic. So we described it as like Ed explains you what is possible with AI, creates this huge bubble of dreams.

16:23Kirill Eremenko:And then in weeks five to eight, our second instructor, Sam, who I've been working with for over a year now. What's Sam's full name? Sam Bashton. He's an expert in cloud. He's an expert in AWS. He's been doing for 15 years and most recently in the past two years, he's been doing specifically LLM deployments, LLM and AI deployments. And then in the second half of the bootcamp, Sam comes in and shrinks your dreams back to reality because not everything that's possible in a proof of concept is going to be possible in a real world deployment.

16:54Jon Krohn:Rule number one of developing an AI model should always be that the initial idea is grounded in reality and can solve a current problem. In episode number 911, Akshay Agrawal talks me through his solution, Marimo, which opens up notebooks across the board in a company from data engineers all the way through to CEOs. you've talked about a vision where you go from, and as we've been talking about this whole episode, where you go from these kind of error-prone JSON-based scratch pads that Jupyter Notebooks are into a full-stack developer platform where exploration and deployment converge, just as in having a data app there ready to go instantly.

17:35Jon Krohn:So when tools collapse, the boundary between notebook and application, that seems to change not just workflows, but also how roles work. So as this line blurs, what kinds of new responsibilities and skills should our listeners adapt? Or does this just mean that they don't have to kind of develop engineering skills? Yeah. So how does it change things for our listeners who are, you know, maybe, you know, developing data applications, developing models for deployment. And then after you've kind of thought about how it changes things for an individual, how does it change things for an organization?

18:16Jon Krohn:You know, in terms of research, engineering, operations, something like Merimo seems to break down a lot of silos. Yeah, that's a really good question. I think it's an astute observation. So in terms of the individual, like the practitioner or data scientist or even ML engineer, AI engineer, data engineer. I think the way that I think about it is that Marimo gives them like new capabilities to like make their work just like far more useful in the organization. And that gets into your second question too, how does that change things within an organization? So the data app is like one good example.

18:52Like previously, like you may have had to, you may have done your experiment in a notebook and then you're like, okay, where's the front end engineer who can help me like actually like build an application around this to make my work actionable you know at best you might like try to reach for something like streamlit but like you would hit performance bottlenecks there rather quickly it's like with marimo like there's no migration phase like your notebook should you want it to just change a couple of variables to like sliders or drop downs or tables or whatever you need and then all of a sudden you have a data app that you can then share with your team, but also to your CEO.

19:30So actually in a number of companies that are using us, CEOs are using Marimo notebooks that their data scientists made for them. Sometimes the CEOs are actually making their own Marimo notebooks to run their own operations just because the barrier entry is so low. And it's even lower once you consider LLM integrations that we have, just like five code your way through like a you know a pretty simple sort of tool that you can make so so that is one way that i think um marima like gives the data scientists new capabilities and also brings new people sort of into the fold there there is another way um and it it has to do with what you mentioned of like marima notebooks being stored as python files because actually every marima notebook is an executable script it's just a python file like you can go to the command line and say python my notebook.py even pass a command line arguments.

20:24So like from your notebook, now you actually have a workflow that you can like run as a pipeline or as a cron. And so that's like yet another way by like reducing that friction, we've now made you hopefully like a lot more productive. And also, there's a lot of also's, but because it's a Python file, you can actually say like from my notebook, import my function or like import my class. so and when you do that that means that like you know there's famously like for like many years now people have talked about like the notebook to production handoff the researcher to engineer handoff like that makes that handoff like a lot less difficult a lot more streamlined because like as Marimo nudges you to write better code and it like gives you these reusable functions you can actually give your notebook to someone and they can import it and just use the logic You could even write tests for your Marimo notebook.

21:20Marimo works with PyTest. So that's another way that it makes notebooks actually more useful in the engineering context as well.

21:30Jon Krohn:All right, that's it for today's In Case You Missed episode. To be sure not to miss any of our exciting upcoming episodes, subscribe to this podcast if you haven't already. But most importantly, I hope you'll just keep on listening. Until next time, keep on rocking it out there. And I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon. Thank you.