⚡️GPT 4.1: The New OpenAI Workhorse

0:07Hey everyone, welcome to the Lit in Space podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host Swix, founder of SmallAI. Hey, and today we have a returning guest as well as a new friend. Welcome, Michelle and Josh. Hi there. Both of you work on the... I guess, Michelle, I think I used to introduce you as manager on the API team. It seems like you've changed your role since we last talked on the podcast. Yeah, now I lead a team on the research side, specifically in post-training. Yeah. And Josh, you are also on post-training. Yep. I'm a researcher on Michelle's team.

0:43Yeah. And I just found an interesting commonality you guys have. You're also both from Waterloo, continuing the tradition of extremely cracked engineers. Oh, yeah. We talked about that last time. That's right. Okay. So we're gathering to talk about GPT-4.1. You launched it. I mean, we got a little preview and it was a little bit rumored, right? It was pre-released, I guess, with Open Router as Quasar Alpha. and then there was also an Optimus version. And I think people are trying to figure out why are we going back from 4.5 to 4.1? There's a whole bunch of other things, but what are the headline facts, I guess, you guys want to emphasize about 4.1?

1:20Yeah, I'll just say we released three new models today, GPT 4.1, GPT 4.1 Mini, and GPT 4.1 Data. And the real focus on these were just making models that were great for developers. so we improved instruction following coding and shipped our first 1 million context models anything to add i don't know if there's anything else that people should really that are like sort of in the fine print no i think the only thing that i would touch on maybe twice is that there's actually a new model in the lineup nano which is even faster for developers that are making you know low latency applications and cheaper what's the any fun story behind the codenames or, you know, I got the strawberry hat as another fun time in the lore of OpenAI.

2:07Yeah. Yeah. We really wanted to get as much developer feedback as possible on this model to make sure it worked well in the real world. And so we tested it kind of through OpenRouter and it was super cool to see people latch onto the names and get the theories going. But the feedback we got from there was super helpful. Yeah. Yeah. It's not even like the name. it's more about just like the API shape. Once we saw like chat kumple, it was like very obviously open AI. Yeah, it's a good note. Yeah, but like, I mean, okay, is there like an emphasis on stars? Like what inference were we supposed to draw from, you know, quote unquote, super massive black holes?

2:50I don't think there's anything really to draw from there. They're just cool. Just find the code names. You know, they make you think of cool concepts. The vibes are good. The other thing about the examples, we're just mining for lore here, right? The very interesting animal comes up a few times on the live stream and on the blog posts. What's up with Tapirs? Who likes Tapirs here? Yeah, our team is just a super big fan of Tapirs. So they just happen to work their way into a lot of our content. Okay, cool. Yeah, go ahead. But I think like the first thing that, yeah, we just want to run through is obviously the 4.1 to 4.5.

3:29I think that's the first thing that everybody was maybe confused about. So I don't know you're deprecating 4.5. It sounds like 4.1 is just like a kick-ass model. And the 4.5 size, maybe it's not as good of a fit. That was just a research preview. So yeah, I don't know. Whatever you want to say to address that. I think it's something we've seen come up also in the Discord. Yeah, totally. Okay, naming is really hard. and we've tried to make this as less confusing as we can, but, you know, nothing's perfect. Basically, the way we got here is that GPT 4.1 is like a pretty big improvement over the 4.0 line, and we really wanted to signify that.

4:08However, it's a model that's like much smaller and cheaper than GPT 4.5, and as a result, you know, doesn't achieve the same like AIME or other intelligence evals. So it doesn't beat 4.5 on all of the evals, And so we didn't think it made sense to increment beyond 4.5, but we do think for most developers, they can kind of replace a lot of their 4.5 usage with 4.1. And then the mini is strictly better than 4.0 mini. Yeah. Yeah. With the nano. But we don't know if 4.1 is a distillation of 4.5 or there's no relationship there. What can we say about the shared linear? Yeah, what I'll say there is we're always using various research techniques to improve our models.

4:55And distillation is something we talked about before. It's really meaningful, especially for the small models. And we've kind of pulled out some of the things that made 4.5 really good. Like it has a lot of the instruction following greatness and also rolled that into 4.1. Awesome. I think one of the, because I strongly remember on the 4.0 launch that their communication was that we're kind of moving to a new model architecture that is omni-model, right? That's the O in 4.0. And then 4.1 is part of this subsequent trend of trying to merge everything, like the reasoning model, the omni-model, everything.

5:34And I think that there's this doubt about whether 4.1, I think it's basically trying to be sold as a strict replacement for 4.0. But I don't know, is it going to be fully omni-model? Is it roughly the same architecture that we think 4.0 has? So we already have different slugs on the real-time API and responses API. So they're already somewhat different checkpoints. We don't have any current plans to release 4.1 in the real-time API. But, you know, things may change. Yeah, and then there's ImageGen and all that, right? And as far as we know, no plans, maybe, but nothing announced. Not right now.

6:15The focus for 4.1 was kind of these three core capabilities for developers. Yeah. Our Discord actually also did a launch, a watch party for the recent 4.5 podcast that Sam Altman did, where I think for the first time, it was basically kind of confirmed that something that people already knew, like Andre Karpathy was already talking about this. that 4.5 was like 10x the size of four. And I think there's a question about like, do we do the linear interpolation of 4.1 is like, you know, zero point, is like, I don't know, 2x the size or something? That's not really how we think about naming the models.

6:50There's a whole bunch of parts that go into the recipe. And so, you know, it doesn't really reflect on just the pre-training recipe, our version numbering. But I think the 4.1 is just because of the large jump that we have in like coding capabilities, long contacts and so on. It's more so what it's like for the end user, more so than anything about the training recipe. We can go a little under the hood on training though. And we'll say that, you know, Nano is obviously a new pre-train. We also have a new pre-train for Mini. And then the larger version is a new mid-train. But we find that actually a significant amount of the gains come from new post-training techniques.

7:29And so I think in the past, the narrative is that you need to pre-train these larger and larger models to get better performance. And we're finding that we're able to squeeze a lot more out of post-training now. Talking about how big a model is, the other side of it is the context window. You have a 1 million context. I know that Sam at Dev Day last year, he said that 1 million was like months away. So right on time. Can you talk about how hard that was to get to 1 million and then maybe where the end game is in your mind? Is it 10 million, 100 million, infinite? What really matters as you start to scale this?

8:09Yeah, Josh worked a lot on long context, so he's the right person to ask. Definitely. So I think the first thing that I thought was really interesting when we were going to long context is actually some of the evals that you see as like headlines on maybe other blogs where it's needle in a haystack. Actually, most of the models do really well right out of the box. But then we had to actually first get a lot of measurement on the longer context for long context reasonings. You know, we actually just open sourced two new evaluations that are about using the context in a more complex way. So, you know, one of them, you have to reason a lot about ordering and the other is actually walking through graphs.

8:44So there's a lot of reasoning that you have to do in those data sets. And that's where doing long context is actually much harder. But single needle in a haystack, we were able to saturate pretty easily. And then most of the work came at these harder tasks. Yeah, I was going to say how, no, just how you think about length of context in terms of consuming documents versus like active kind of like thinking and planning. I think there's obviously a whole part on the prompting side around building a gentic workflows. do you think that people maybe like still think too much of it about yeah needle in a haystack kind of document retrieval versus like traversing very long plans and kind of like iterations in context yeah i think the mental model that i have is maybe actually has some more variables in it so there's single there's the needle in a haystack where you have like some amount of distractors and some you know needles that you're trying to find and i think that it's more so about how dense of the context you need to use.

9:44So like summarization, you're actually just using the entirety of the context, whereas, you know, needle in a haystack, it's very sparse. And then I also generally think about orderedness. If you're going to make some sort of inference on this, are you just looking, you know, sort of front to back, or do you need to move around in the context in order to generate a good answer while the model's sampling? Yeah. Is that something that you worked on with GraphWalks? Is that... Yeah, that was sort of the most synthetic and clean way to measure the model. And then, you know, we worked on a lot of other training techniques, data to sort of test the model's ability and train in the model's ability to reason throughout the context in a sort of shuffled way.

10:24Yeah, you know, actually I have the ability, I like to give people a little bit of visual aid with these things. So I actually went into your Hugging Face release and got an example of the graph task. And so there's a few versions of this, right? There's like the BFS and DFS version. And also, I guess it's very character specific. So I don't know, maybe could you tell us like, you know, design choices around this? Like what was surprisingly hard? You know, anything like that. Yeah. So the idea here is you take a graph and you encode it into the context by looking at the edge lists and just putting that into the context and then asking the model to do an operation.

11:01And then, you know, under the hood, we're actually just executing the real operation and using that to then evaluate the model's ability to work. One of the things that I found surprising at first was the what the model would do when it wasn't sure how to use its context. You know, early versions of the model just sort of looping, saying like, oh, no, I can't find this edge that I think should be there. And yeah, I think I was actually very surprised how all models seem to have more difficulty than I would have expected on a task that, you know, we would find very simple or like, you know, maybe an undergrad could write a Python script to run in a couple of minutes.

11:42Yeah, right. Okay, so like, what is the real life task that this is meant to model, I guess? You know, I feel like the other one, MRCR, seems a little bit more intuitive where, you know, you have like four different stories and you pick out the second one. And that's a real task that people have. But people don't really traverse graphs. Like this is a bit more theoretical, but like, you know, was there any sort of correlation study done? Yeah, this is actually meant to be sort of the idealized version of like a multi-hop reasoning benchmark. So we have a lot of things where, you know, you're putting hundreds of documents into the context, and then you might ask a question that you actually have to traverse 10 documents for.

12:25But there, the edges are, they're implicit, right? Like there is some underlying graph that's connecting all of these documents that you need to traverse in order to answer the question. But they're actually much harder to traverse because the edge isn't actually given to you. And so the question there was like, okay, if I actually just give you all of the IDs of these things that you need to traverse, can the model even do that? Where it's actually just a lower bound on how well the model can do. And I think that's actually somewhat well reflected in some of the internal benchmarks we have that are using more natural data.

12:57Imagine something like a tax return, right? Where you upload the entire tax code. And to figure out what to put into this box, you'll need to reference all of these boxes. And so this is like a similar level of multi-hop reasoning. But again, like Josh said, all of the references are implicit. Yeah, I think that some kind of backtracking, if it's needed, is also super interesting, especially for agent work. For listeners who've been listening to us for a while, we actually covered this paper in Europe's last year, two years ago, COG-Eval, where they actually modeled graphs for graph traversals for agent planning.

13:33And it reminds me closely of that. It's just that they never came up with this exact format that you have here, which basically is the same thing. I also like that you included blank answers because sometimes people do hallucinate or models do hallucinate answers and you have a fair amount of blank ones. Thank the random sampling over graphs I did, I guess. Yeah. Is this tied also to the file search API that you released recently? Like how should people think about how everything kind of comes together in the API? Yeah, I think oftentimes with retrieval, you might be using RAG to fill the context.

14:14And a lot of this is to get around the limitation of a short context window. So we do expect a lot of developers to start uploading their full context more directly to the model. So for smaller tasks, you maybe don't need the whole vector store. But we do anticipate this to play well with that paradigm as well. Like maybe you can just insert way more chunks into the context. So we think it'll play nice. Yeah. Any relationship to the memory upgrades in ChatGPT that we recently got? Is long context just directly usable for memory or should we just always have a separate memory system? Yeah, it's a good question.

14:52So right now, the dreaming feature, we kind of have some of these memories embedded in the context. But, you know, they are separate features. So 4.1 is powering the API, whereas the enhanced memory is chat GPT only. Yeah. Awesome. Yeah, I think that's interesting. I guess the one last thing I'll call out on long context, which is kind of unintuitive, intuitive or maybe there's an explanation which was the you had two needle for mrcr and then we had four and eight and everything kind of just regresses to some kind of baseline of like let's say 30 or 20 as that but it's interesting to see where the smaller models sometimes match or outperform the larger models i was wondering if there's anything unusual there or do you think it was like a bad roll of the dice i think it's probably just a bad roll of the dice i think i I would probably look more so at the larger one.

15:50These things are grasses you and increase the number of needles because there's sort of more complex reasoning that has to do about the order of different things in its context. Awesome. Yeah, cool. Happy to move on from there. Yeah, we have a whole bunch of other evals that we can go over. So I had in my notes that we could talk over anything that you want. There was also like Kali from Shunyu, who we have on the podcast for instruction following. And I realized that, you know, he joined OpenAI, and I wonder if he had a role to play in that one. No, we did not collab a ton on it. Honestly, I think it's best when eval authors and model developers don't collab too much because you want things, you know, as objective as possible, not trying to game any evals.

16:35Yeah. And then I think there was also, like, for the first time, the announcement of the, or shout out of the internal instruction following benchmark from API data. Yeah. People have had the ability to opt in to share data for a while. Actually, I posted a tweet because I found it in the dashboard that you can just opt in. And there's basically 16 days left for this program where you can just get free inference. So I'm just kind of curious, what you found from that kind of IF eval that might be different from the normal IF eval that people have? Yeah, totally. A lot of the instruction following evals that are open sourced are crafted in a way that are easy to craft.

17:17So, for example, GraphWalks is somewhat easy to craft. You can create this graph and verify it easily, but it is not exactly aligned with what the users are doing. And this is true for some of the instruction following evals where you ask the model to output exactly four words or three paragraphs or stuff like that. things that you can verify easily in code. And these are useful instructions, but we find that many of the really interesting instructions are actually challenging to grade. And so the open source evals often don't have them. And so getting this like real world diverse set of data actually helps us find like, what are the commonalities in what developers are doing?

17:58What is a really good example of like a negative instruction? And then we can go from there and figure out how to evaluate it. yeah i think that's there's also an interesting question of like what domains do people use you on and i wonder if like there's a way to tell you because sometimes it can be very confusing if i for especially because maybe i'm building an app and letting people use my key but other people are building apps on top of me so you have just a lot of chaos of like multiple degrees of abstraction where you just have to parse through the prompts. Yeah, it's true. Well, I will say we do use our own products internally where we can.

18:40And so we're not manually by hand reading every prompt. After they're like anonymized, we scrub them with any identifying data. Then we use our models to take passes to categorize them. And so if we get feedback that like we're not doing well on ordered instructions, then we can kind of do a pass over all of our data and find some good examples of those. so there's an instruction following section in this great prompting gvt 401 models i think maybe we can go through some of these examples the first one that caught my mind that it's not necessary to use all caps and other incentives like bribes or tips but developers can experiment with this for extra emphasis so i think that second part leaves me confused are you saying that people should still try and do this and sometimes the model responds positively to it do you feel like it's still just part of the lore i'm curious why i would have loved for you to say either yes it works or like no you should stop it looks silly i guess the truth is somewhere in the middle the truth is always messy reality is that our models have gotten a lot better at following instructions just stated once and clearly but we find honestly developers often become the best experts at prompting our models because you know you're building your livelihood on this thing and get to know the details of it really intimately.

19:59So I will say stuff like that won't hurt the performance of the model, but we kind of always want to leave it open to people to figure out what works best. Yeah. Yeah, and then you had to always start with a response rules or instructions section. Are those keywords meant to be taken kind of like verbatim? Like those are kind of like the tokens that work the best or is it just like an example? More of an example, yeah. Okay. cool yeah this is great i feel like until today we did an episode with like the prompt report on like all these prompting techniques but then it's also unclear for which model which ones work best so it's super useful and then you had a in the agentic workflows one you have a persistence thing it's like yeah please keep going how much and and i think i read that improves like the sweet the sweet bench like 20 just by having like the persistence i wouldn't it's not that this one prompt improves sweet bench 20 it's that we found this is the most effective harness for our model um and combined with all the post training improvements it results in the big improvement yeah like the model is trying a lot to be helpful and often it wants to check back in with the user and be like you know should i keep doing this like is am i on the right track and so prompt like this makes sure it keeps going doesn't bother you again and just gets the task done yeah yeah i think like there's this interesting trade-off between persistence and yielding back to the user the more agentic a model wants to be the the more persistent it should be but then sometimes it just goes off the rails and i wonder how you solve this trade-off because sometimes it just goes too far there's been criticisms of claude sonnet trying to rewrite too many files at once when I just wanted to make one thing, for example.

21:50And that's a form of bad persistence. What are the axes here in which you think about it? Yeah, I think one interesting thing that comes to mind here is that we had an extraneous edits eval where you ask the model to make an edit and classify like were all of its changes related to what it was asked to do or did it go off and do a little too much? And we found that from 4.0, which got 9%, It's pretty crazy. 9 % of the time making an experience at it is a lot. 4.1 % is at 2%, so it's a pretty big improvement. So yeah, I will just say focusing on this, we've heard feedback about this. We made an eval, and we made sure to track it and improve it during training too.

22:34Yeah, yeah. I mean, everything comes out of the evals as it's no surprise to anybody. That's true. There's another interesting eval that I think is causing some noise. For the first time, I think also that you being the master of structured outputs should know that JSON is bad now and we should all use XML. I wouldn't say that. I don't know which eval you're talking about. It's in the prompts guide, which maybe you guys didn't write. So we're kind of springing this on you. Yeah. Noah and Julian on our team wrote the prompt guide and did a great job. I do think XML is very helpful for structuring prompts, whereas for parsing outputs, maybe the story is a bit different.

23:12Like, sometimes it's really useful to get outputs in JSON, so you can plug them directly into your application. But I do think the models work particularly well with XML as inputs. But Chris, do you need to add? No. Cool. I mean, I think people always care a lot about code tool calls and structured outputs, as you well know. And so any updates to instructions over there is good. People also are interested in this concept of that apparently putting the instructions and user query at the top and the bottom, so duplicating it at the top and the bottom in the context, is better than putting it top only and much better than putting it bottom only.

23:54Again, this is from the prompt guide, so I don't know how aware you guys are on this. Yeah, I think part of that was just like, you know, empirical. We tried all three for when we were evaluating the model and having that redundancy is definitely the best. But then using the instructions at the beginning, the model is going to be able to then take that into account as it does processing. Yeah. I think a lot of people would see this as running counter to prompt caching because obviously you want to put the things that change a lot at the bottom. Basically, is this fixable in post-training? Can we just tell models to take instructions or user queries only at the bottom because we want to optimize for prompt caching?

24:37When we figure it out, we will do that. I mean, it seems doable. It seems like a post-training thing. I don't know. Maybe my mental model of post-training is wrong. So I think actually having things at the beginning of the prompt, you would still get prompt caching there. If you're putting in, for example, like a big needle on HaySack and you have the data changing each time, like per user, there's still different ways you can be putting the prompt at the beginning and getting a lot of the cache hits. It sort of just depends on your use case. yeah awesome the the other thing i noticed i know you made a note of this sean too is our chain of thought and reasoning and how people should think about this model versus their reasoning model yeah what's your yeah should i just use 4.1 and prompt it to do a chain of thought should i use a one and make a plan and then use 4.1 to implement the plan how should people think about composability yeah it's a great question we have found that 4.1 is a lot better at doing planning and thinking through its steps in COT when prompted than our previous non-reasoning models.

25:43But our reasoning models are designed to have kind of more coherent plans and be able to reason over longer horizons than these non-reasoning models. And you can see that reflected in things like intelligence benchmarks. So AME, GPQA, stuff like that, you'll see the reasoning models do much better. So in general, I would say, like the question you're really getting at is like, I'm a developer, which model should I be using? And I think the answer is always going to be the fastest model that accomplishes your task, right? So maybe you start prompting 4.1 as a starting point. If it does your task super well, then maybe you drop down to 4.1 mini and save latency or even nano.

26:27Whereas if 4.1 is struggling a bit little, maybe needs more coherent reasoning over longer time horizons, then maybe you upgrade to a reasoning model. Is there a quick way to get through these heuristics? I know one thing that a lot of people do is they use a one for a plan, and then they put that plan in cursor, and then have the plan apply to their code base. It sounds like there's maybe not a rule to when to do which, it's just task dependent. Yeah, I would say we're all kind of figuring out the best way to use these models together. And so I do think reasoning models for planning and using kind of more targeted models to execute is definitely a good architecture.

27:06Cool. If there's nothing else on that side, I'd love to go into the coding, which is something that we're emphasizing a lot. It's doing super well. It's better than 01 and Sweetbench. Was that expected? Not really. Yeah. Like what's the, I mean, what's the story there? There's also Sweet Lancer, which is a newer one, which attaches a money value to things. and basically what should people understand is going on here? Is it a better coding-based model or just a coding agent model? And I think there's also a question about how important to coding is it if I'm not using a coding use case? Yeah, so I'll start by saying we just set out to make a model that was great at coding, both in your terminal or in your editor or wherever you want to use it.

27:54And so we kind of broke that down into the problems. that it encompasses. So developers want the model to produce better diffs, for example, or they want the model to explore the code base correctly, or they want to produce code that compiles or produce code that writes tests. And so our approach was kind of teaching the model all of these various facets. There's kind of just a bunch of work streams that all coalesced around GBT 4.1. Yeah, I think much improved post-training all over to make for a better coding model. yeah i think there's like different kinds of coding right like it's it's interesting for me to observe that there for example so i'm just going to pull it up on the chart here because i always like to show people visuals you're 55 on sweet bench and 01 gets like a 41 but then on oh i don't think i i don't think i have the others like but but aider is it is less it is not at 01 level and so i think i i think i struggle to get some kind of intuition of when like like what are the different elements of coding i guess there is like you know single file edits whether it's like a diff or a whole file and then there is entire project edits is that a reasonable split are there more to this yeah that's one way to think about it basically where gbt 4.1 can it kind of explore go through a repo.

29:18It's been trained to do that particularly well. Whereas, you know, to just get some code and produce a change, a reasoning model might do better because it can kind of reason over the entire file. And so that's one good way to think about it. Yeah, yeah, that's fair. Any understanding of like the smaller ones, the smaller models, like basically for coding, I should only use 4.1 and forget the rest? you might like want to use the smaller models maybe if you have like if you have an ide where you need an autocomplete feature for example or if you want something super fast if you're building like i don't know a text to sql thing you might want the first version to populate instantly so you can see like 4.1 mini is actually quite significantly better than 4.0 mini but not that far away from the old 4.0 so i do think that model will find use case in a bunch of these coding niches.

30:17And I know you might not be able to talk about this, but the clip of an AI CFO talking about the agentics suite has been going viral, I think, today. It seems like every lab is putting a lot of emphasis into coding. So yeah, I'm just curious if there's anything you can share about how people should think about OpenAI encoding. Obviously, today you don't have clodest, clock code, you have anything related to coding. And I think the Windsor partnership today, they're giving 4.1 for free for a couple of weeks. It's maybe like one of the first OpenAI endorsement, I guess, on the live stream. But yeah, I know there might not be an answer that the PR team might approve, but I'm curious if you have any takes and thoughts.