DevDay 2025: Apps SDK, Agent Kit, MCP, Codex and why Prompting is More Important than Ever

0:04Hey, everyone. Welcome to the Layden Space podcast. This is Alessio from the Kernel Labs and I'm joined by SWix, editor of Layden Space. Hello, hello. And we are here in the OpenAI Dev Day studio with Sherwin and Christina from the OpenAI Platform team. Welcome. Thank you for having us. Yeah, it's always here. Yeah, it's so it's such a nice thing. We've been we've covered like three of these Dev Days now. And this is like the first time it's been like so well organized that we have our own little studio podcast studio in the Dev Day venue. And it's really nice to actually get a chance to sit down with you guys.

0:37So thanks for taking the time. Yeah, I feel like Dev Day is always a process. And we've only had three of them. And we try to improve it every time. And I know for a fact that I think we have this podcast studio this time. Because the podcast interviews and the interviews with folks like yourselves last time went really well. And so I want to lean into it a little bit more. I'm glad that we were able to have this studio for you all. We were kneeling on the ground interviewing Michelle last year. I don't know. I just saw it post-production. We had to have people cordon off the area so they wouldn't walk in front of the cameras.

1:07People just come up, hey, good to, I'm like, we're recording. I guess if you guys have been to three, what stood out from today? Or what's your favorite part? I feel like the vibes are just a lot more confident. You are obviously doing very well. You have the numbers to show it. Every year in Dev Day, you report the number of developers. This year it's 4 million. I think last year was like three. And I have more questions about that kind of stuff. But also just very interesting, very high-confidence launches. And then also, I think the community is clearly much more developed. I think there's just a lot more things to dive into across the API surface area of OpenAI than I think last year, in my mind.

1:56I don't know about you. Yeah, and we were at the OG Dev Day, which was the Dali hack night at OpenAI in 2022. And I think Sam spoke to like 30 people. So I think it's just crazy to see the... Yeah, honestly, I think it's kind of similar to this podcast studio, which is I think we've had a number of Dev Days now. We honestly were slowly figuring things out as a company over time as well, both from a product perspective and also from how we want to present ourselves with Dev Day. And at this point, we've had a lot of feedback from people. I actually think a lot of the attendees will get an email with a chance for feedback as well.

2:28And we actually do read those and we act on those. And one of the things that we did this year that I really liked were all of those, there was some art installations and the little arcade games that we did, which came up via engaging with the feedback. Yeah, the arcade games were so fun. I loved the theme of all the ASCII art throughout. This was my first SF Dev Day, but I've been to the Singapore one. That was actually my first week. Oh, yeah, that's the one I spoke to. Yeah, I saw you there. That was my first week of OpenAI. So really in the deep end. Put around a plane to Singapore. Yeah. Yeah, that's awesome.

3:00Well, so congrats on everything, and kudos to the organizing team. We should talk about some developer API stuff. Yeah. So we're going to cover a few of the things. You're not exactly working on Apps SDK, but I guess what should people just generically take away? What should developers take away from the Apps SDK launch? How do you internally view it? So the way that I think about it is I actually view OpenAI since the very beginning as a company that has really valued, kind of like opening up our technology and like bringing it out to the rest of the world. One thing we talk about a lot internally is, you know, our mission at OpenAI is to one, build AGI, which we're trying to do.

3:39And then, but two, you know, potentially, you know, just as important is to bring the benefits of that to the entire world. And one thing that we realized very early on is that we as a company, it's very difficult for us to just bring it to every, truly every corner of the world. And we really need to rely on developers, other third parties to be able to do this, which is, you know, Greg talked about the start of the API and like kind of how, you know, that was formulated. But that was part of, you know, that mentality, which is we need to rely on developers and we need to open up our technology to the rest of the world so that they can partake for us to really fulfill our mission.

4:12So the API obviously is a very natural, you know, way of doing that, where we just literally expose API endpoints or expose tools for people to build things. But now that we have, you know, ChatGPT with its, I don't know, like 800 million weekly active users, I forgot the stat that we shared. I think it's like now the fifth or like sixth largest website in the world. And the number one and number two most downloaded on the Apple App Store. Oh, yeah, with Sora. Yeah, but that one, like, it moves around all the time, so it's kind of hard to celebrate. Just screenshot it when it's good. Yeah, we definitely screenshot it and shared it when it was good.

4:47But kind of going back to my main point is like we've always kind of engaged the developers as a way for us to bring the benefits of AGI to the rest of the world. And so I view this as actually a natural extension of this. Candidly, we've actually been trying to do this a couple of times with last dev day with GPTs, two dev days ago with, sorry, two devs ago with GPTs and plugins, which was, I think, not tied to a dev day. So I view this as like, again, we love to deploy things so iteratively. And I view it as just a continuation of that process and also engaging deeply with developers and helping them benefit from some of the stuff that we have, which in this case is chat GPT distribution.

5:21Okay. And when, so Apps SDK is built on the MCP protocol. When did OpenAI become MCP-built? I'm sure internally you must have had, you know, design discussions before about doing your own protocol. When did you buy into it and how long ago was that? I think it was in March, I want to say. It's hard for me to remember kind of like the exact. March was the takeoff of it. Okay. Yeah. Yeah. So we built the Agents SDK and we launched that alongside the Responses API in early March. And I think as MCP was growing, that felt like a really, and, you know, we're building kind of a new agentic API that can call tools and just be much more powerful.

6:00MCP was kind of like the natural protocol that developers were already using to bring all the tools into their system. And I think, like, in March is when we added in MCP to Agents SDK first, and then soon after with kind of our other products. Yeah, I think there was, like, a tweet or something we did where it was, like, OpenAI, you know, is... Yeah, there was definitely a moment. I think there was a specific moment in a specific tweet. But what I will say, though, is like, and this is honestly like credit to the team at Anthropic that kind of created MCP, is I really do think they treat it as an open protocol.

6:27Like we work very closely with, I think, like David and the folks on the like, you know, consortium. And they are not, you know, really viewing it as this like thing that is specific to Anthropic. They really view it as this open protocol. There is like it is an open protocol. The way in which you make changes feels very open. And we actually have a member of our team, Nick Cooper, who is sitting on kind of like that steering committee for MCP as well. And so I think they are really treating it as something that is easy for us and other companies and everyone else to embrace, which I think they should because they do want it to be something that is very embraced by all.

7:00And so because of that, I think it makes it a little bit easier for us to embrace it. And honestly, it's a great protocol. It's very general. It's already solved. Why would you make it? Yeah, it's very general. There's obviously still more to do with it, but it was very easy for us to integrate because of how streamlined and how simple it was. Yeah. My final comment on apps SDK stuff, and then we'll move to AgentKit, is I always see abstractly when you wireframe a website or an AI app, it used to be that the initial AI integration on the website would be you have the normal website and then you have a little chatbot app.

7:37and now it's kind of like inverted where there's ChatGPT at the top layer and then there's like the website embedded inside of it. And it's kind of like that inversion that I honestly have been looking for for a little bit. And I think it's really well done. Like actually all like the integrations and like the custom UI components that come up, you had like Canva on the keynote there and it looks like Canva, but like you can chat with it in all the context of your ChatGPT. That is an experience I've never seen. Yeah. Yeah, and I think that's kind of back to the iterative learning that we've had.

8:09That, I think, was because we've learned a lot from plugins. So when we launched plugins, I remember one of the feedback that we got. I don't know if people here really remember plugins. It was like March 23. One of the points of feedback was like, oh, you can integrate. We told all these companies that you can integrate these plugins into ChatGPT, but they really didn't have that much control over how exactly it was used. It was really just like a tool that the model could call, and you were just really bound by ChatGPT. And so I think you can kind of see the evolution of our product with this.

8:36And this time, we realized how important it was for companies, for third-party developers to really own and steer the experience to make it feel like themselves, help them really preserve their own brand. And I actually don't think we would have gotten that learning had we not had all these other steps beforehand. Awesome. Christina, you were the star today on stage with the Agent Kit demo. You had eight minutes to build an agent. You had a minute to spare. And then you have some issues with the download done. Yeah, I wasn't sure. Honestly, I was like, let's do a little bit less testing. And maybe we, I don't know how much time I killed on the widget.

9:11I was extremely stressed when the download came. Yeah, I was stressed out. If a UI bug is what takes the demo down, I'd be so sad. I think it was a full screen, yeah, like focus thing. I heard the window wasn't in focus or something. Maybe you want to introduce AgentKit to the audience. Yeah, so we launched AgentKit today. Full set of solutions to build, deploy, and optimize agents. I think a lot of this comes from working with API customers and realizing how hard it actually is to build agents and then actually take them into production. Hard to get kind of that confidence and the iterative loop and writing prompts, optimizing them, writing evals, all takes a lot of expertise.

9:51And so kind of taking those learnings and packaging them into a set of tools, that makes it a lot easier and kind of intuitive to know what you need to do. And so there's a few different building blocks that can be used independently, but they're kind of stronger together because you then get the whole end-to-end system and releasing that today for people to try out and see what they build. Yeah. So I find it hard to hold all the building blocks in my head. But actually, chronologically, it's really interesting that you guys started out with the agent SDK first. And then you have agent builder, you have a connector registry, you have chat kit, and then you have the eval stuff.

10:29Am I missing any major components? Those are the main moving parts, right? Yeah, I think that's it. And then we also still have the RFT fine-tuning API, but we technically group it outside of the agent kit umbrella. Got it, got it, got it. Yeah, so it's weird how it develops, and it's now become the full agent platform, right? And I think one thing that I wasn't clear about when I was looking at the demo was, it's very funny because what you did on stage was build a live chat app for Dev Day's website. Yeah, did you get a chance to try it out? Yeah, I tried to try it out. It was an awesome night.

11:05And actually, I kind of wanted to ask how to deploy. Where's merch? Yeah, exactly. I was like, where'd you click the merch? Anyway, and this is very close to home because I've done it for my conferences. And it's a very similar process. But I think what was not obvious is how much is going to be done inside of Agent Builder. I see there's some actually very interesting nodes that you didn't get to talk about on stage, like user approval, that's like a whole thing. And, you know, like transform and set state. Like there's like a kind of like a Turing complete machine in here. Yeah. Yeah. So, I mean, I think, again, like this is the first time that we're showing Agent Builder.

11:40And so it's definitely the beginning of what we're building. And human approval is like one of those use cases that we want to go pretty deep on, I think. The node today that I showed is pretty simple, like binary approval. It's similar to what you'd see for MCP tools of approving that an action can take place. But I think what we've seen with much more complex workflows from our users is that it's actually quite advanced human-in-the-loop interaction. Sometimes these could be over the course of weeks. It's not just simple approval of a tool. There's actual decision-making involved in it. And I think as we work with those customers, we definitely want to continue to go deeper onto those use cases, too.

12:21Yeah. What's the entry point? So are developers also supposed to come here and then do the two-code export, like just segment the use cases? Yeah, so I think the two reasons that you would come to Agent Builder are one, kind of more as a playground, right? To kind of model and iterate on your systems and write your prompts and optimize them and test them out. And then you can export it and run it in your own systems using Agents SDK, using kind of other models as well. The second would be kind of to get all of the benefits of us deploying that for you, too. So you can kind of use maybe like natural language to describe what type of agent you want to build, model it out, bring in subject matter experts so that you really have this canvas for iterating on it and getting feedback, you know, building data sets and kind of getting feedback from those subject matter experts as well.

13:10And then being able to deploy it all without needing to handle that on your own. And that's a lot of the philosophy around how we're building it with ChatKit as well, right? You can kind of take pieces of it. You can have a more advanced integration where it's much more customized. But you also get a really natural path of going live with really kind of easy defaults as well. Do you see it as a two-way thing? So I build here. I go to code. Then maybe I make changes in code. And then I bring those changes back to the agent builder. I think eventually that's definitely what we want to do. So maybe you could start off in code.

13:44You could bring it in. will also probably have like ability to, you know, run code and the agent builder as well. And so I think just a lot of flexibility around. The one thing I'd say too, is a lot of the demos that we showed today, I think were like, you know, aired on the side of simplicity just so that the audience could kind of see it. But like, if you talk to a lot of these customers, like they're building like pretty complex, like you got to like zoom out on that canvas quite a bit to kind of like see the full flow. And that, and then for us, we, you know, we were kind of like working with a lot of customers who are doing this.

14:13And then, you know, if you turn that into like an actual agents SDK, like file, it's like pretty, it's pretty long. And so we saw a lot of like benefit from having the visual setup here, especially as the, as the setup grows, grows longer and longer. It would have been a little difficult to kind of showcase this, but even on like some of the, right. Yeah, you can do it in eight minutes, but like even with some of the presets that we have on the website. So one of the things, yeah, one of the things that we launched today as well, alongside just like the canvas is a set of templates that we've actually gathered from our engineers who are working in the field with customers directly of the kind of common patterns that they have in our own, basically like playbooks when we're working with customers on customer support, document discovery, and so kind of publishing those as well.

14:54Data enrichment, planning helper, customer service, structured data Q &A, document comparison, that's nice. Internal knowledge assistant. Yeah. And I think we just plan to add more to those as we can kind of build those out. I always wonder if there should be, so we're not the only agent builders, but obviously by default of being an open AI, you are a very significant one. Any interest in a protocol or interop between different open source implementations of this kind of pattern of agent builder? I think we've thought about it, especially around, I'd say, agents SDK. I would actually say maybe even zooming out a bit more from just this is like, yeah, we were also sitting here and kind of observing things being made over and over again.

15:36Even besides agent workflows, We're kind of watching what the industry is trying to do with responses, like what we've done with the responses API, like stateful APIs. And so, you know, obviously we were the first one to launch responses API, but like a couple of other people have kind of adopted. I think Grok has it in their API. I think I saw LMSys just did something recently in walls, but not, you know, not everyone. And so, unfortunately, I don't have a great answer today of like yes or no, but we are kind of like assessing everything and trying to see like, hey, you know, there has been a lot of value with MCP, MCP, hopefully with our commerce protocol as well.

16:15ACP, yeah, I definitely did not forget the name. And so even thinking about what we want to do with agents, with the agent workflow, the portability story around that, as well as the portability, I'd say even of responses API, it would be great if that could be a standard or something, and developers don't need to build three different stateful API integrations if they want to use different models. Yeah, and I think that's one of the... So it's not exactly a protocol, but one of the things that we launched today with evals too is ability to use third-party models as well and kind of bring that into one place.

16:47And so I think definitely kind of see where the ecosystem is at, which is using multi-models and kind of having... Third-party models as in non-open-air models? Yeah, it'll work with evals starting today. Okay, got it. We have a really cool setup with Open Router where we're working with them, and then you can bring your open router setup. And then with that, you can actually, you know, you write your evals using our data sets tool or use our data set tool to create a bunch of evals. And you'd actually be able to hit a bunch of different model providers, you know, take your pick from wherever, even like open source ones on together and see the results in our product.

17:25Yeah, that's awesome. Speaking more about evals, right? Like I think I saw somewhere in the release docs that you basically had to expand the evals product a little bit to allow for agent evals. Maybe you can talk about what you had to do there. Yeah. Yeah, I was going to say, so I actually think agent evals is still a work in progress. So I think we've made maybe 10 % of the progress that we need here. For example, I think we could still do a lot more around multimodal evals. But the main progress that we made this time was kind of allowing you to take traces. So the Agents SDK has this really nice traces feature where if you define things, you can have a really long trace.

18:09Allowing you to use that in the evals product and be able to grade it in some way, shape, or form over the entirety of what it's supposed to be doing. I think this is step one. I think it's good to be able to do this. But I think our roadmap from here on out is to really allow you to break down the different parts of the trace and allow you to eval and measure each of those and optimize each of those as well. A lot of times this will involve human in the loop as well, which is why we have the human in the loop component here too. But if you kind of look at our evals product over the last year, it's been very simple.

18:41It's been much more geared towards this simple prompt completion setup. But obviously, as we see people doing these longer agentic traces, like how do you even evaluate a 20-minute task correctly? And it's a really hard problem. We're trying to set up our evals product and move in that way to help you not only evaluate the overall trajectory, but also individual parts of it. Yeah. I mean, the magic keyword is rubrics, right? Everyone wants LMS judge rubrics. Yeah. Yeah. Obviously, where does this look go? Okay, great. The other thing I think online, I see the developer community are very excited about is sort of automated prompts optimization, which is kind of evals in the loop with prompts.

19:20What's the thinking there? Where's things going? Yeah, so we have automated prompt optimization, but again, I think this is an area that we definitely want to invest more in. We, I think, did a pretty big launch of this when we launched GPT-5, actually, because we saw that it was pretty difficult as new models come out to kind of learn all the quirks about a new model. Yeah, the prompts are the major. Right. There's like we have a big prompting guide, right, for every model that we launch. And I think building out a system to make that a lot easier, we definitely want to tie that in like completely with evals.

19:50We should be able to kind of improve your prompts over time, improve your agents over time as well if they're kind of made in the agent builder based on the evals that you've set up. And so I think we see this as like a pretty core part of the platform of basically suggested improvements to the things that you're building. I actually think it's a really cool time right now in prompt optimization. I'm sure you guys are seeing this too. It's like not only are there a lot of products kind of like gearing around this, so like kind of what we're thinking about. But I also think like there's a lot of interesting research around this, like GEPA with like the Databricks folks are actually doing really cool stuff around this.

20:20We're obviously not doing any of the cool GEPA optimization right now in our product, but we'd love to do that soon. And also, it's just an active research area. So whatever Matei and the Databricks folks might think about next, what we might think about internally as well, whatever new prompt optimization techniques come out, I think we'd love to be able to have that in our product as well. And it's interesting because it's coming at a time when people are realizing that prompt. I feel like two years ago, people were like, oh, at some point, prompting is going to be dead. No. And it's like, you know.

20:52It's gone up. Yeah. Yeah. Yeah, and if anything, it has become more and more entrenched. And I think that there's this interesting trend where it's becoming more and more important, and then there's also interesting, cool work being done to further entrench prompt optimization. And so that's why I just think it's a very fascinating area to follow right now, and also is an area where I think a lot of us were wrong two years ago, because if anything, it's only gotten more important. Yeah, I would say what... Shin, you used to work at OpenAI, now it's an MSL. So we call this kind of like zero gradient fine-tuning or zero gradient updating because you're just tweaking the prompts.

21:28But it is so much prompt that you end up with a different model at the end of it. There's a lot of things that make it more practical, too, just even from our perspective. We have a fine-tuning API, and it is extremely difficult for us to run and serve all of these different snapshots. Like, you know, Laura's great MSL just, you know, or sorry, Thinking Labs just published, John Schumann just had a cool blog post about this. But, like, man, it is, like, pretty difficult for us to, like, manage all of these different snapshots. And so if there is a way to, like, hill climb and, yeah, do this, like, zero gradient, like, optimization via prompts, like, yeah, I'm all for it.

22:04And I think developers should be all for it because you get all these gains without having to do any of the, you know, fancy fine-tuning work. Since you are part of the API team, you know, you lead the API team, and since you mentioned Tinky, I got to throw a cheeky one in there. What do you think about the Tinker API? So, yeah, it's a good one. So it's actually funny. When it launched, I actually DMed John Schulman. And I was like, wow, we finally launched it. So the... Because you used to work with him. Yeah. Yeah. So we... It's actually funny.

22:35So right when I joined OpenAI, this has actually been, I think, a passion project of John's. He's been talking about doing something in this shape for a while, which is a truly low-level research fine-tuning library. And so we actually talked about it quite a bit when he was at OpenAI as well. It's actually funny. I talked to one of my friends who said that when he was at Anthropic, he also worked on the idea for a bit. He's a man on a mission. Yeah, I mean, John's so great in this regard. He's so purely just interested in the impact of this because, one, it's a really cool problem, and then, two, it also empowers builders and researchers.

23:13You saw all the researchers who express all this love for Tinker because it is a really great product. And so I'm just really happy to see that they shipped it, and I think he was really happy to kind of get it out there in the world as well. Yeah, this is very much a digression, but it's weird as someone passionate about API design that it took this long to find a good fine-tuning API abstraction, which is effectively all he wanted. He was like, guys, I don't want to worry about all the infra. I'm a researcher. I just want these four functions. And it's kind of interesting. Yeah, yeah. Cool. Before the OpenAI Coms team barges in the room.