1. Podcasts
  2. Latent Space: The AI Engineer Podcast
  3. Why Dwarkesh is Wrong about Computer Use + How OpenAI shipped its Jev competitor in 1 Week
Latent Space: The AI Engineer Podcast

30 Sep 2026 · 39m · 17 chapters

Episode artwork

Why Dwarkesh is Wrong about Computer Use + How OpenAI shipped its Jev competitor in 1 Week

In short

The episode is a Dev Day discussion about OpenAI’s computer use agents and new platform APIs, plus a debate about whether “computer use hasn’t advanced in two years.” Ari (product/engineering lead for computer use agents) says the biggest progress is reliability: agents now start tasks and, crucially, debug and retry. He highlights GPT 6.1 Sol for computer use cost/speed (claimed 1/5 of Astra cost; 1/7 for computer-use-specific). He also explains “Dots” personal assistants now get a persistent Linux VM in the cloud, enabling full desktop apps. Examples: app shots (accessibility/DOM-rich context vs screenshots), 15-minute meal-prep ordering vs 2 hours manually, and YouTube automation via VM/AB testing. Bottlenecks shift to harness/inference speed and web/app load delays.

Guests

Ari (computer use agents lead) and Nikunj (API product lead; GPT-6 API features like async tool calling, mid-turn steering, WebSockets; also Decisions API/Jev competitor). Nikunj frames Decisions API as fast, parallel, confidence-oriented classification built on Luna weights, enabling snappier tool use with GPT Live and other workflows.

Written by AI. May contain mistakes.

Chapters

0:45

Highlights from OpenAI Dev Day Keynote

Discussion of key announcements and features from OpenAI Dev Day, including new products and capabilities.

“because of sort of the cost and speed advantages.”
3:06

Exploring the Decisions API and Dots

In-depth exploration of the Decisions API and Dots, highlighting their functionalities and user applications.

“but now you get your own entire Linux computer in the cloud.”
7:24

The Evolution of Computer Use

Discussion on the advancements in computer use technology over the years and the significance of recent improvements.

“which is a great speed up and great capability.”
11:28

Future Predictions for Computer Use Agents

Speculation on the future of computer use agents and their potential impact on user experience and productivity.

“It's when you double hit command, it pulls in what looks like a screenshot.”
14:00

The Complexity of Event-Driven Actions

Learn about the complexities and timing of event-driven actions in programming.

“A lot of the time is actually waiting for doordash.com itself to load.”
15:50

Leveraging Bots for Customer Support

Discover how bots can enhance customer support interactions and improve response times.

“obviously you also have to dog food your own products and all these things.”
17:50

Evolving Trust in AI and Computer Use

Explore the evolving trust in AI technologies and the importance of safety checks.

“Because traditionally, you know, you build something in codex, and then the codex builds it for you, and then you have to test it, and you are now like QA for the agent, right?”
19:10

API Changes and New Features

Learn about recent changes to the API and the launch of async capabilities.

“This is not the last that we're going to talk.”
21:50

UltraFast: Speeding Up AI Processes

Understand the implications of UltraFast technology on AI performance and speed.

“I think before I go into the API, the most fun part of UltraFast has been just watching the inference team cook with Astra.”
24:00

The Decisions API and Its Impact

Delve into the Decisions API and its significance in fast classification and decision-making.

“But also our internal teams are like, we need a much faster classification system.”
Show all 17 chapters
28:03

OpenAI's Iterative Deployment Strategy

Learn about OpenAI's approach to deploying new models and iterating based on user feedback.

“All sorts of like inference techniques people are working on to try to make it as fast as possible.”
28:31

Innovations in AI and RLHF

Explore the latest innovations in AI, including RLHF and its implications for model confidence and calibration.

“and then we'll make more model improvements as needed.”
29:18

Speculation on AI Architecture

Discuss the potential future architectures of AI models and the implications for speed and accuracy.

“And then architecture-wise, the other thing that's in the debate, obviously nobody knows because Jeff doesn't talk about it, but the two speculations are, one, maybe the fusion model instead of autoregressive.”
30:18

Use Cases for Decisions API and Agents

Examine the use cases and internal applications of the Decisions API and Agents API within OpenAI.

“and people are going to do so much cool stuff and everyone's going to learn from each other.”
32:04

Enhancements in Caching Strategies

Learn about the new caching strategies and performance enhancements in OpenAI's APIs.

“The whole edit spaces, pages, collaborate, add in your dot.”
34:10

Compression Techniques in AI

Discover the different compression techniques used in AI and their importance for performance.

“For builders, they should implement because it's significantly cheaper.”
36:47

Future Directions for AI APIs

Discuss the future of AI APIs and the challenges of building higher-level primitives on core capabilities.

“So yeah, lots of cool stuff going on around InCompaction as well.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Latent Space: The AI Engineer Podcast, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:04Okay, we're very excited to be here. Today is OpenAI Dev Day. We have Ari here who leads the product and engineering team for computer use agents. Before we kick in and dive deep on computer use, you want to give a quick recap? What was announced? What's the quick slew of announcements you guys had today? Yeah. Yeah, it was a super exciting day. We just got out of the keynote. It was really sick. There were a bunch of computer use announcements that I think are worth thinking about. We have Dots, which is the new sort of personal assistant product. and that has some really exciting computer use features.

0:40There's GPT 6.1 Sol, which is this amazing new model that I think is particularly great for computer use because of sort of the cost and speed advantages. I think we shared that it's a fifth of the cost as Astra and a seventh of the cost if you're looking at computer use specifically, which is really amazing. Sorry, there were so many things I'm trying to sort through it. And the API. Agents API, which now has computer use in it, which is really cool because now developers can build on the same computer use that is part of Codex and ChatGPT. And then there were some demos of our existing computer use features like app shots, where you can take the context of something you're doing on your computer and bring it into Codex and ChatGPT really fast.

1:23And then like native computer use on your Mac, where Roman had it taking screenshots of his app automatically, and he could do other things on his computer while computer use was using his applications. So yeah, really exciting keynote. And not to mention the Decisions API. Decisions API. Off the bat, are they all the same model? Like this is the same data set distilled to different models? Like basically, is computer use using Decisions API or are they like kind of separate? So what's really cool about the Decisions API is, you know, it has all these new capabilities. It does inference in parallel.

1:57It doesn't have reasoning. It's a smaller model than the ones we use for computer use. and so those capabilities make it really fast. They also make it a little bit less good at doing long horizon, sort of sophisticated tasks and so I would say it's still an open area of research for how we bring those approaches together but yeah, I'm really excited to see what people build with the decisions API. One of the interesting things is DOTs now have attached personal computers so it seems like they're very much more persistent. You've been using them for a while. How should people push the bounds? What should people aim for?

2:37What should they try? Personally, right now, I use it for a lot of customer service. Like, oh, this was wrong. I don't want to sign in. I don't want to authenticate. Find whatever and just get it fixed. How should we push further? What should people try? Dots are a really cool product because each dot has access to its own Linux virtual computer in the cloud, which is different from our other products. Traditionally, we have access to a browser in the cloud or it has access to your own computer, but now you get your own entire Linux computer in the cloud. And so we can run full desktop applications.

3:11And it can also use a web browser. And so, yeah, I think the powerful thing about computer use and the reason why I think it's so exciting is because it makes it so that the agent can do anything you as a person can do because all the software in the world was designed for humans and now agents can use that same software and you can delegate to the agent. So yeah, like anything that you would do on a computer, you can ask a dot to do. Yeah, I think what particularly useful is going to really depend on who the end user is and what's valuable in their life. But yeah, I would just start by thinking about like one of the things that you spend time on and how could you delegate those to an agent.

3:46Yeah, a lot of flight booking and shopping and honestly, even like playing a game or whatever. Totally. Yeah, I don't know. for me, something I did recently, I've been working on, I subscribed to a meal prep service because I was trying to like eat healthy, you know, and I really liked this meal prep service I found because it lets me customize the meals I order to like a high degree of granularity. So I can say like, I want this many grams of chicken and this many grams of rice. But it was so complicated. It took me two hours to do an order. And I found that I could ask computer use to do it for me and it did it in 15 minutes.

4:23So I actually saved two hours. Both did it eight times faster than I could and it saved me two hours on GPT 6.1 Sol. So those are the kinds of tasks that I feel like are really powerful. As a creator, I can tell you automatically, immediately, my number one use case is automating YouTube because YouTube doesn't expose a lot of things via API. And you have to just put it in a VM and just run it for, let's say, their A-B testing feature or making community posts, none of this is available by API because they hate developers. Anyway, so... I've heard that from our developer experience. They use it with YouTube a lot.

4:58Yeah, it's really awesome. So I want to draw, you know, let's say I want to get a little bit spicy. One of the leading AI podcasts, our friends, is famous for saying that computer use hasn't advanced in the last two years. Yeah. Which is a very interesting statement. And I think you're one of the best people in the world to talk about this, like how things have progressed. Yeah, you know, they said that a few months ago, I think, and I hope they have a different perspective now because computer use is like 180 degrees different than it was. He's a tough guy, it's impressed. Yeah, okay. We'll work on it.

5:30You worked on, you've like basically spent your whole career working on like some kind of computer automation, right? Yeah. Like shortcuts at Apple and then Sky and then joining OpenAI. Can you draw like what your through line is for like what is driving you and what wasn't possible back then maybe, and what your sort of milestones were. I guess to add on to that as a follow-up question, what's the major change from using codex computer use from last week to today? Is it model? Is it dots? Is it harness? So all the history plus, what really just changed in today's announcements? Yeah, on the through line, I guess I've always been excited about automation and helping people automate tasks because then you can save time in your life and focus on things that are more important to you than operating a computer very intricately.

6:17And so, yeah, that was why we worked on some of those products. I was at Apple before. We made a company called Sky. We ended up joining OpenAI, which was really exciting. And I think something that was special. And almost like you have to hack around Apple until Apple was like, fine, we'll just hire you and you can just work on the inside, right? It was a cool place to get to work. What was really interesting looking back at Sky is we were working on computer use there as well. and the models were so much less capable. And now the models just in the last one year have become extraordinarily capable of computer use.

6:49I think the biggest delta that I see is before they could like reliably start tasks, but then they would run into problems and now they're really good at debugging. They're really good at trying again, introspecting what is and isn't working. And I think we've also brought the computer use, the computer use field itself has moved forward. I think we're using more techniques. Now computer use often writes code. So if you actually look at it in Codex and you expand the tool calls manually, you can see that it's not just doing one action at a time. It's actually writing JavaScript code that the computer executes to perform sometimes many actions at once, which is a great speed up and great capability.

7:28We use more accessibility sort of multimodal interfaces. So the model may use screenshots. It may use accessibility. It may use Playwright. It can use a lot of different mechanisms based on the task at hand. And then, yeah, the model acceleration has been just amazing. So, yeah, what's different today? I think we're making computers better all the time. So I think just like one day's difference is probably a little bit less consequential than like even the past month or the past two months. But yeah, I think the computer use in DOT is really exciting as well as the new model that we came out with.

8:02On the keynote, Tejal was mentioning 7x improvements in computer use speed, a lot better on a few benchmarks. How do you guys think about measuring it? Computer use is one of those things where, as you say, it's improvements over time. Is it harness? Is it model? Is it post-training? How do you guys look at it internally about measuring how good it is? And what were the changes with the new model? We actually have a bunch of different ways of measuring it, some of which are on different permutations and configurations of the harness. It's a bit of a complicated story because our production products have some more safety checks, and those are configured differently based on the needs of the task at hand.

8:47So there's a lot of ways to measure it. But I think regardless of how we measure it, we find pretty consistent gains. And those gains are sometimes in the harness and sometimes in the model. And yeah, I was really excited by this result that GPT 6.1 is even more cost effective for computer use than its baseline cost improvement as compared to Astra. It's like really cool to see. Yeah. I mean, one of the visuals I really like from the live stream was that you're sort of improving the Pareto frontier of your curve. And there was a lot of talking about how you're improving it together with the harness.

9:23Can you give some examples of aha moments that you had, whether it's on like model driving the harness, harness driving the model, whatever? I don't mean to repeat myself, but I think like introducing more modalities has been really powerful. One more specific example of that is in the past, I think we saw a lot of computer use products had to spend a lot of time like scrolling, you know. So it would like take a screenshot. It would try to do something. It would be like, oh, I got to like scroll down to the next page of results. and then it would take a screenshot and then would try to do something and scroll down again.

9:54And so I think with accessibility and direct access to the DOM and other things like that, now the language model can actually see an entire page or an entire application. It can write code that can do multiple steps at once. And so I think those have been probably the biggest single aha moments. There's a lot of tiny ones that are less exciting in comparison, but actually we do find also that a lot of speed improvements are driven by a lot of little paper cuts that we've got to go in and interact. A lot of really hard engineering. I mean, app shots in general, I think people don't quite get the difference because there's a nice visual in Codex when you take an app shot, but they don't maybe get the difference that you are able to actually drive each button and you have each text in a very optimal representation.

10:40Yeah, yeah, exactly. Yeah, it's kind of fun, actually. If you want to be really nerdy about it, you can go into Codex, take an app shot by hitting the two command keys. So you grab the content from whatever app you're working with, bring it into the Codex or ChatTBG chat. And then if you click on the attachment and you click on this little tiny button in the top right, you can see the raw text and you see the raw accessibility representation. And yeah, we've put a lot of work into... Just dumping everything out, yeah. Dumping it out, but also making it token efficient, doing it efficiently. There's a bit of an art to it.

11:11And it turns out that the same technology that was invented for humans who maybe have accessibility needs, who want to use a screen reader technology. That technology is really helpful for them to be able to use computers. It's also really helpful for LLMs to be able to use computers. So that's been really fun to get to work on. For context, I feel like a lot of people don't understand AppShots. They don't even know it's a feature. It's when you double hit command, it pulls in what looks like a screenshot. And you're like, oh, why have I opened up just a screenshot and thrown in? No, it's actually pulling all the metadata, all the code, everything.

11:40Yeah, exactly. So it's like, if you take a screenshot of a web page that has a link, the screenshot doesn't include where the link goes. It doesn't include, you know, maybe you take a screenshot of your calendar, the event titles are truncated, you know. But when you take an app shot, it gives like the language model like full context about everything. And that lets it just sort of like do much more. Yeah. For those who want to see more, Jason Liu, I invited him to do a full workshop on this at AI Engineer. Amazing. You did a great job. I have a broader vision question on computer use agents. So your example of take a screenshot, scroll page, take a screenshot where we were.

12:17Today they can automate a lot. What are the bottlenecks? Is it models? Is it harnesses? Where do you see it going in like two years? Do you see it just running for hours? How do we get there? Any predictions on where computer use goes? Yeah, I mean, I think what's really crazy that I think the team's accomplished over the past couple of months is that now computer use is faster at accomplishing tasks than the average human, probably in most cases. And I think that the next frontier is to have computer use be literally superhuman in its performance, where it actually is as fast or faster at using software than expert computer users like us.

12:59And I think that'll be really consequential and exciting when that happens, because I think we'll be able to all of a sudden build products that provide just much more real-time experiences. And I think it'll also, lowering the barrier to entry of, or the activation energy, I suppose, of using computer use, I think will make us start to default to doing certain things in agents that we've become accustomed to doing manually. And I think that's exciting also because it'll save us a ton of time. And I think there's, you know, there are a lot of different little paper cuts and bottlenecks that are sort of standing in the way of that.

13:36I think that there's things on the model side, there's things on the inference side, there's things on the harness side, there's things in the representation. We find that as computer use gets faster, we're increasingly bottlenecked by the speed of doing an operation. For example, a non-trivial amount of time in our benchmarks of computer use tasks is actually like, let's say you're automating a task on doordash.com. A lot of the time is actually waiting for doordash.com itself to load. you know? Yeah, then you just write a wait and then you execute the wait. Yeah, totally. And you want to get, actually, it's actually really important that you get that you want as little delay as possible between when it finally finishes loading and when you go and trigger the element, do the next action which is actually itself a statistical science.

14:20Like an event-driven way maybe to do that. When possible you want it to be event-driven and JavaScript has load events for, the web browser has load events for web navigation, but there's other types of events that actually really can't be event-driven, so there's a lot of complexity The one that comes to mind is like chatting with customer service. Yeah. Applies could take 30 seconds, could take three minutes. Oh, right. I have dealt with so many bots with Codex. It's great. But I also wonder if the other side knows that they're talking to a bot because I'm answering in complete sentences. Like I'm capitalizing correctly.

14:51I'm giving full reference numbers and everything. It's clearly too good. I don't care. I'm just trying to get my support case. I've prompted it to like, you know, don't pretend you're a bot. Be very annoyed human. Short one-liners. Push it. Do all this. I'd also tell it, while you're waiting for responses, use subagents to research better ways to figure out what we need. Nice. It's just human little intervention. That's awesome. I also feel like half the time it's a bot on the other end, so now you've got the bots talking to each other. I will also say, one milestone of computer use that we're at now is three, four years ago, we were scared of hooking up LLMs to the web.

15:31to our devices. And now I'm having it configure DNS for me. I'm having it pay my bills and really tens of thousands of dollars of stuff I'm just sending it over and YOLOing with computer use. And what's the worst thing that happened? So that's all really good. I think now that you've, obviously you also have to dog food your own products and all these things. Now that you've sort of released this in API, what are some pitfalls or tips that you want to tell developers because they're about to, I guess, encounter all this firsthand? First of all, I'm just really excited that we brought computer use into the Agents API.

16:08I think this is really great because obviously a lot of developers are building applications that want to be able to work with third-party websites and services. And so computer use has this universality to it. It can work with anything. So now all of a sudden, developers can build using the same computer use implementation that we're building on. I think there's great work to be done if you want to build your own computer use harness, but it's hard. and also we train our models on our computer use harness so there is an advantage to using the one that's in distribution for the model there actually might be a speed and cost and and accuracy advantage so i think it's really great for people to get to build on top of that and um yeah you know i think kind of to to the point that you were making like i think we're all sort of still in the process and maybe like some of us are ahead of many people in the world of like getting comfortable with this technology and trusting it.

16:58And so I think it's incumbent on us to sort of build that trust over time by making sure we're building things that are reliable, by building the right kinds of safety checks, by asking for the user's consent before doing something consequential, like making a payment, by asking, you know, maybe depending on the application, making sure you're only letting it access the websites or applications that it actually needs for the task. So that's, I think, something important to think about. But yeah, I'd really encourage people to try the new Agents API, build all kinds of cool stuff on it. We'd love to hear your feedback, depending on how it goes.

17:31Have you seen any changes in the way it affects dev workflows? One of the things with Dots is you're seeing it in Slack, you're seeing people use voice and build. The example Roman showed of change this app and send me screenshots along the way and all this. Is there anything that you're seeing there in adoption about how people are using computer use for coding workflows? Any tips people should take from that? One of my favorite use cases for computer use, actually, and one that we see a lot in the wild, is computer use letting the agent actually test the software that the agent has built, which is far more consequential than it sounds.

18:09Because traditionally, you know, you build something in codex, and then the codex builds it for you, and then you have to test it, and you are now like QA for the agent, right? So with computer use, you can complete the software development lifecycle where the agent can build software, it can test it. So I have a lot of fun building stuff, having the agent test it. By the time it comes to me, it's already working. I have extra fun because sometimes I'm developing computer use itself. And so now I have a computer use agent that's using my computer use agent that's using something else. So yeah, I really think this is a super powerful class of use case.

18:43I have a visual playtest skill that I've developed that really catches a lot of design issues. Nice. Normally when you just look at code, you wouldn't really pick it up. It's also really good for cloning apps. So if you're using a shitty SaaS and you want to kill the SaaS, you just clone it screen by screen by screen. And obviously computer use can completely drive everything, take screenshots, note it down, and then clone everything with Codex. That's really cool. But yeah, thanks for all your progress. I think that is our time. This is not the last that we're going to talk. Yeah, cool. This has been really fun.

19:13Thank you guys for having me. All right. All right. Okay, we're strict cut off. We're just going to dive right in. Let's do it. Yeah. Okay, so Nikunj, we're very excited to have you. You've shipped a lot on the API side, like we just talked about with Ari. You can now build with computer use agents. Anything you want to highlight the API side of changes and introduce yourself a little on what you do? Yeah, for sure. My name is Nikunj. I lead product for the API team. I've been here for roughly three years. I've been working on launching models. I feel like that's just been a constant thing throughout my time here at OpenAI.

19:48And with every new model, we try to basically work super closely with the post-training team, the research team, to figure out what's new in it. And then we expose those capabilities in the API. So that's the basic way of putting it. And if you just look at everything that's new with GPT-6, the cool new capabilities that we launched were, firstly, async function calling. So what you see with like, like a lot of the things that you're seeing in like Codex and Dots and everything is that tool calls take so long that you don't have to like pause the model's execution while the tool is running. So you could just like kick off a tool call, keep running, keep reasoning, and then check back in.

20:29So we launched async tool calling. We launched like mid-turn steering. So now you can like inject messages while the model is reasoning in the middle. So as your tool call finishes, you can put in that instructions. And that's also partially a model alignment capability, right? Like they have to train in the ability to train. Exactly, yeah. I feel like we've had it in the app. You could always, as it's reasoning, you could steer. Yes. It wasn't the best. It's gotten much better. Yeah. I'm excited to see how it does this. Yeah, and like our main goal in the API is to like put things in the API once it's trained into the harness.

21:05And so we kind of wait for that moment until it's good enough. And a lot of that is like actually being powered by WebSockets, which we launched a few, I want to say months ago. And so WebSockets just opens this whole bi-directional communication thing with the model. This is not the GP Live thing, I'm just talking about GPT-6. And you can do all these async tool calling, async reasoning, injecting messages. So it's a really fun API to work on. I think I'm really enjoying it. This is why we are the engineering podcast, because we get to talk about WebSockets. This also pairs very well with UltraFast.

21:38that is now, I think, for the first time ever available in the API. Yes. Which is basically the theoretical fastest speed you can ever get from Tearot Intelligence. Yeah, it's been so exciting to work on that project. I think before I go into the API, the most fun part of UltraFast has been just watching the inference team cook with Astra. They're just constantly having these codex agents running, trying to squeeze out more performance. And I would say like at least for a couple of months, a lot of it was focused on efficiency and driving the cost down, which is how we like were able to cut the lunar price by like 80 percent.

22:16It was like a lot of that was driven by like all the inference improvements they landed. And then now they've like shifted gears towards like how can we make this run as fast as possible? And so ultra fast has just been like amazing to see on a model like Astra, like to go that fast has been really cool. And yeah, WebSockets is like, actually it was like, the first time we launched WebSockets, it was for GBD 5.3 Codex Spark, which was, can't believe we named a model that, but you know, that's what we launched it for. And obviously it helps so much because like, you got to have the tool calls, you got to like really reduce the overhead of going back and forth with tools.

22:54And so WebSockets is awesome for that. Yeah. It's always cute to see, like, I have my reset usage limit and then I have my Spark usage limit. that I never use. But it's there if I want it. I think it's gone. Finally, it's gone. I know it's gone. You're slowly killing off all the old things. It was a great week. I mean, it was the first time we had Frontier Intelligence at extreme speeds. People really liked it. Yeah, yeah. So, first of all, it comes back. 5.3 Spark is explicitly attributed to Cerebris. You guys are not confirming or denying that UltraFast is related to Cerebris, but people are...

23:30I'll just say that people do care and are wondering about it. And you have your own Silicon as well. Elephant in the Room, Decision Models, Decision API. We were the first podcast to do a big Jeff deep dive with Diogo, and I also featured him at AI Engineer. How quickly did you see Jeff and go like... Yeah, first, like, huge props to Diogo and, like, the Jeff team for, like, really inspiring the whole segment in the market. Like, obviously, Jeff comes out, everyone's, like, losing their minds over it. our users are hitting us up. But also our internal teams are like, we need a much faster classification system.

24:09I don't want to get ahead of some of the dots features that are going to come, but you're going to see some cool, really snappy, fast things built on top of the Decisions API. But yeah, props to Jeff for inspiring this whole thing. Obviously, a bunch of people at OpenAI get nerd sniped by that. and they're like, how can we make this work? We're not going to train a new model. Four weeks ago, this was not on Devin. No, not at all. This is Jeb-inspired. I think you're officially the first Frontier Lab to clone and adopt this. I feel like OpenAI is such a strong hacker culture and people get excited about things.

24:51And so a guy from Inference, this one awesome guy from the Infra team are like, this is amazing. We're going to hack on it. They built a prototype, it works, and now we're just hill climbing on latency and trying to make this as fast as possible. And we want to launch it in the coming days. So as soon as we hit our latency target, we'll try to get this out. It's interesting. At the same time of hacker culture, you also, as Sam said, 99%, one of the most reliable APIs with, I think, probably the most usage, which is your team directly. How should people see Decisions API? Yeah, I feel like a lot of people saw Jev, heard the buzz, haven't built with it.

25:30You're making it very mainstream. What should people see it as? How should they use it? Yeah, I think the main use cases we've seen is like really fast classification. All the computer use demos have been amazing and really cool. I think there will be limitations, of course, in terms of, you know, having Astra, like write like a JavaScript, like script to control your computer versus having Luna pick like one action at a time. I think it's not going to be at the same intelligence level, but maybe there's some computer use tasks that this is good enough for. So excited to see that come through.

26:04The other cool prototype I've seen internally is people hooking it up with GPD Live. So GPD Live is our bidirectional real-time API. And it's built on this model of front-end models and back-end models. So GPD Live is this super fast thinker-talker thing. So GPD Live is the talker. super fast, really good at delegation, and you have something like Astra sitting at the back. But tool calling has always felt really slow in GPD Live. And so people have been putting together these tool calling demos of GPD Live controlling a computer. It just feels so much more snappy and natural. So I'm kind of excited to see what people do with Live and with Luna on Decisions API.

26:53So that'll be pretty exciting. So I want to iron this out for people, especially from the product side, because a lot of people have been putting out JEV clones. There's been about 100 in the last two weeks. Oh, really? That's amazing. Okay, in the first three days. But like, they can clone a JEV API, which is honestly structured outputs, which opening out was first too. Yeah. Right? So like, I think let's iron out for people, what is a decision model as far as like, what is important? It is not just latency, it's not just structured output, right? Because I could just have Luna, the decision model is priced the same as Luna, have turned off reasoning and then have structured output.

27:30Do I have a JEV? No, right? And that's the real... There's a confidence there. Yeah, totally. I think the way that... So we haven't trained a new model for this. We're building this purely on top of the same Luna weights that we have. So yeah, this is really just Luna. and on top of that, what you're doing is you're constraining, so like structured outputs is a big part of it. You're really optimizing the inference stack to like get very fast on TTFT and because you can have multiple questions, what you do is like you basically run those in parallel. As a batch. Yeah, you run those as a batch.

28:09All sorts of like inference techniques people are working on to try to make it as fast as possible. But I'd say like at least our implementation of it at the start and this first version is like zero-shotting this on top of Luna to see how it goes. And obviously you want to put it out there. This is OpenAI's classic iterative deployment thing. Put it out there, see what people think, and then we'll make more model improvements as needed. So yeah, that's the decisions API. Yeah, and obviously as a benefit, you have vision, they don't have vision, right? That's true. Yeah, yeah, yeah. We get it for free with Luna.

28:45Yeah, I do think that some of the innovations, it sounds like it's still to come if it's still the same lunar waves, which is the confidence stuff and calibration is something that we've talked about on the podcast with benchmarking calibration. Because basically, the whole point is that RLHF kind of collapses you towards what you want to hear, but not actually what the amount of confidence is. Yeah, totally. I'm eager to see how it pans out. Maybe these are going to be the key areas where we may have to hill climb. with a future model release. But yeah. And then architecture-wise, the other thing that's in the debate, obviously nobody knows because Jeff doesn't talk about it, but the two speculations are, one, maybe the fusion model instead of autoregressive.

29:29But you are able to achieve the parallel generation in your way. And then the other one is some mech-interp type thing that you're analyzing the activations and then just outputting the weights. Which you guys have all done the research on this. people have speculated. There have been demos on both of these as well. I think Gemini shared a Gemini diffusion, Gemma diffusion on a JEV style output. And Inter people have also, you know, pulled out Inter from a middle layer, but this is all speculation. It's just like, what are you trying to aim for, right? Because you can achieve the API. Everyone can achieve the API.

30:04It's actually pretty trivial. But like, then there's the speed, then there's the accuracy, then there's the other calibration features. I don't know what else. Yeah, no, totally. It's so cool that this whole space has been kicked off now and people are going to do so much cool stuff and everyone's going to learn from each other. I'm excited. I feel like being on the platform team, a lot of your job is to empower builders. What do you think people should build with Decisions API and also computer use agents? Any stuff that you've been building with internally that you think really opens up after the new changes?

30:39Yeah. Okay, let's think. Decisions API, use cases internally have been pretty obvious. The user ops team was jumping on it. We were like, we were going to classify all of our support tickets. What else came up? Obviously, there were the really cool GPD live demos. I'm sure the Codex app team might pick this up and try to do something cool with it. This whole thing started a week ago, so it's very, very early. Oh, one week? There was a big push in evals, LM as a judge, having really low latency there. Right, yeah, yeah. That'll be interesting to see. And then with the agents API, we're basically having a bunch of first-party products at OpenAI built fully on top of it.

31:28We've had the codec security stuff that just went out that's fully built on top of the agents API. we have sort of we're having like a meetings type of thing launching today. I think there was like a demo do you remember like the plugin extensions when Sam was showing it? There was like a demo for like you're in a calendar, you can sort of like have your meeting notes like drop into a Zoom call, like Granola type thing. Yeah, and so all of that stuff is like fully built on top of the agency API and yeah, I'm like just excited to see like we're just getting this out and let's see what people build on top of it.

32:04I think you showed it off very well. The whole edit spaces, pages, collaborate, add in your dot. That's a lot. So there's a lot of inspiration people can build. Yeah. All possible with Astra. Things just move so fast now. People go from idea to execution so quickly. It's amazing. Is there something that you want people to focus on to give you feedback? Maybe you're just putting this out there and there's a fork in the road and you want developers to help you decide. I think Agents API and Decisions API, these are our newest products. Would love any and all feedback on that to figure out where to take them.

32:44I think over here we're very open. On Responses API, which is our workhorse over here, really focused on performance right now. And the performance comes in two main ways. First is just latency. We've been rewriting the whole responses APIs track to make it as fast as possible from a DTFT perspective, DBT perspective. That continues to be a main area of focus for us. The second thing we've been trying to do is really go deep on caching, particularly with these personal agents that are basically a single thread that just goes on and on forever. We've been trying to really up our game on caching.

33:24We provide now guarantees of cache hits within 30 minutes. For one of our users, we just launched a much longer cache window. So we have a 12-hour caching guarantee that we offer so that you have guaranteed cache hits. Is that a public API? Not yet. That's in preview. We're going to try to get that out to everyone as soon as possible. But just pay a little bit more for the cache, right? And we guarantee cache reads for a much longer period. So even if like your instinct thread, for example, like you just like do something on it and then come back to it like three to four hours later, you're still getting the caching performance out of it.

34:04And you cut the pause there quite a bit too, right? With the new model. Oh, yeah, that's right. Yeah, we're like driving down cache reads. Yeah. For builders, they should implement because it's significantly cheaper. Yeah, just like building your apps to be very cache aware and sort of like use our prompt diagnostics or cache diagnostics tool to figure out where things are dropping off. So the caching part is really important. Yeah, I also want to talk about pre-warming. We have that in the API now. So if you know that, hey, I'm going to get a cache, sorry, I'm going to get this prompt. I just want to pre-warm the cache, pay the cache write fee right now, and then have it sort of ready to go for the next 30 minutes for whenever.

34:49And it can spawn many instances of that thread. Exactly, yeah. You can just keep going and have tons and tons of that. And so, yeah, I'm very excited about getting feedback on the low-level performance things so that we can keep making Responses API the most performant and reliable way to build on top of an LLM. And then you basically have our new products where I'm just looking for any and all feedback. Yeah, just use that, right? So yeah, just tell us what this, yeah. Define our roadmap for us, please. So yeah. I think for me, the caching thing, great, right? Like obviously very needed, but at the end of the day, you're still bumping out against a million token context and that's probably not going to change for the foreseeable future.

35:31Like you still need good compression. What is the best practice there? Yeah, yeah, totally. So firstly, OpenAI has its own like proprietary compression, which is in compaction. Compaction, it's in the API. You decide for us, right? Yeah, exactly. So in the agents API, it comes built into the harness. And if you're in responses API, there's like two ways of doing it. One is what we call server-side compaction, which is you basically tell responses API that if you ever hit this threshold of tokens, just auto-compact it and go back. Or sorry, reduce the context being used. And the second way is like slash compact, which is like if you want full control so you can like slash compact at any time, have your own logic on when to like...

36:17It's not EGI. It's EGI. But it is the manual way. Yeah, it is the manual way. And like, I don't know, but a lot of the big coding agents like to do it manually. I mean, like if you look at the Codex implementation of it in the open source Codex harness, you can see that they use slash compact and do it. And there's also like new, by the way, new compaction techniques that we are working on. some of them you will be able to see in the Codex Harness. It's already implemented in the Codex Harness, and so they're some file-based systems that we are experimenting with. So yeah, lots of cool stuff going on around InCompaction as well.

36:57Cool. We're running out of time. I think you've talked a lot about performance and talked a lot about the new APIs that you're launching. Can you give us any other hints as to things that you're interested in as far as the future of the platform is concerned? We're obviously very low level. I used to work at Stripe before this and at Stripe, a lot of the game was building these higher level primitives and products on top of the core payments primitives. I'm always curious about what the best way of doing that is in AI. I think we've had a couple of attempts at that. We had launched Assistant's API way back in the day and wasn't really the right fit.

37:38We were going off with this agents API and it gives you the codex harness, but what's the right amount of flexibility to give in that? That's an open question. How should we have memory vaults and all of these higher level API objects to abstract away more storage concepts? This is a whole space that I'm very curious about figuring out how we design. I think a lot of things in AI are just have a low-level API primitive and see an example harness and go and have your coding agent implement that. But how much of that do we build into the API is a constant question that I'm thinking about. So I don't know if folks have thoughts on that.

38:26If anyone has ideas, it'll be super interesting to hear. Yeah, the analogy I always bring back to end there is you're building an AI cloud, right? Like, which is something that Sam said a year ago. And it's almost like you're kind of doing the AWS invention and you have to do, okay, this is EC2 and this is S3. And this is like, but you're doing the AI native versions of each of these. There are a lot of analogies. So you're pre-warming caches for stuff that, you know. And it's nice that it's all exposed to builders because it just opens up ways that you can build new things. Yeah, absolutely. Okay.

39:01Awesome. Thank you guys. Thank you.

From the publisher

Three months ago Dwarkesh, who has been posting incredible blogs and episodes about RL, posted a framing question for his video essay on RLVR which upset a lot of Computer Use folks: We are no strangers to learning in public and are no strangers to the...

More from Latent Space: The AI Engineer Podcast

All episodes