From API to AGI: Structured Outputs, OpenAI API platform and O1 Q&A — with Michelle Pokrass & OpenAI Devrel + Strawberry team

22:09So we wanted a really clear way for developers to program against this. So if you get something back in the content, you know it's valid, it's JSON parsable. But if you get something back in the refusal field, it makes for a much better UI for you to kind of display this to your user in a different way. And it makes it easier to program against. So really, there was a few goals, but it was mainly to allow the model to continue to refuse, but also with a really good developer experience. Yeah. Why not offer it as like an error code? Because we have to display error codes anyway. Yeah, we falafeled for a long time about API design, as we are wont to do.

22:41And there are a few reasons against an error code. Like you could imagine this being a 4xx error code or something. But, you know, the developer is paying for the tokens. And that's kind of atypical for like a 4xx error code. We pay with errors anyway, right? So 4xxs don't. That's a U error. Right. And it doesn't make sense as a 5xx either, because it's not our fault. It's the way the API model is designed. I think the HTTP spec is a little bit limiting for AI in a lot of ways. Like there are things that are in between your fault and my fault. There's kind of like the model's fault and there's no, you know, error code for that.

23:18So we really have to kind of invent a lot of the paradigm here. Make a 6xx. Yeah, that's one option. There's actually some esoteric error codes we've considered adopting. We're still figuring that out. But I think there are some things like, for example, sometimes our model will produce tokens that are invalid based on kind of our language. And when that happens, it's an error. But, you know, it doesn't. 500 is fine, which is what we return, but it's not as expressive as it could be. So, yeah, just areas where, you know, web 2.0 doesn't quite fit with AI yet. If you had to put in a spec, I was going to just change.

23:53What would be your number one proposal to like rehaul? The HTTP committee to reinvent the world. Yeah, that's good. I mean, I think we just need an error of like a range of model error. And we can have many different kinds of model errors. Like a refusal is a model error. 601, auto refusal. Yeah, again, like, so we've mentioned before that chat completions uses this chat ML format. So when the model doesn't follow chat ML. That's an error. And we're working on reducing those errors, but that's like, I don't know, 602, I guess. A lot of people actually no longer know what chat ML is. Yeah, fair enough.

24:26Because that was briefly introduced by OpenAI and then kind of deprecated. Everyone who implements this under the hood knows it, but maybe the API users don't know it. Basically, the API started with just one endpoint, the completions endpoint. And the completions endpoint, you just put text in and you get text out. And you can prompt in certain ways. Then we released ChatGPT and we decided to put that in the API as well. And that became the ChatCompletions API. And that API doesn't just take like a string input and produce an output. It actually takes in messages and produces messages. And so you can get a distinction between like an assistant message and a user message, and that allows all kinds of behavior.

25:03And so the format under the hood for that is called ChatML. Sometimes, you know, because the model is so out of distribution based on what you're doing, maybe the temperature is super high, then it can't follow chat ML. Yeah, I didn't know that there could be errors generated there. Maybe I'm not asking challenging enough questions. It's pretty rare. And we're working on driving it down. But actually, this is a side effect of structured outputs now, which is that we have removed a class of errors. We didn't really mention this in the blog, just because we ran out of space. But what we're here to do.

25:33Yeah, the model used to occasionally pick a recipient that was invalid. And this would cause an error. But now we are able to constrain to chat ML in a more valid way. And this reduces a class of errors as well. Recipient meaning, so there's a few number of defined roles, like user, assistant, system. So recipient as in picking the right tool. So the model before was able to hallucinate a tool, but now it can't when you're using structured outputs. Do you collaborate with other model developers to try and figure out this type of errors? How do you display them? Because a lot of people try to work with different models.

26:10Yeah. Is there any? Yeah, not a ton. We're kind of just focused on making the best API for developers. A lot of research in engineering, I guess, comes together with evals. You published some evals there. I think Gorilla is one of them. What is your assessment of the state of evals for function calling and structured output right now? Yeah, we've actually collaborated with BFCL a little bit, which is, I think, the same thing as Gorilla. a function calling leaderboard. Kudos to the team. Those evals are great and we use them internally. Yeah, we've also sent some feedback on some things that are misgraded.

26:42And so we're collaborating to make those better. In general, I feel evals are kind of the hardest part of AI. Like when I talk to developers, it's so hard to get started. It's really hard to make a robust pipeline. And you don't want evals that are like 80 % successful because, you know, things are going to improve dramatically. And it's really hard to craft the right eval. You kind of want to hit everything on the difficulty curve. I find that a lot of these evals are mostly saturated, like for BFCL, all the models are near the top already. And kind of the errors are more, I would say, like just differences in default behaviors.

27:16I think most of the models on the leaderboard can kind of get 100 % with different prompting, but it's more kind of, you're just pulling apart different defaults at this point. So yeah, I would say in general, we're missing evals. You know, we work on this a lot internally, but it's hard. Did you, other than BFCL, would you call out any others just for people exploring this space? Sweetbench is actually like a very interesting eval. If people don't know, you basically give the model a GitHub issue and like a repo and just see how well it does at the issue, which I think is super cool. It's kind of like an integration test, I would say, for models.

27:47It's a little unfair, right? What do you mean? A little unfair because like usually as a human, you have more opportunity to like ask questions about what it's supposed to do. And you're giving the model like way too little information to do the job. Yeah, but yeah, SweetBunch targets like how well can you follow the diff format and how well can you like search across files and how well can you write code? So I'm really excited about evals like that because the pass rate is low, so there's a lot of room to improve. And it's just targeting a really cool capability. I've seen other evals for function calling where I think might be BFCL as well, where they evaluate different kinds of function calling.

28:21And I think the top one that people care about for some reason, I don't know personally that this is so important to me, but it's parallel function calling. I think you confirmed that you don't support that yet. Why is that hard? Just more context about it. So yeah, we put out parallel function calling at DevDay last year as well. And it's kind of the evolution of function calling. So function calling v1, you just get one function back. Function calling v2, you can get multiple back at the same time and save latency. We have this in our API. All of our newer models support it. But we don't support it with structured outputs right now.

28:53And there's actually a very interesting trade-off here. So when you basically call our API for structured outputs with a new schema, we have to build this artifact for fast sampling later on. But when you do parallel function calling, the kind of schema we follow is not just directly one of the function schemas. It's like this combined schema based on a lot of them. If we were going to do the same thing and build an index every time you pass in a list of functions, if you ever change the list, you would kind of incur more latency. And we thought it would be really unintuitive for developers and hard to reason about.

29:23So we decided to kind of wait until we can support a no added latency solution and not just kind of make it really confusing for developers. Mentioning latency, that is something that people discovered is that there is an increased cost in latency for the first token. For the first request, yeah. First request. Is that an issue? Is that going to go down over time? Is there just an overhead to parsing JSON that is just insurmountable? It's definitely not insurmountable. And I think it will definitely go down over time. We just kind of take the approach of, and if there's nothing in there you don't want to fix, then you probably ship too late.

29:55So I think we will get that latency down over time. But yeah, I think for most developers, it's not a big concern because you're testing out your integration, you're sending some requests while you're developing it, and then it's fast and prod. So it kind of works for most people. The alternative design space that we explored was like pre-registering your schema, so like a totally different endpoint, and then passing in like a schema ID. But we thought, you know, that was a lot of overhead and like another endpoint to maintain and just kind of more complexity for the developer. And we think this latency is going to come down over time.

30:25So it made sense to keep it kind of in chat completions. I mean, hypothetically, if one were to ship caching at a future point, it would basically be the superset of that. Maybe. I think the caching space is a little underexplored. Like we've seen kind of two versions of it. But I think, yeah, there's ways that maybe put less onus on the developer. but we haven't committed to anything yet, but we're definitely exploring opportunities for making things cheaper over time. Is AGI in agents just going to be a bunch of structure output and function calling one next to each other? How do you see, there's the model that's everything.

30:58Where do you draw the line? Because you don't call these things an agent API, but if I were a startup trying to raise a C round, I would just do function calling and say, this is an agent API. So how do you think about the difference and how people build on top of it for agentic systems? Yeah, love that question. One of the reasons we wanted to build structured outputs is to make agentic applications actually work. So right now it's really hard. Like if something is 95 % reliable, but you're chaining together a bunch of calls, if you magnify that error rate, it makes your application not work.

31:26So that's a really exciting thing here from going from like 95 % to 100%. I'm very biased working on the API and working on function calling and structured outputs. But I think those are the building blocks that we'll be using kind of to distribute this technology very far. It's the way you connect natural language and converting user intent into working with your application. And so I think there's no way to build without it, honestly. You need your function calls to work. Yeah, we wanted to make that a lot easier. And do you think the assistance API thing will be a bigger part as people build agents?

31:59I think maybe most people just use messages and completion. So I would say the assistance API was kind of a bet in a few areas. One bet is hosted tools. So we have the file search tool and code interpreter. Another bet was kind of statefulness. It's our first stateful API. It'll store threads and you can fetch them later. I would say the hosted tools aspect has been really successful. People love our file search tool and it saves a lot of time to not build your own RAG pipeline. I think we're still iterating on the shape for the stateful thing to make it as useful as possible. Right now, there's kind of a few endpoints you need to call before you can get a run going.

32:35and we want to work to make that much more intuitive and easier over time. One thing I'm just kind of curious about, did you notice any trade-offs when you add more structured output, it gets worse than some other thing that you didn't think was related at all? Yeah, it's a good question. Yeah, I mean, models are very spiky and RL is hard to predict. And so every model kind of improves on some things and maybe is flat or neutral on other things. Yeah, it's very rare to just add a capability and have no trade-offs in everything else. So yeah, I don't have something off the top of my head, but I would say, yeah, every model is a special kind of its own thing.

33:11This is why we put them in API dated so developers can choose for themselves which one works best for them. In general, we strive to continue improving on all evals, but it's stochastic. Yeah. Able to apply the structured output system on backdated models like 4.0 May, as well as Mini, as well as August. Actually, the new response format is only available on two models. It's 4.0 mini and the new 4.0. Okay. So the old 4.0 doesn't have the new response format. Okay. However, for function calling, we were able to enable it for all models that support function calling. And that's because those models were already trained to follow these schemas.

33:49We basically just didn't want to add the new response format to models that would do poorly at it because they would just kind of do infinite white space, which is, you know, the most likely token if you have no idea what's going on. I just wanted to call out a little bit more in the stuff you've done in a blog post. So in blog posts, just use cases, right? I just want people to be like, yeah, we're spelling it out for you. Use these for extracting structured data from unstructured data. By the way, it does vision too, right? So that's cool. Dynamic UI generation. Actually, let's talk about dynamic UI.

34:17I think Gen UI, I think is something that people are very interested in. It's your first example. What did you find about it? Yeah, I just thought it was a super cool capability we have now. So the schemas, we support recursive schemas and this allows you to do really cool stuff. Like, you know, every UI is a nested tree that has children. So I thought that was super cool. You can use one schema and generate like tons of UIs. As a backend engineer who's always struggled with JavaScript and front end, like for me, that's super cool. I've now, we've now built a system where I can get any front end that I want.

34:49So yeah, that's super cool. The extracting structured data, like the reality of a lot of AI applications is like you're plugging them into your enterprise business and you have something that works, but you want to make it a little bit better. And so the reliability gains you get here is like, you'll never get a classification using the wrong enum. It's like, it's just exactly your types. So really excited about that. Like maybe hallucinate the actual values, right? So let's clearly state what the guarantees are. The guarantees is that they fit the schema, but the schema itself may be too broad because the JSON schema type system doesn't say like, I only want to range from one to 11.

35:26You might give me zero. You might give me 12. So yeah, JSON schema. So this is actually a good thing to talk about. So JSON schema is extremely vast and we weren't able to support every corner of it. So we kind of support our own dialect and it's described in the docs. And there are a few trade-offs we had to make there. So by default, if you don't pass in additional properties in a schema, by default, that's true. And so that means you can get other keys, which you didn't spell out, which is kind of the opposite of what developers want. You basically want to supply the keys and values and you want to get those keys and values.

35:58And so then we had a decision to make. It's like, do we redefine what additional properties means as the default? That felt really bad. It's like, there's a schema that's predated us. Like, you know, it wouldn't be good. It would be better to play nice with the community. And so we require that you pass it in as false. You know, one of our design principles is to be very explicit. And so developers know, you know, what to expect. And so this is one where we decided, you know, it's a little harder to discover. But we think you should pass this thing in so that we can have, like, a very clear definition of what you mean and what we mean.

36:25There's a similar one here with required. By default, every key in JSON scheme is optional. But that's not what developers want, right? You'd be very surprised if you passed in a bunch of keys and you didn't get some of them back. And so that's the trade-off we made, is to make everything required and have the developers spell that out. Is there a require false? Can people turn it off or they're just getting all? So developers can, basically what we recommend for that is to make your actual key a union type. And so, yeah, make it union, int, and null. and that gets you the same behavior. Any other of the examples you want to dive into?

36:58Math, chain of thought? Yeah, you can now specify like a chain of thought field before a final answer. This is just like a more structured way of extracting the final answer. One example we have, I think we put up a demo app of this math tutoring example, or it's coming out soon. Did I miss it? Oh, okay. Basically, it's this math tutoring thing and you put in an equation and you can go step by step and insert. This is something you can do now with structured apps. In the past, a developer would have to like specify their format. and then write a parser and parse out the model's output, which would be pretty hard.

37:29But now you just specify steps, and it's an array of steps. And every step you can render, and then the user can try it, and you can see if it matches and go on that way. So I think it just opens up a lot of opportunities. For any kind of UI where you want to treat different parts of the model's responses differently, structured outputs is great for that. I remembered my question from earlier. I'm basically just using this to ask you all the questions as a user, as a daily user of the stuff that you put out. So one is a tip that people don't know, and I confronted you on Twitter, which is you respect descriptions of JSON schemas, right?

37:59And you can basically use that as a prompt for the field. Totally. I assume that's blessed and people should do that. Intentional, yeah. One thing that I started to do, which it could be a hallucination of me, is I changed the property name to prompt the model to what I wanted to do. So, for example, instead of saying topics as a property name, I would say like brainstorm a list of topics up to five or something like that as like a property name. I could stick that in the description as well. But is that too much? Yeah, I would say, I mean, we're so early in AI that people are figuring out the best way to do things.

38:36And I love when I learn from a developer, like a way they found to make something work. In general, I think there's like three or four places to put instructions. Yeah. You can put instructions in the system message. And I would say that's helpful for like when to call a function. So it's like, you know, let's say you're building a customer support thing and you want the model to verify the user's phone number or something. You can tell the model in the system message, like here's when you should call this function. Then when you're within a function, I would say the descriptions there should be more about how to call a function.

39:04So really common is someone will have like a date as a string, but you don't tell the model, like, do you want year, year, month, month, day, day, or do you want that backwards? And that's what a really good spot is for those kind of descriptions. It's like, how do you call this thing? And then sometimes there's like really stuff like what you're doing. It's like name the key by what you want. So sometimes people put like, do not use. And, you know, if they don't want, you know, this parameter to be used, except only in some circumstances. And really, I think that's the fun nature of this. It's like you're figuring out the best way to get something out of the model.

39:35So you don't have an official recommendation is what I'm hearing. Well, the official recommendation is, you know, how to call a model system instructions. Exactly, exactly. the hierarchy. Do you benchmark these type of things? So like same with date. It's like description, it's like return it in like ISO 8. Yeah. Or if you call the key date in ISO 86001. I feel like the benchmarks don't go that deep. But then all the AI engineering kind of community, like all the work that people do, it's like, oh, actually this performs better. But then there's no way to verify. Right. You know, like even the I'm going to tip you a hundred thousand dollars or whatever.

40:10Like some people say it works. Some people say it doesn't. Do you pay attention to the stuff as you build this? Or are you just like, the model is just going to get better. So why waste my time running evals on these small, small things? Yeah, I would say to that, I would say we basically pick our battles. I mean, there's so much surface area of LLMs that we could dig into. And we're just mostly focused on kind of raising the capabilities for everyone. I think for customers, and we work with a lot of customers, really developing their own evals is super high leverage. Because then you can upgrade really quickly when we have a new model.

40:41You can experiment with these things with confidence. So yeah, we're hoping to make making evals easier. I think that's really generally very helpful for developers. For people, I would just kind of wrap up the discussion for structured outputs. I immediately implemented, we use structured outputs for AI News. I use Instructor and I ripped it out. And I think I saved 20 lines of code. But more importantly, it was like we cut it by 55 % of API costs based on what I measured because we saved on the retries. Nice. Yeah, love to hear that. Yeah, which people I think don't understand when you can't just simply like add instructor or add outlines.

41:17You can do that, but it's actually going to cost you a lot of retries to get the model that you want. But you're kind of just kind of building that internally into the model. Yeah, I think this is the kind of feature that works really well when it's integrated with like the LLM provider. Yeah, actually, I had folks, even my husband's company who works at a small startup, they thought we were just retrying. inside to make the reflog post. We are not retrying. We're doing it in one shot, and this is how you save on latency and cost. Awesome. Any other behind-the-scenes stuff, just generally on structured outputs?

41:47We're going to move on to the other models. Yeah, I think that's it. That's excellent products, and I think everyone will be using it, and we have the full story now that people can try out. So roadmap would be parallel function calling. Anything else that you've called out as coming soon? Not quite soon, but we're thinking about does it make sense to expose custom grammars beyond JSON schema? What would you want to hear from developers to give you information, whether it's custom grammars or anything else about structured output? What would you want to know more of? Just always interested in feature requests, what's not working.

42:16But I'd be really curious, what specific grammars folks want? I know some folks want to match programming languages like Python. There's some challenges with the expressivity of our implementation. And so, yeah, just kind of the class of grammars folks want. I have a very simple one, which is a lot of people try to use GPT as judge, right? Which means they end up doing a rating system. And then there's like 10 different kinds of rating systems. There's a Likert scale, whatever. If there was an officially blessed way to do a rating system with structured outputs, everyone would use it. Yeah, yeah, that makes sense.

42:49I mean, we often recommend using logprobs with classification tasks. So rather than sampling, you know, let's say you have four options like red, yellow, blue, green. Rather than sampling, you know, two tokens for yellow, you can just do like A, B, C, D and get the log probs of those. You know, the inherent randomness of each sampling isn't taken into account and you can just actually look at what is the most likely token. I think this is more of like a calibration question. Like if I asked you to rate things from 1 to 10, a non-calibrated model might always pick 7, just like a human would. Right.

43:23So like actually have a nice gradation from 1 to 10 would be the rough idea. Yeah. And then even for structured outputs, I can't just say have a field of rating from 1 to 10 because I have to then validate it. And, you know, it might give me 11. Yeah, absolutely. Yeah. So what about model selection? Now you have a lot of models. When you first started, you had one model endpoint. I guess you had like DaVinci. But like most people are using one model endpoint. Today you have like a lot of competitive models. And I think we're nearing the end of the 3.5 run RIP. How do you advise people to experiment, select, both in terms of task and cost?

44:00What's your playbook? In general, I think folks should start with 4.0 Mini. That's our cheapest model. And it's a great workhorse. Works for a lot of great use cases. If you're not finding the performance you need, maybe it's not smart enough, then I would suggest going to 4.0. And if 4.0 works well for you, that's great. Finally, there's some really advanced frontier use cases. And maybe 4.0 is not quite cutting it. And there I would recommend our fine-tuning API. Even just like 100 examples is enough to get started there and you can really get the performance you're looking for. We're recording this ahead of it, but you're announcing some fine-tuning stuff that people should pay attention to.

44:38Yeah. Actually, tomorrow we're dropping our GA for GPT-4O fine-tuning. So 4O Mini has been available for a few weeks now and 4O is now going to be generally available. And we also have a free training offering for a bit. I think until September 23rd, you get 1 million of free training tokens a day. This is already announced, right? Am I talking about a different thing? So that was for 4.0 Mini and now it's also for 4.0. So we're really excited to see what people do with it. And it's actually a lot easier to get started than a lot of people expect. I think they might need tens of thousands of examples, but even 100 really high quality ones or a thousand is enough to get going.

45:12I think people's concerns about fine tuning is that they're kind of locked into a model. And I think you're paving the path for migration of models. As long as they keep their original data set, they can at least migrate nicely. Yeah, I'm not sure what we've said publicly there yet. But we definitely want to make it easier for folks to migrate. It's the number one concern. I'm just, you know, it's obvious. Absolutely. I also want to point people to, you have official model selection docs, where it's in the guide, we'll put it in the show notes, where it says to optimize for accuracy first. So prompt engineering, rag, evals, fine tuning.

45:44This was done at DevDay last year, so I'm just repeating things. And then optimize for cost and latency second. And there's a few sets of steps for optimizing latency, so people can read up on that stuff. Yeah, totally. We had one episode with Nicola Scarlini from DeepMind, and we actually talked about how some people don't actually get to the boundaries of the model performance. You know, they just kind of try one model and it's like, oh, LLMs cannot do this, and they stop. How should people get over the hurdle? It's like, how do you know if you hit the model performance or like you hit skill issues?

46:14You know, it's like your prompt is not good or like try another model and whatnot. Is there an easy way to do that? That's tough. Some people are really good at prompting and they just kind of get it right away. And for others, it's more of a challenge. I think there's a lot we can do to make it easier to prompt our models. But for now, I think requires a lot of creativity and not giving up right away. Yeah. And a lot of people have experience now with ChatGPT. You know, before ChatGPT, the easiest way to play with our models was in the playground. But now kind of everyone's played with it, with a model of some sort, and they have some sort of intuition.

46:43It's like, you know, if I tell you my grandma is sick, then maybe I'll get the right output. And we're hoping to kind of remove the need for that. But playing around with chat GPT is a really good way to get a feel for, you know, how to use the API as well. Will prompt engineering be here forever, or is it a dying art as the models get better? I mean, it's like the perennial question of software engineering as well. It's like, as the models get better at coding, you know, if we hit 100 on Sweebench, what does that mean? I think there will always be alpha in people who are able to clearly explain what they're trying to build.

47:14Most of engineering is figuring out the requirements and stating what you're trying to do. And I believe this will be the case with AI as well. You're going to have to very clearly explain what you need. And some people are better than others at it. And people will always be building. It's just the tools are going to get far better. That's two weeks you release two models. There's GPC 4.0, 2034, 08, 06. And then there's also ChatGPC 4.0 latest. I think people are a little bit confused by that. and then you issued a clarification that it was once chat tuned and the other is more function calling tuned.

47:43Can you elaborate? Yeah, totally. So part of the impetus here was to kind of be very transparent with what's on ChatGPT and in the API. So basically, we're often trading models and they're different use cases. So you don't really need function calling for user defined functions in ChatGPT. And so this gives us kind of the freedom to build the best model for each use case. So in ChatGPT latest, we're releasing kind of this rolling model. The weights aren't pinned as we release new models. This is literally what we use. Yeah. So it's in what's in ChatGPT. So it's very good for like chat style use cases.

48:17But for the API broadly, you know, we really tune our models to be good at things that developers want, like function calling and structured outputs. And when a developer builds their application, they want to know that kind of the weights are stable under them. And so we have this offering where it's like, if you're tuning to a specific model and you know your function works, you know it will never change the weights out from under you. And so those are the models we commit to supporting for a long time. And we think those are the best for developers. But we want to give it up. You know, we want to leave the choice to developers.

48:46Like, do you want the ChatGPT model or do you want the API model? And you have the freedom to choose what's best for you. I think it's for people, they do want to pin model versions. so I don't know when they would use ChatGPT like the rolling one unless they're really just kind of cloning ChatGPT which is like, why would they? I mean, I think there's a lot of interesting stuff that developers can do when unbounded and so we don't want to limit them artificially so it's kind of survival of the fittest like whichever model is better you know, that's the one that people should use I talked about it to my friends as like, this isn't a new thing and basically OpenAI has never actually shared with you the actual ChatGPT model and now they do.

49:28Well, it's not necessarily true. Actually, a lot of the models we have shipped have been the same, but you know, sometimes they diverge and it's not a limitation we want to stick around. Anything else we should know about the new model? I don't think there was no evals announced or anything, but people say it's better. I mean, obviously, LMSYS is way better on everything, right? It's like number one in the world. Yeah, we published some release notes. They're not as in-depth as we want to be yet because it's still kind of a science and we're learning what actually changes with each model and how can we better understand the capabilities.

50:01But we are trying to do more release notes in the future and keep folks updated. But yeah, it's kind of an art and a science right now. You need the best evals team in the world to help you figure this out. Yeah, evals are hard. We're hiring if you want to come work on evals. Hold that thought on hiring. We'll come back to the end on what you're looking for because obviously people want to join you and they want to know what qualities you're looking for. So we just talked about API versus chat GPT. What's, I guess, like the vision for the interface? You know, the mission of OpenAI is like build AGI that is accessible.

50:34Like where is it going to come from? Totally. Yeah. So I believe that the API is kind of our broadest vehicle for distributing AGI. You know, we're building some first party products, but they'll never reach every niche in the world and kind of every corner and community. And so really love working with developers and seeing the incredible things they come up with. I often find that developers kind of see the future before anyone else, and we love working with them to make it happen. And so really the API is a bet on going really broad. We'll go very deep as well in our first-party products, but I think just that our impact is absolutely magnified by every developer that we uplift.

51:08They can do the last malware where you cannot. Like ChatGPT is one type of product, but there's many other kinds. In fact, you know, I observed, I think in February, Basically, ChatGPT's user growth stopped when the API was launched because everyone's going to be able to take that and build other things. That has not become true anymore because ChatGPT growth has continued to grow. But then you're not confirming any of this. This is me quoting similar web numbers, which have very high variants. Well, the API predates ChatGPT. The API is actually opening on its first product and the first idea for commercialization that predates me as well.

51:43Wide release. Like GA, everyone can sign up and use it immediately. That's what I'm talking about. But yeah, I mean, I do believe that. And that means you also have to expose all of OpenAI models, right? Like all the multimodal models. We'll ask you questions on that. But I think that API mission is important. It's interesting that hottest new programming language is supposed to be English, but it's actually just software engineering, right? It's just, you know, we're talking about HTTP error codes. Right. Yeah, I think, you know, engineering is still the way you access these models. And I think there are companies working on tools to make engineering more accessible for everyone.

52:21But there's still so much alpha in just writing code and deploying. Yeah. One might even call it AI engineering. Exactly. Yeah. So there's lots of war stories from building this platform. We started at the start of your career and then we jumped straight to structured outputs. There's a whole thing, like two years that we skipped in between. Right. What have become your principles? What are your favorite stories that you like to tell? We had so much fun working on the Assistance API and leading up to Dev Day. You know, things are always pretty chaotic when you have an externally, like a date that is hard and there's like a stage and there's like a thousand people coming.

52:56You can always launch a wait list. We're trying hard not to because, you know, we love it when people can access the thing on day one. And so, yeah, the Assistance API, we had like this really small team and just working as hard as we could to make this come to life. But even actually the morning of, I don't know if you'll remember this, but Sam did this keynote and Roman came up and they gave free credits to everybody. So that was live, fully live, as were all of the demos that day. But actually maybe like two hours before that, we had a little outage and everyone was like scrambling to make this thing work again.

53:32So yeah, things are early and scrappy here. And, you know, we were really glad. We were a bit on the edge of our seat watching it live. What's the plan B in that situation? If you can share. Play a video. It's just classic DevRel, right? I don't know. I mean, I actually don't know what the plan B was. No plan B. No failure. But we just, you know, we fixed it. We got everything running again. And the demo went well. Just hire cracked Waterloo tracks. Skill issues as usual. Sometimes you just got to make it happen. I imagine it's actually very motivating. But I did hear that after Dev Day, the whole company got a few weeks off just to relax a little bit.

54:08Yeah, we sometimes get, like, we just had the week of July 4th off. And yeah, it's hard to take vacation because people are working on such exciting things. And it's like you get a lot of FOMO on vacation. So it helps when the whole company is on vacation. Mentioning Ace Assistance API, you actually announced a roadmap there and things have developed. I think people may not be up to date. What's the offering today versus, you know, one year ago? Yeah, so we've made a bunch of key improvements. I would say the biggest one is in the file search product. Before, we only supported, I think, like 20 files per assistant.

54:39And the way we used those files was less effective. Basically, the model would decide based on the file name whether to search a file. And there's not a ton of information in there. So our new offering, which we shipped a few months ago, I think now allows 10K files per assistant, which is dramatically more. And also, it's a kind of different operation. So you can search semantically over all files at once rather than just kind of the model choosing one up front. So a lot of customers have seen really good performance. We also have exposed more like chunking and re-ranking options. I think the re-ranking one is coming, I think, next week or very soon.

55:13So this kind of gives developers more control and more flexibility there. So we're trying to make it the easiest way to kind of do RAG at scale. Yeah. I think that visibility into the RAG system was the number one thing missing from DevDay. And then people got their first impressions and then they never looked at it again. So that's important. The Reranker is a core feature of, let's say, some other foundation model labs. Is OpenAI going to offer a Reranking service, a Reranker model? So we do Reranking as part of it. I think we're soon going to ship more controls for that. Okay, got it. And if I'm an existing Lanchain, Lama Index, whatever, how do you compare?

55:51Do you make different choices? Where does that exist in the spectrum of choices? I think we are just coming at it, trying to be the easiest option. And so ideally, you don't have to know what a re-ranker is. You don't have to have a chunking strategy. And the thing just kind of works out of the box. So I would say that's where we're going. And then giving controls to the power users to make the changes they need. Awesome. I'm going to ask about a couple other things, just updates on stuff also announced at DevDay. And we talked about this before. Determinism, something that people really want.

56:22DevDay will announce the seed per M as well as system fingerprint. And objectively, I've heard issues. Yeah. I don't know what's going on. Yeah, the seed parameter is not fully deterministic and it's kind of a best effort thing. So you'll notice there's more determinism in the first few tokens. That's kind of the current implementation. We've heard a lot of feedback. We're thinking about ways to make it better, but it's challenging. It's kind of trading off against, you know, reliability and uptime. Other maybe underrated API only thing, logic bias. That's another thing that kind of seems very useful and that maybe most people are like, it's a lot of work.

56:56I don't want to use it. Do you have any examples of like use cases or like products that are made a lot better through using it? So yeah, classification is the big one. So you launch it by your valid classification outputs. And, you know, you're more likely to get something that matches. We've seen that people launch it by its like punctuation tokens, maybe trying to get more succinct writing. Yeah, it's generally very much a power user feature. And so not a ton of folks use it. I actually wanted to use it to reduce the incidence of the word Delve. Yeah. Have people done that? Probably. I don't know.

57:29Is Delve one token? You're probably, you got to do a lot of permutations. It's used so much. Maybe it is. Depends on the tokenizer. Are there non-public tokenizers? I guess you cannot answer or you would admit it. Are the 100K and 200K vocabs, like the ones that you use across all models? Yeah, I think we have docs that publish more information. I don't have it off the top, but I think we publish which tokenizers for which model. Okay. So those are the only two. The tiering rate limiting system. I don't think there was an official blog post kind of announcing this, but it was kind of mentioned that you started tying fine-tuning to tiering and feature rollouts.

58:05Just from your point of view, how do you manage that? And what should people know about the tiering system and rate limiting? Yeah, I think basically the main changes here were to be more transparent and easier to use. So before developers didn't know what tier they're in, and now you can see that in the dashboard. I think it's also, I think we publish how you move from tier to tier. And so this just helps us do kind of gated rollouts. For the fine-tuning launch, I think everyone tier two and up has full access. That makes sense. I would just advise people to just get to tier five as quickly as possible.

58:36Sure. Like a gold star customer, you know? I don't know. It seems to make sense. Do we want to maybe wrap with future things and kind of like how you think about designing and everything? So you just mentioned you want to be the easiest way to basically do everything. What's the relationship with other people building in the developer ecosystem? ecosystem? Like I think maybe in the early days, it's like, okay, we only have these APIs and then everybody helps us, but now you're kind of building a whole platform. How do you make decisions? Yeah, I think kind of the 80-20 principle applies here. We'll build things that kind of capture, you know, 80 % of the value and maybe leave the long tail to other developers.

59:11So we really prioritize by like, how much feedback are we getting? How much easier will make this, will this make something like an integration for a developer? So yeah, we want to do more in this space and not just be an LLM as a service, but kind of AI development platform as a service. Ooh, okay. That ties into a thing that I put in the notes that we prepped. There are other companies trying to be AI development platform. So you will compete with them or they just want to know what you want build so that they can build it? Yeah, it's a tough question. I think we haven't determined what exactly we will and won't build, but you can think of something, if it makes it a lot easier for developers to integrate, you know, it's probably on our radar and we'll, you know, stack rank by impact.

59:55Yeah. So there's like cost tracking and model fallbacks. Model fallbacks is an interesting one because people do it. I don't think it adds a ton of value, but like if you don't build it, I have to build it because if one API is down or something, I need to fall back to another one. Yeah. I mean, the way we're targeting that user need is just by investing a lot in reliability. And so we have - Just don't fail. I mean, we have improved our uptime pretty dramatically over the last year. And it's been the result of a lot of hard work from folks. So you'll see that on our status page and our continued commitment going forward.

1:00:28Is the important thing about owning the platform that it gives you the flexibility to put all the kind of messy stuff behind the scenes? Or how do you draw the line between what you want to include? Yeah, I just think of it as like, how can we onboard the next generation of AI engineers, as you put it? Like what's the easiest way to get them building really cool apps? And I think it's by building stuff to kind of hide this complexity or just make it really easy to integrate. So I think of it a lot as like, what is the value add we can provide beyond just the models that makes the models really useful?

1:01:00Okay, we'll touch on four more features of the API platform that we prepped. Batch, Vision, Whisper, and then Team Enterprise stuff. So you wanted to talk about Batch. Yeah. So the rough idea is the contract between you and me is that I give you the Batch job. you have 24 hours to run it. It's kind of like spot inst for the API. What should people know about it? So it's half off, which is a great savings. It also works with like 4.0 Mini. So the savings on top of 4.0 Mini is pretty crazy. Like the stuff you can do - Like 7.5 cents or something per million. Yeah, I should really have that number top of mind, but it's like staggeringly cheap.

1:01:37And so I think this opens up a lot more use cases. Like let's say you have a user activation flow and you want to send them an email, like maybe every day or like at certain points in their user journey. So now you can do this with the batch API and something that was maybe a lot more expensive and not feasible is now very easy to do. So right now we have this 24 hour turnaround time for half off and curious, would love to hear from your community, like what kind of turnaround time do they want? I would be an ideal user of batch and I cannot use batch because it's 24 hours. I need two to four.

1:02:06Two to four hours. Okay. Yeah, that's good to know. But yeah, just a lot of folks haven't heard about it. It's also really great for like evals, running them offline. You generally don't need them to come back within two hours. I think you could do a range, right? Two to four for me, I need to produce a daily thing. And then 24 for the average use case. And then maybe a week, a month, who cares? For people who just have a lot to do. Yeah, absolutely. So yeah, that's Fatch API. I think folks should use it more. It's pretty cool. Is there a future in which six months is free? What? Is there super small shards of GPU runtime that like over a long enough timeline, you can just run all these things for free.

1:02:46Yeah, it's certainly possible. I think we're getting to the point where a lot of these are like almost free. That's true. Why would they work on something that's like completely free? I don't know. Okay, so Vision. Vision got GA'd. Last year, people were so wild by the GPT-4 demo. That was primarily Vision. What was it like building the Vision API? Yeah, the Vision API is super cool. We have a great team working there. I think the cool thing about Vision is that it works across our APIs. So there's, you can use it in the Assistance API, you can use the Batch API and Chat Completions that works with structured outputs.

1:03:16I think it just helps a lot of folks with kind of data extraction where, you know, the spatial relationships between the data is too complicated and you can't get that over text. But yeah, there's a lot of really cool use cases. I think the tricky thing for me is understanding how frequent to turn vision from single images into effectively just always watching. And right now, I think people just send a frame every second. Will that model ever change? Will there just be like, I stream you video and then... Yeah, I think it's very possible that we'll have an API where you stream video in. And maybe, you know, to start, we'll do the frame sampling for you.

1:03:54Because the frame sampling is the default, right? Right. But I feel like it's hacky. Yeah, I think it's hard for developers to do. And so, you know, we should definitely work on making that easier. Is there in the batch API, do you have like a time guarantees, like order guarantees? Like if I send you a batch request, of like a video analysis. I need every frame to be done in order. For batch, you send like a list of requests and each of them stand alone. So you'll get all of them finished, but they don't kind of chain off each other. Well, if you're doing a video, you know, if you're doing like analyzing a video.

1:04:24I wasn't linking video to batch, but that's interesting. Yeah, well, a video is like, you know, if you have a very long video, you can just do a batch of all the images and let it process. That's a cool idea. You could offer like sequential truth. Yeah, yeah, yeah. Yeah, exactly. But the whole point of batch is you're just using kind of spare time to run it. Let's talk about my favorite model, Whisper. Oliver, I built this thing called SmallPodcaster, which is an open source tool for podcasters. And why does Whisper API not have diarization when everybody is transcribing people talking? That's my main question.

1:04:58Yeah, it's a good question. And you've come to the right person. I actually worked on the Whisper API and shipped that. That was one of my first APIs I shipped. Long story short is that like Whisper V3, which we open sourced, has, I think, the diarization feature, but there's some like performance trade-offs. And so Whisper V2 is better at some things than Whisper V3. And so it didn't seem that worthwhile to ship Whisper V3 compared to like the other things in our priorities. I think we still will at some point, but yeah, it's just, you know, there's always so many things we can work on. It's tough to do everything.

1:05:29We have a Python notebook that does the diarization for the pod, but I would just like, you can translate like 50 languages, but you cannot tell me who's speaking. That was like the funniest thing. There's like an XKCD thing about this, about hard problems in AI. I forget the one. Yeah, yeah, yeah, exactly. Tell me if this was taken in a park. And like, that's easy. And it's like, tell me if there's a bird in this picture. And it's like, give me 10 people in a research team. It's like, you never know which things are challenging. And diaritization is, I think, you know, more challenging than expected.

1:05:59Yeah, it still breaks a lot with like overlaps, obviously. sometimes similar voices it struggles with. Like I need to like double read the thing. Totally. But yeah, great model. I mean, it would take us so long to do transcriptions. And I don't know why like small podcasts has better transcription than like mostly every commercial tool. It beats the script. And I'm like, I'm just using the model. I'm literally not doing anything. You know, it's just a notebook. So yeah, it just speaks to like sometimes just using the simple OpenAI model is better than like figuring out your own pipeline thing.

1:06:30Totally. I think the top feature request there just would be, again, using you as a feature request dump is being able to bias the vocab. I think there is, in raw whisper, you can do that. You can pass a prompt in the API as well. But you pass it in a prompt? Okay. Yeah. There's no more deterministic way to do it. So this is really helpful when you have acronyms that aren't very familiar to the model. And so you can put them in the prompt and you'll basically get the transcription using those correctly. We have the AI engineer solution, which is just a dictionary. Nice. with like all the way misspelled it in the past and then G-sub and like replace the thing.

1:07:04If it works, it works. Like that's engineering. It's like, you know, Llama with like one L or like all these different things or like Langchain and like transcribes Langchain and like capitalization does a bunch of like three or four different ways. Yeah. You guys should try the prompt feature. I love these like kind of pro tip. Okay, fun question. I know we don't know yet, but I've been enjoying the advanced voice mode. It really streams back and forth and it handles interruptions. How would your audio endpoint change when that comes out? We're exploring, you know, new shape of the API to see how it would work in this kind of speech-to-speech paradigm.

1:07:40I don't think we're ready to share quite yet, but we're definitely working on it. I think just the regular request response probably isn't going to be the right solution. For those who are listening along, I think it's pretty public that OpenAI uses LiveKit for the ChatGPT app, which seems to be the socket-based approach that people should be at least up to speed on. I think a lot of developers only do request response, and that doesn't work for streaming. Yeah. When we do put out this API, I think we'll make it really easy for developers to figure out how to use it. It's hard to do audio. It'll be a paradigm change.

1:08:10Okay. And then I think the last one on our list was team enterprise stuff. Audit logs, service accounts, API keys. What should people know? What's in the enterprise offering? Yeah, we recently shipped our admin and audit log APIs. And so a lot of enterprise users have been asking for this for a while. The ability to kind of manage API keys programmatically, manage your projects, get the autolog. So we've shipped this and for folks that need it, it's out there and happy for your feedback. Yeah. Awesome. I don't use them. So I imagine it's just like build your own internal gateway for your internal developers to manage your deployment of OpenAI.

1:08:44Yeah. I mean, if you work at like a company that needs to keep track of all the API keys, it was pretty hard in the past to do this in the dashboard. We've also improved our SSO offering. So that's much easier to use now. The most important feature of the enterprise company. Yeah, people love SSO. All right, let's go outside of OpenAI. What about just you personally? So you mentioned Waterloo. Maybe let's just do, why is everybody at Waterloo cracked? And why are people so good? And like, why have people not replicated it? Or any other commentary on your experience? The first is the co-op program.

1:09:16It's obviously really good. You know, I did six internships, learned so much in those. I think another reason is that Waterloo is like, you know, it's very cold in the winter. It's pretty miserable. There's like not that much to do apart from study and like hack on projects. And there's this big like hacker mentality, you know, there's a Hack the North is a very popular hackathon. And there's a lot of like startup incommuters. It's kind of just has this like startup and hacker ethos. Then that combined with the six internships means that you get people who like graduate with two years of experience and they're very entrepreneurial and they're down to grind.

1:09:50I do notice a correlation between climate and the correctness of engineers. So it's no coincidence that Seattle is the birthplace of Microsoft and Amazon. I think I had this compilation of Denmark where people like, so it's the birthplace of C++, PHP, Turbo Pascal, Standard ML, BNF, the thing that we just talked about, MD5 Crypt, Ruby on Rails, Google Maps, and V8 for Chrome. and it's because according to Bjorn Solstrup the Kratos C++, there's nothing else to do. Well, you have Lena Storbald's in Finland. I mean, you hear a lot about this in relation to SF. People say, you know, New York is way more fun.

1:10:28There's nothing to do in SF and maybe it's a little by design that all tech is here. The climate is too good. If we also have fun things to do. Nature is so nice, you can touch grass. Why are we not touching grass? You know, restaurants close at like 8 p.m. Like that's what people are referring to. There's not a lot of like late night dining culture. Yeah. So you have time to wake up early and get to work. You are a book recommender or book enjoyer. What underrated books do you recommend most others? Yeah, I think a book I read somewhat recently that was very formative was The Making of the Prince of Persia.

1:10:59It's a Stripe Press book. That book just made me want to work hard like nothing I've ever read. It's just like this journal of what it takes to build incredible things. So I'd recommend that. Yeah. It's funny how video games are, for a lot of people, at least for me, kind of like the some of the moments and technology like when i played the sense of time on ps2 was like my first playstation 2 game i was like man this thing is so crazy compared to any playstation one game and it's like wow my expectations for like the technology i think like open ai is a lot of similar things like the advanced voice it's like you see that thing and then you're like okay what i can expect from everybody else is kind of raised now you know totally another book i like to plug is called Misbehaving by Richard Thaler.

1:11:42He's a behavioral economist and talks a lot about how people act irrationally in terms of decision making. And I actually think about that book like once a week, probably at least when I'm making a decision and I realized that, you know, I'm falling into a fallacy or, you know, it could be a better decision. Yeah. You did a minor in psych. I did. Yeah. I don't know if I learned that much there, but it was interesting. Is there like an example of like a cognitive bias or misbehavior that yeah you just love telling people about yeah people so let's say you won tickets to like a taylor swift concert and i don't know how much they're going for but it's probably like ten thousand dollars oh okay or whatever sure and like a lot of people are like oh i have to keep these like i won them it's ten thousand dollars but really it's the same decision you're making if you have ten thousand dollars like would you buy these tickets and so people don't really think about it rationally like would they rather have ten thousand dollars or the tickets for people who want it a lot of the time it's going to be the ten thousand dollars but their bias is because they want it the world organized itself this way and you should keep it for some reason yeah oh okay i'm pretty familiar with this stuff there's also a loss version yes loss aversion version of this where it's like if i take it away from you you respond more strongly than if i give it to you yes if people are like really upset if they like don't get a promotion but if they do get a promotion they're like okay phew it's like not even you know excitement it's more like we react a lot worse to losing something which is why like When you join a new platform, they often give you points and then they'll take it away if you don't do some action in the first few days.

1:13:10Yeah, totally. The book references people who operate very rationally as econs, as a separate group to humans. And I often think, what would an econ do here in this moment and try to act that way? Okay, let's do this. Are LLMs e-cons? I mean, they are maximizing probability distributions. Minimizing loss. Yeah. So I think way more than all of us, they are Yukons. Whoa. Okay. So they're more rational than us? I think their optimization functions are more clear than ours. Yeah. Just to wrap, you mentioned you need help on a lot of things. Yeah. Any specific roles, call outs, and also people's backgrounds?

1:13:49Like, is there anything that they need to have done before? Like, what people fit well at OpenAI? Yeah, we've hired people with all kinds of backgrounds, people who have PhD and ML or folks who just done engineering like me. And we're really hiring for a lot of teams. We're hiring across the Applied Org, which is where I sit for engineering and for a lot of researchers. And there's a really cool model behavior role that we just dropped. So yeah, across the board, we'd recommend checking out our careers page and you don't need a ton of experience in AI specifically to join. I think one thing that I'm trying to get at is like what kind of person does well at OpenAI?

1:14:25I think objectively you have done well and I've seen other people not do as well and basically be managed out. I know it's an intense environment. I mean, the people I enjoy working with the most are kind of low ego, do what it takes, ready to roll up their sleeves, do what needs to be done and unpretentious about it. Yeah. I also think folks that are very user-focused do well on kind of API and chat GVT. Like the YC ethos of build something people want is very true at OpenAI as well. So I would say low ego, user-focused, driven. Cool. Yeah, this was great. Thank you so much for coming on. Thanks for having me.

1:15:01That was an excellent conversation with Michelle, which we're just about to ship before we then heard that the OpenAI Strawberry, now named O1 Release, was coming. So we delayed our podcast to organise an emergency meet-up with more friends from OpenOI and Latent Space Enjoyers in San Francisco to capture quick takes on O1, as well as answer some questions.

1:15:40Scaling up inference for second to none. We set the count at GPT's passe. O1's the hotshot, the H-E-I way. Not a system, just one model. That's on the rise, an extraordinary alien. Of unknown size. O1 minis, it'd be quick. O1 preview's just a tease. Encoding STEM, minis to B's knees. Token counts the same. GPT for those old news. Hiding thought chains like a cognitive field. The state's news, O1's begun. Left Michelle and GPT for O just for fun. Strawberry fields where the bikes all run Scaling up inference for second to none Bigger context loading, no more scrolls Handling long tasks, the chunking's got holes Tools not in the kit yet, the functions will call Multimodal magic, O1's on the ball RL's the secret sauce in our code, brew GPT-40 can't catch up even in day bug mode You think it seems slow cause we summarize the jest But when answers drop, they're faster than the rest Hiding firm, where O1's the gun Lafmachelle and GPT-4 just for fun Strawberry fields where the bites all run Scaling up in France, we're second to none

1:16:53There are two sections to our O1 coverage today. The first is our recorded meetup audio from Thursday, which will also be posted on YouTube, recapping the most important takeaways about O1 from SWIX, and then an open-ended Q &A session with Romaine Hewitt and other OpenAI representatives. We apologise in advance for the audio quality, but it was the best we could gather on short notice. The second is a recap of the official OpenAI Developers AMA on Friday, which you'll hear about from me. Finally, you'll hear some O1 demos from Min Kim of Lighthouse and your favourite latent space co-host Alessio.

1:17:36Stay tuned. Yeah, thanks everyone for joining this emergency 01 meetup. And thank you to OpenAI for the model and the food. Very much appreciated for feeding our brains and our stomachs. So I'm just going to yap a little bit for the AI News recap that we did today. You know, the one thing I really appreciate is that they generally tend to make it API available, even if the API can be surprisingly hard to work with. So anyway, so there are a few data points that I'm just going to recap, and then we'll let the actual experts answer any questions or at least say a few words as well. So there are effectively two models released today.

1:18:13I like to show it in this chart because I think that people don't really understand that the O1 preview is really just a checkpoint to a broader thing and they already have, and that one's still in process. The other notable thing that a lot of people are talking about that was in our summary of Discord discussion and Twitter discussion today was the pricing. That's notably a little bit higher than, for example, Opus. And we'll talk about the other stuff. There were a bunch of blog posts. It was actually a little bit hard to find or read, but this is what I think everyone should read. If you signed up to the newsletter, you would have got it.

1:18:50So each model has its own blog post. There's a technical research blog post. There's a system card. And then there's also docs. And then there's some other videos if you care about the humans behind the project. I tried to collect together the alpha that people drop, because obviously that is less. The main thing, so this is actually, if you remember the Goodwill Hunting Board problem, I gave it that, and it solved it. And so the tweet that I was going to work on was basically, 01 is Matt Damon, then it can solve the Goodwill Hunting problem. And the new paradigm really is in ChatGPT, it shows this chain of thought.

1:19:27As far as I understand, in the API, it doesn't actually show that. And this chain of thought is only shown in ChatGPT, and it's not the actual chain of thought. It is a hidden, obscured summary of the chain of thought. And that is materialized as reasoning tokens, which you do pay for. The token limits are expanded. And this was the nastiest part about me actually using the API today, was that I could not just one-for-one port over the code. Things like Instructor no longer work because there's no system role anymore. There's no temperature, there's no tool calling, there's no streaming. And especially there's no max tokens because you have no constraints or visibility on how many tokens are being used.

1:20:03But there is a lot of really interesting information in the docs that gives you some mental model of how these reasoning tokens are given you. And how if they take too many tokens and I ran it into exactly this for this task. Because it only solved the first problem and it could not solve second, third, fourth. Because it ran into the 128 tokens issue. okay the the third thing that i want to talk about is also the the evals we're very used to eat to really good evals but i think this is obviously again new highs in a lot of the categories that we're we're used to the the thing i think that is more important than just the evals is also scaling laws for test time compute this one i i really recommend people check out jim fan's comments on this which is just an excerpt from the the technical blog post that we now not only do train time compute, but also test time compute.

1:20:53And they seem to scale in a log later fashion that we can actually predict and invest in. And so that seems to be very, very helpful. As far as I understand, I basically covered, looked at the entire Strawberry team's Twitter, which is the unofficial documentation. I really, I think the main thing that people are trying to figure out is how much can you do in the wild versus must you train the models to do this? And the only hints that we have so far is that they're definitely doing chain of thought using RL, and it's much better than just prompting. And that's something that, obviously, as engineers, we don't have access to.

1:21:33And you can read Jason's post for the rest. And then finally, because I don't want to take up too much time, it seems like a lot of people are calling out O1 Mini for its relative outperformance. Usually, when there's two models, one big, one small, or small, medium, big. Usually the big model gets all the love and then the other stuff is just sizably smaller. But O1 actually outperforms if you go all the way back. Lucas from DeepMind actually called out, if this is true, why do you use O1 Preview at all? You should just use Mini or O1. And that's because O1 doesn't exist yet. But Mini basically scores better in STEM because it was trained to do that.

1:22:12That does not mean that it is better than O1 Preview. It's just trained to do better in STEM. so that's my quick recap we're lucky enough to have a couple people from open ai here uh do you want to say a few words and sure yeah what people should know yeah i'll just leave it open and here you can take this if you want just gonna just to magnet it yeah cool all right hey everyone thank you so much swix for organizing this like emergency o1 meetup thank you sean so thank you Vipoo, MindDB, Decibel for hosting. But yeah, super exciting day at OpenAI with the release of O1. Sounds like Sean already gave you all the lowdown on the launch day.

1:22:53But yeah, it's a new day for us. I think we're very excited to see what you builders, developers, are going to start building with this model, in particular agents or agentic apps that maybe before you had some challenges making them work. So yeah, we're very, very jazzed at the idea of like you kicking the tires of the model. It's obviously the very beginning. We have a lot of work ahead of us to give access to everyone and increase the rate limits and so on and so forth. But yeah, very excited to be here and can't wait to see the demos tonight and in the days to come. We have a few members, by the way, of the team tonight.

1:23:25There's Lindsay from our comms team. There's Edwin also from our developer experience team. But yeah, excited. And Anuj from our engineering team as well. So yeah, excited to be here and answer questions and can't wait to see the demos. Thanks again. Do people have questions? Oh yeah. Yeah. That is a great question. I think we'll... Lindsay, you want to take that one? So we decided to reset the counter with O1 because the capabilities that this model presents are just totally new and different from the GPT series. So we're still going to invest in the GPT series. Like, 4.0 is not going away. We're still very much invested in multi-modality.

1:24:05But we wanted to represent sort of this shift in capabilities towards reasoning, which is just something that, while models have shown us that we're being able to reason by stitching things together in the past, this is just net gain. So we started over with O1. O stands for Open AI, not O like Omnimodel and GPT-4 O. We know that we're not great at naming models. But yeah, that's where O1 is. Just on the naming, I'll just follow up. I read someone saying somewhere, I cannot find the source, that there was no GPT-5. Someone said this is the end of GPT-5, and now they're 01 and restarting. Is it RIP-GPT?

1:24:47We're still going to invest in the GPT series. I don't know if we're going to have the GPT-5, but this is 01, it's different. I think you should expect us to continue doing the kinds of model where historically people were thinking would be GPT-5, what we actually call it as a name. Something else. But again, our naming has always been confusing, so just a little bit of a problem. They're bad at me. Everyone is kind of bad at maybe models. So it's like similar to OSX to Apple 11? Yeah, kind of like that. You can think of it as that. Question, how would you just phrase it? So for people that don't follow super closely, there's the 4.0 series, the traditional 4, these new ones, really the category.

1:25:29I mean, you can think about the GPT 4.0 series, for instance, like that's your workhorse. We still expect developers, builders to rely on GPT-4O heavily, and we keep on investing in that series for sure. And O1 will be for this new class of apps that require steps of reasoning, for instance. On the API team and the developer experience team and overall at OpenAI, we expect that developers, builders like all of you will actually use both in concert. It's very likely that you'll need O1 and you'll reach for it for very specific, complex, hard problems. and then for a lot of tasks that work great already with gpt for foro like summarization labelization and all that you'll still have the photo foro mini to rely on so we definitely expect builders are going to use both in concert Will there be a way to speed up the chain of thought?

1:26:18I don't know how we didn't have time. Do you want to take that? Definitely a ton of ideas to speed up these models because in the end, a lot of the classical approaches to make these masters still apply and we just started documentation. So now, my team is actively working on a bunch of improvements that we have in this game. It's also day one for the API. So as Sean mentioned, like when he did his recap, not everything is exposed in the API just yet because it's a net new model. But over time, we'll keep on adding features as well and the tools, etc. to the model. I could ask all of it. Basically, it's still going to be a change, though, right?

1:27:01The better it gets are easy in many different steps. Maybe you guys can make the steps shorter, but there is a very funny other. Bring the attention kind of vibe to speeding up the easy base. Right? I mean, I can just think about, yes, there's a change of thinking, but there are easy. are you saying that are the restriction of the machine mechanisms that can be helped to make it possible? Obviously we're going to look at all of that moving forward. I think the obvious things that we are aware of is pretty context of them. As you can see, because I know I'm on my next I'm not so lazy. So that is something that I would say is like that.

1:27:47And then I think we need to go to I hope it is getting faster and faster as we walk in the nation. And we deserve and we never did that but for now it is taking what we have which seems like a good quality but using it it's really good, right? Or something I don't think it's going to be so we have worked a lot in terms of She did what I did, so I got to get out of it. And got this really well. So if I may, I don't expect that to kind of make people run, but I did make it around the sets of alternating. Yeah. That's a lot of it in the way we could have been doing how we can do it. It's still in fact so we have to be able to do it.

1:28:40Will you be able to decide or specify how much you do you want to spend at this time? because we're just people of the model and have always been able to decide how much the things it was to spend on it. Is it better than models and the question, deciding what's to do, like, is life or obviously there are also some other issues training, post-training, and other and, like, all these things that I have to offer, but the core of the business is not going to be there. This weekend we're doing a big hackathon here, and of course people will use the new model, and this is like remote as well. So we've got over a thousand participants and like 300 here in SF.

1:29:20But as you guys release a new model and it's different than just traditional elements, what do you want people to build? Like any crazy different ideas other than just, you know, summarization, bot, whatever. Like, is it? Hey, summarization is important. I think that's a great question. The reason why we're putting these things like out there on day one and rolling out to all developers is because, you know, We know developers are going to be coming up with the most interesting use cases for these models. The recommendation would be anything you've tried to build in the past with a GPT-4 class model that may not have worked so well or things that you felt like were limitations of the model.

1:29:59Try that again with O1. Are there complex challenges, especially around coding, for instance, like refactoring a large code base or potentially trying to troubleshoot an issue in a large code base? those kind of clever things that GPT-4OB would maybe not be the best model for. Try reaching out for like O1 Mini for this, who's particularly great at coding. Yeah, and basically the reason why we're here, the reason why we want to talk to all of you is because we love feedback. We're definitely going to hear from what you build and hopefully make those models better for you. So how do we get access to O1?

1:30:32Some of the scheduling types are access to O1. We all have like O1 preview. O1 is not like released yet. So for now, it's all about O1 Preview and O1 Mini. We'll communicate later on about O1. I think that guy works at OpenAI, wherever this is. Yeah, yeah. In the chat community app, you can kind of see the thinking steps are on site. Yeah. Are you planning on exposing that eventually? Where people can then go in maybe direct on a single step on the channel? Possibly, yeah. I think for the APIs, it's really early days. I think we wanted to just make it work on day one with chat completions, like the most popular API that people have.

1:31:10But over time, we'll be trying to add more knobs for the developers to kind of understand what's happening. It's also for the first time that it's the first time we have a model that may take like 30 seconds to think hard about the problem. Like how do we make the developer experience better for your app? I think there's a lot of work ahead of us. Yeah. Following up on what he said, the fact that you can now see like little by little like the way it's piercing together, the problem and the solution, Does that mean that there's no visibility towards why it's giving the answer? Because there's a big black box and by AI it's on-played response.

1:31:45So does this mean that now there's more visibility towards this? Why is this answer the one coming out? Yeah, I think what's interesting about this new model is that you can see the reasoning, right? For instance, you might have seen in the blog post today that we had a few videos with a quantum physicist, for instance, like using the model to go through like equations and looking step by step at like, oh yeah, I understand what the model is actually doing. So yeah, there is a bit more about like, the chain of thought turns into like a tokens where you understand like what's happened during the thinking process.

1:32:19In multi-comer conversations, do the recent tokens also go against the context? So what's the idea of the context? So if it's a model that you keep having conversations and all the reasoning it does, I think we documented what's happening in our platform docs. I don't have the exact answer top of mind, but we shared that. I think you had that in your diagram right there. So I ran into this using ChatGPT, and the thing just stalls. You're not able to enter any more messages. So it doesn't do any truncation for you. It lets you continue chatting. Yeah, and the API does here, but I think something we have to inform.

1:33:01You see a set of outputs, like what you're doing. So thinking and explaining what thought is the answer of explaining what thought. All of that gets added. So what you see is added in the thinking. However, each additional step does another thinking process as you continue to think. And the amount of tokens it has to generate to do the thinking also is just like a splashback. So as you add in the chat GPD the subsequent multi-turn step, what you see in the UX is the total accumulated tokens, but then the thinking tokens are also acquired which we don't show the count of. But they don't accumulate, they take you to five thinking steps, all the hidden tokens for those thinking steps will accumulate.

1:33:56So if I do another chat message after that, only the summary that the US bank will say. One thing I noticed also about the thinking tokens is the Sweepbench verified. So Sweepbench now requires you to submit your thinking trajectories. And that's why these guys co-sign, which I don't know if you talked to them at all while they were here. I have not personally, but some people at the company I have, yes. So they actually scored higher than 0.1 on Sweebench verified and higher than, this is full Sweebench. But they could not get on the leaderboard because the leaderboard requires them to submit their thinking trajectories.

1:34:42So basically, Cosign did the same thing that you are doing, which is they're not showing the thinking trajectories because it's IP. I thought it was an interesting observation. yeah cool there's yeah nothing nothing to say beyond that it's like it's it's kind of cool but like it's so o1 reported their own results and if you put their results side by side they're slightly lower this was before i think yeah yeah this is this is a blog post this is from the blog post of sweetbench verified and then the o1 results were somewhere here one of these guys I don't know where the speed bench one is. Somewhere here.

1:35:20Yeah. Yeah. You put them in the system card. System card. Okay. Yeah. Anyway. Cool. I'm going to keep you too long, but thank you for that. You had one more? I'm wondering, isn't one of the ones that show kind of like a scale of compute? I got this one. And it looks like it's about one or two. I was wondering if you could experiment with more what is the kind of co-scaling for the events in the future? Yeah.

1:35:54I think all we have to share on this is already on the blog post at this time. Yeah.

1:36:02Is the current model that's available it seems to be on 20 seconds with a 30-part average? Is it Is there a maximum of compute? Like, would it ever be a vendor of thinking? Or is that a long model? Like, is there a cap on the focus you guys are not thinking part of the current model? I think we shared, like, on the API side, like, the current way it's working. But for everything else, like, I think we are very flexible at the moment. I'm sure we'll learn along the way. I don't think there's anything in particular that would be like, oh, this is the number, this is the cap. I see. So in theory, if a question is complex, it could be both being for longer than 100 % of questions.

1:36:48Is that all right? Possibly, yeah, yeah, yeah. Can you do that already? Yeah. Yeah, when you ask how many R's in strawberry, for instance, now it gets it right, but it's also pretty fast. Versus like getting a very complex problem. Yeah, this question, this is the Google hunting question. and it took 121 seconds. Oh, wow, wow. Yes. So, on that, like the same way, the whatever, or the whatever, was cut off sometimes, because you didn't have the focus, you can answer, just made that at some point, you can just cut off the thing, and you just don't get that, because it's a tough cap, or the reasoning for those.

1:37:28That's right. That's right, yes. Okay. One last question. Right, so, So, given that you are doing one layer of the foundation model, and trying to innovate one layer higher, does this kind of mean that we have reached some kind of a foundation model here? That we can't get a picture of that? And we are trying to debate, like, the higher level. I said that. I'm not sure I'm going to do this. I don't think we're going to do it. I don't think we're going to do it. Okay. and other than you. And that. That is also. Oh, boy. OK, last one, and then we'll move on. Yeah, we'll move on. Sorry. . I emailed something to talk about how GP4 or what chain of thoughts were the same number of thoughts as O1 for one question?

1:38:32Is there anything? What was the difference, basically, in emails? Don't think we've published any of that. I mean, I'll think we've published anything. Take a look at our research club. There's a technical report out there. Yeah. There's a lot of material on the O1 hub. There's two research clubs, one for a model. There's a system card that's 85 pages long if you want to be holding. We do paper club Wednesdays. Probably do. It's a great forum for it. There's a lot of material on there, so read through it. There's a lot of emails. A lot of support to some of this, please. Sorry? I need to go more to summarize it.

1:39:07Great idea. It cannot use tools and doesn't have images, so very hard, very hard. But thank you so much. Cool. No, thank you all. Thank you, Sean. Thank you, everyone.

1:39:19I think we have two demos. We're going to host an AMA on Twitter tomorrow morning. If you have more questions, we'll have a bunch of researchers. Right? Yeah. from the company I dance handle we'll send a party full text and what else was I going to say so yeah it should take place around 10am tomorrow so I think we'll post a tweet between 8 or 9 something like this to kind of source questions and then we'll have some researchers to go deeper into some of those questions

1:39:53awesome

1:39:56thank you all Yeah, I think we have like two demos. And first is the O1 visa demo, and then the second is the O1 model demo. but actually it's also AI enabled so you can come up. That was the O1 emergency meetup that happened the evening of Thursday. See show notes for full video and photos and a huge thank you once again to the OpenAI team for supporting us. More excellent updates are coming for OpenAI Dev Day 2024 and you can bet on the latent space crew being there to take you through it. For part two of our O1 coverage, I'm going to recap the top takeaways from the Friday Twitter AMA done by the research team.

1:40:41Tybor Blaho asks, how did you come up with the new names for O1, O1 Preview and O1 Mini? World's Fair speaker Romaine Hewitt from OpenAI Answers. Reasoning represents a new level of AI capability, so we decided to reset the counter back to one and introduce this series as OpenAI O1. Preview because it's a preview of the capabilities and mini because it's smaller. And O stands for OpenAI. Former guest Simon Willison asks, why is it called O1 mini and not O1 mini preview? Shengjia Zhao from OpenAI answers, You're exactly correct here. O1 Preview is a preview of the upcoming O1 model, while O1 Mini is not a preview of a future model.

1:41:33O1 Mini might get updated in the near future as well, but there is no guarantee. Max Schwarzer from OpenAI adds, We can't discuss the precise sizes of the two models, but O1 Mini is much smaller and faster, which is why I can offer it to all free users as well. O1 Preview is an early version of O1 and isn't any larger or smaller. Friend of the pod, Alex Volkov asks, Can you guys clarify, is O1 a system that runs chain of thought behind the scenes and gives us an answer or a model that reasons with special tokens and just hides those tokens from us at the output and just shows the final answer?

1:42:16Many folks are confused by this. Noam Brown from OpenAI answers I wouldn't call 01 a system. It's a model, but unlike previous models, it's trained to generate a very long chain of thought before returning a final answer. We don't have plans to reveal KOTI to users either in the API or chat GPT. There is no guarantee the summariser is faithful. Hyeongwon Chung from OpenAI adds Answer tokens are typically, though not necessarily, a lot shorter than the QoT. It's a single model. We're not sharing param counts right now. Thinking stage is a summary of the thought process. So it appears to be slower than it is.

1:43:01O1 is a single model. Jason Way from OpenAI adds, I think it's safe to say that no matter how hard you prompt GPT-4-0, you probably won't get an IOI gold. Many, many people ask, Will lower upper bounds of thinking time or test time compute be a controllable variable via API? Noam Brown from OpenAI Answers In the future, we'd like to give users more control over how much time the model spends thinking. Hyeong-won Chung from OpenAI Answers OpenAI 01 Preview doesn't use tools. For the future models, we are considering adding support for function calling, code interpreter and browsing. John O 'Whitaker from AnswerAI asks, How are the team finding it for research code?

1:43:53HTML Snake is cool, but I'd love to hear their own use cases from the research trenches. Wukas Kondratziuk, Strawberry Training Infra Lead at OpenAI Answers. A couple of PRs to the OpenAI codebase was already authored solely by O1. Many, many people ask, What are the plans for the next steps? Preview phase duration, availability of the reel, O1 from benchmarks, and missing functionality tools, etc. Ahmed Elkishki from OpenOI Answers. While we don't have the exact preview duration, we plan to iteratively deploy additional functionality, including tool capabilities like code interpreter and browsing.

1:44:35Today's guest, Michelle Pocras of the API team, adds, We know we're missing a lot of the features that developers need. We're working to enable function calling, structured outputs, developer system messages, and our other standard params. Jerry Tworek from OpenAI responds to a question about vision, a.k.a. image recognition. Hopefully soon, working hard on it. Not providing any hard date yet. Wenda Joe from OpenAI adds, OpenAI 01 is built with multimodal and achieves SOTA on MMMU. We're working hard on making it safe to share with everyone. Nikunj Handa from OpenOI adds, We'll add batch AP support once the rate limits are higher.

1:45:19Many, many people asked about pricing. Shangjia Zhao from OpenOI answers, Historically, prices go down 10x everyone. Two years, the trend will probably continue. For example, the cost per token of GPT-40 mini dropped by 99 % since TextAVinci003. Swix asks, What inverse scaling have you seen under O1? Jason Wei from OpenAI answers, I don't know of any great examples of inverse scaling. You can see from our blog post that on some types of prompts like personal writing, it seems like OpenAI O1 preview is not much better than GPT-4-0 or even slightly worse. Finally, I will read you the recap from Tyba Blaho, which we link in the show notes.

1:46:08Model names and reasoning paradigm. Open AIO, one is named to represent a new level of AI capability. Preview indicates it's an early version of the full model. Mini means it's a smaller version of the O1 model, optimized for speed, O as OpenAI. O1 is not a system. It's a model trained to generate long chains of thought before returning a final answer, size and performance of O1 models. O1 Mini is much smaller and faster than O1 Preview, hence offered to free users in future. O1 Preview is an early checkpoint of the O1 model, neither bigger nor smaller. O1 Mini performs better in STEM tasks, but has limited world knowledge.

1:46:54O1 Mini excels at some tasks, especially in code-related tasks compared to O1 Preview. Input tokens for O1 are calculated the same way as GPT-4O using the same tokenizer. O1 Mini can explore more thought chains compared to O1 Preview input token context and model capabilities. Larger input contexts are coming soon for O1 Models. O1 Models can handle longer, more open-ended tasks with less need for chunking input. Compared to GPT-4-0, O1 can generate long chains of thought before providing an answer, unlike previous models. There is no current way to pause inference during CodeT to add more context, but this is being explored for future models, tools, functionality and upcoming features.

1:47:43O1 Preview doesn't use tools yet, but support for function calling, code interpreter and browsing is planned. Tool support, structured outputs and system prompts will be added in future updates. Users might eventually get control over thinking time and token limits in future versions. Plans are underway to enable streaming and considering reasoning progress in the API. Multimodal capabilities are built into O1, aiming for state-of-the-art performance in tasks like MMMUCOT, Chain of Thought. Reasoning. O1 generates hidden chains of thought during reasoning. no plans to reveal Cotee tokens to API users or chat GPT.

1:48:27Cotee tokens are summarised, but there is no guarantee of faithfulness to the actual reasoning. Instructions and prompts can influence how the model thinks about a problem. Reinforcement learning, RL, is used to improve Cotee in O1, and GPT-4O cannot match its Cotee performance through prompting alone. Thinking stage appears slower because it summarises the thought process. Even though answer generation is typically faster, API and usage limits, O1 Mini has a weekly rate limit of 50 prompts for ChatGPT Plus users. All prompts count the same in ChatGPT. More tiers of API access and higher rate limits will be rolled out over time.

1:49:10Prompt catching in the API is a popular request, but no timeline is available yet. Pricing, fine-tuning and scaling. Pricing of O1 models is expected to follow the trend of price reductions every one, two years. Batch API pricing will be supported once rate limits increase. Fine tuning is on the roadmap, but no timeline is available yet. Scaling up O1 is bottlenecked by research and engineering talent. New scaling paradigms for inference compute could bring significant gains in future generations of models. Inverse scaling isn't significant yet, but personal writing prompts show a one preview performing only slightly better than GPT-40, or even slightly worse, model development and research insights.

1:49:59O1 was trained using reinforcement learning to achieve reasoning performance. The model demonstrates creative thinking and strong performance in lateral tasks like poetry, O1's philosophical reasoning and ability to generalise, such as deciphering ciphers, are impressive, O1 was used by researchers to create a GitHub bot that pings the right code owners for review. In internal tests, O1 quizzed itself on difficult problems to gauge its capabilities. Broad world domain knowledge is being added and will improve with future versions. Fresher data for O1 Mini is planned for future iterations of the model, October 2023 currently, prompting techniques and best practices.

1:50:44O1 benefits from prompting styles that provide edge cases or reasoning styles. O1 models are more receptive to reasoning cues and prompts compared to earlier models. Providing relevant context in retrieval augmented generation, our age, improves performance. Irrelevant chunks may worsen reasoning general feedback and future enhancements. Rate limits are low for O1 preview due to early-stage testing but will be increased. Improvements in latency and inference times are actively being worked on remarkable model capabilities. O1 can think through philosophical questions like, what is life? Researchers found O1 impressive in its ability to handle complex tasks and generalise from limited instruction.

1:51:33O-1's creative reasoning abilities, such as quizzing itself to gauge its capabilities, showcase its high-level problem solving. That's it for the AMA. Now for demos from Min and Alessio. So exciting. Thanks so much for having us. Wow, super fun. This is coincidental. Hi, everyone. My name is Min. I'm the founder and CEO of Lighthouse, where we make US immigration go fast. This is coincidentally a fun day because we're talking about the O-1 model and we specialize as in the O-1 visa. Is anybody in the room like on an O-1 by any chance? Just out of curiosity. Oh, amazing. Cool. Just in case for anybody who's unaware, I was born in South Korea.

1:52:15This is like near and dear to my heart, but mostly because I've spent my whole career working with early stage technologists and really amazing talent. And we make it really hard for you to continue to be here and build what you want to build. And the interesting thing about something about U.S. immigration that you might not know is that there's a sort of best kept secret called the O-1 visa which is a an employment visa that is meant for really popularized by startup by the technology or the entertainment industry originally so for a long time it was used by models musicians so on and so forth you know Melania Trump got to got in on an O-1 type thing and then over the last five years five six seven years or so it's become really popularized by the startup and technology industry for like highly technical talent and entrepreneurs.

1:52:58But the problem is that it's still really underused. So if you think about, just to give you some context, there's only about 11 ,000 people who get it every year. And that is in contrast to the 750 ,000 people who applied for the H-1B three years ago. So it's just hugely underused visa. And why is that the case? It's because most lawyers are really bad at it. Most providers don't know how to do it. And then when you go through the process, you end up having to do a lot of it yourself. And for those of you who have been through the process, you may or may not have had understand this, where the actual application is like a three to four hundred page research report about you and all the amazing things that you have done and how those professional or academic achievements map to the criteria that USCIS wants to see.

1:53:42It's sort of like applying like a really, really intense college application like on steroids. And at Lighthouse, we want to make this process go way faster because traditionally if you go through this, one, you'll often get told you don't qualify. And then two, you'll end up having to do so much for yourself and it'll take you six months. And it really doesn't have to be that way. So what we do is try to make the process as simple as possible. We kind of like 30 days or fewer, you can do it in four weeks. And then the government can back, we'll get back to you within two to three weeks or so.

1:54:10And so the whole thing doesn't have to really take more than two months. And we were really lucky to get to support SWIX. And so in some ways, this is SWIX's Oh, one, congratulations party as well. Yay. And we would love to have loved to support more people. And part of the reason I'm here today is because I kind of wanted to give you a behind-the-scenes look of we've basically built a platform that's sort of like a paralegal out of the box so that you don't have to spend time going back and forth with an attorney and spending hours and hours and hours drafting a lot of this yourself. We are happy users and customers of OpenAI.

1:54:42Please give us access to the mini because the model itself actually said, I think in your paper that it is really good for STEM problems. And a lot of the people that we work with are STEM talent. And so a lot of what we end up having to do is translate the technical papers that they've published and the work that you have done in a way that a lay person, a USCIS officer who's probably, you know, 40, 50 years old, probably spent some time in the military and doesn't really have context about what makes AI special right now. And an amazing thing is right now everybody wants AI talent in the US and so if you are working in that field we definitely want to help you be able to stay and come here.

1:55:20So what I'm going to show you is actually I'm just gonna do this on the fly and so we'll test the boundaries of all this. But let's say we were gonna do let's I actually just picked Daniel because he's the second author on the paper and so we're gonna do it in the fly and so let's say he is an expert in artificial intelligence just ignore that. Hold on, let me just make sure I can. Yes? Oh, oh my god, can you not see? Okay, is that better? Amazing. This is what we've built so that we can make this process go fast for you. And let's say it goes to OpenAI. Cool, let's just ignore that. We don't need everything right then and there and what is really cool is once we can oh okay give me one second we're gonna do it here but let's pretend that daniel needs we have to talk about his papers we can just add his page and let's pick his like first five papers and what this is actually going to do is there's a whole bunch of stuff we've built in the background here, but we've basically fetch all the papers, read all of them, summarize them, and then we've layered on top of it a ton of industry know-how, basically like all the legally speak based on like tons of feedback from the legal experts that we work with.

1:56:45And so it takes a little bit of time because there's a lot of content that it's going through, but it'll basically spit out the entire thing in terms of what we need to do. And then all we need to do is edit and revise it, and then we'll show it to you, and then you make sure that it like is sound right and we can do this with a whole bunch of other things it'll usually take like a minute or two because it's like literally picking up all the papers and then reading them and then summarizing them but so we'll let that run but basically we can do that and we do it to find out whether or not your pay comparables are high we do it to figure out what the papers that you've written or research that you've done if you honestly we can put like this into it and have it have it like synthesized in a way that is intended for uscis to read and yay it's done and so it'll basically tune it to whatever we decide that we wanted to focus on right and so this in this case we happen to want it to be focused on machine learning product and so everything is kind of like oriented around that and it will summarize all the papers that we asked for the five that we wanted it'll pick up all the papers that we want and so if you're familiar, SWIX, we gave you this 500-page document that we said we're going to literally physically print and ship to USCIS.

1:58:04And we basically turn a process that's probably 120 hours when you work with a traditional provider into something that's probably closer to 20. And so that's how we can do it fast. And we would love to work with the O1 mini model because a lot of what we have to do is make sure that this is as precise as possible while still being precise as possible. Yeah. And then just for kicks, I want to show you what it might look like, what it might say if 01 Mini. What might it say if we asked it to go read this?

1:58:54And it should. Oh, yay. So we would say Daniel developed OpenAIM-E, and it's a cost-efficient reasoning model designed to excel in STEM fields. And so it'll basically summarize all of this in the way that it needs to be kind of like written for the officer. and in essence the way that it would typically work is either you would have to write this for the lawyer because they wouldn't really know how to summarize this and you don't really know what good looks like because you know probably really for reasonably you would say well I wrote an abstract that's what the abstract is for but it's not written in the way that USIS would understand so you would probably have to spend a lot of time doing this or they would do it and they would do a shoddy job.

1:59:39And so we kind of combined the industry know-how and the magic of AI now to make this process a lot more efficient.

1:59:49Yay! So if you are looking for an O-1 visa, we also support all employment-based visas in the U.S., so come find me. My name is Min.

2:00:04All right. So the idea here, as Sean mentioned, the O1 API doesn't actually have a lot of the things like a structure output, tool calling, everything like that. So I spent the last three hours trying to give those tools to it. I'm using a thing called E2B. I do not work at E2B, but we invested in E2B, and E2B is the only thing that actually does this. It's basically like a cloud sandbox. for your LLM to use. So the way I built it, I take a prompt. In this case, I'm from Rome, so I wanted to see a visualization of the growth of the Roman Empire, as one does. And I had to remind O1 that it has access to a code interpreter, even though it doesn't know it.

2:00:44And then what I did is use 4.0.mini to extract the code. So O1 is just doing the planning. Obviously, it writes the code in the actual output. So if you see, this is what O1 actually returns. So it's like, you know, these are the steps. This is like something we can use for the data visualization. This is like explaining all, you know, a lot more information than it's going to go in the actual thing. This is how you process it. And then what I've done is just putting 4.0 mini as a way to extract all of it. So it's like, you know, you're a software engineer that receives a plan. And then from the plan, you create one single script that does everything that you need to execute it.

2:01:24and I've created this simple kind of like structure output model. So there's the code that needs to be run, all the pip packages that are required to run the code, the file name, and then the command to execute. So what I'm doing after that is basically open a sandbox. Actually, the only reason why this doesn't work, we cannot get the PNGs out of the sandbox right now. We're trying to figure out the right path. But basically, you install all the pip packages, and then you run the command that the model generated. it and then there's also a self healing loop so if there's a error with the sandbox it it tells you what the error is i feed that back again to the model and it's like hey actually this happened and then i use for mini again to extract the new code rewrite to the file that i was running before and kind of like loop loop through it and so what that looks like see as we cannot actually get this file out but it starts from this document which is like you know a very kind of like it's a good plan but like it's not something you can actually run it goes through 4.0 so there's one error which is most of the time people are not using negative dates in pandas so the the o1 is actually saying hey you know in the year minus 23 and pandas is like what does that even mean nobody ever says that but it is a valid date so you actually have to convert to a different format to can only do to do before christ dates so that's something that o1 didn't do but 4.0 many can do and update the code.

2:02:49So yeah, I'll publish this on GitHub just so people have it. But this is like an easy way to get from 01 planning to like for a mini code extraction and then actually have a runtime that does it. As you might have seen in the security review of the model, it's like quite good at jailbreak. So I did not want to run it even in Docker in the local environment. So if E2B gets hacked, Vasek is right there. He's going to be mad. But yeah, this is it. And we'll send out the link after. And I'm actually going to run the code locally. I just kind of copy-pasted that thing here. So if you just do Python test.py, these are all the charts that actually get generated.

2:03:32So this is the empire of regulation over time. This is the territorial areas over time. And these are kind of overlapped together. So again, this is a data set that there's not really a good source out there for. There's kind of like all these different pieces. So I think one of the most exciting things about these models is also like data set generation, you know, like how do we take all of this text data that has been trained into it and bring it out in a structured way. So yeah, that's it. Thank you.