Is GPT-OSS Actually Any Good?

0:00Today on the AI Daily Brief, I look at the day one reactions to the major model releases that happened yesterday. Before that, in the headlines, you guessed it, more model releases. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

0:17All right, friends, quick announcements before we dive in today. First of all, thank you to today's sponsors, Blitzy, Vanta, and Plum. To get an ad-free version of the show, go to patreon.com slash ai daily brief. Welcome back to the AI Daily Brief headlines edition, all the daily AI news you need in around five minutes. It is quite blurry right now between the headlines and the main, as everything this week is just all new model releases. Yesterday, as OpenAI was getting in the open source game again for the first time since like 2019, and Google was resetting our expectations of world simulation models, Eleven Labs, who some of you know as the company that I have frequently used to do a version of my own voice for reading articles on this show, launched an AI music service.

0:59Now, there are a bunch of things that are interesting about this. First of all, this is Eleven Labs' first expansion beyond generative speech, where they are at this point one of, if not the default market leader in voice cloning, text-to-speech, and translation. This new music release, called Eleven Music, is a feature-complete music generation suite with instrumental and lyrics, meaning it competes directly with services like Suno and Udeo. You can create an entire song based on a prompt.

1:41The example you just heard is from the prompt Powerful Choir, Orchestral, Large Crescendo, Ethereal, Spacious Reverb, Expansive, Cinematic, Classical Music. Or this one, Roller, Liquid Drum and Bass, Fast Technical Percussion, Thick Bass, Influences from IDM, Soft Female Vocals, Heavily Processed. All the truth we push away All the truth we push away

2:12Eleven Labs say the model is capable of generating a tune within minutes, and yet as impressive as it is, and the quality at first glance is incredibly impressive to me, the biggest selling point over others in this space is that the Eleven Music model is claimed to be legal for broad commercial use. Both Suno and Udio are currently facing lawsuits from the major record labels for using their recordings and training data. Now, it's very unclear how music generated using those models is going to be treated under copyright law. Those are big battles to be fought. But Eleven Labs is just trying to short-circuit that entirely, abstaining from including music from the major labels in their training sets.

2:47Instead, they signed agreements with independent rights management firms Cobalt Music and Merlin Network, with Cobalt saying that their artists had been given the choice to opt into their music being used in the training data or not. Ed Newton-Rex, who is the CEO of Fairly Trained, an AI copyright advocacy non-profit, posted positively, saying, Co-founder of Eleven Labs confirms that their new AI music model is trained only on songs they've licensed. This is really good to see. When a handful of AI companies try to tell you generative AI can only be built with scraped copyrighted work, remember that the majority of AI music models license their training data, including now Eleven Labs' model.

3:18Certainly first impressions just from a quality standpoint are very good. Chubby writes, 11 Labs music was not on my bingo card. Holy moly, does this sound good. We're now entering the era of real AI music. We are witnessing history. Now, even though 11 Labs says this is cleared for commercial use, when you look into the terms, there's still some questions in here. There are restrictions around, for example, whether you can distribute these things to music streaming services and exactly where you can use them. But it's pretty clear where this has a lot of disruptive potential is around commercial uses of music like game development, advertising, startup launch videos, basically anything that would have had you digging around before for the perfect sound clip or for the perfect licensed song, you can now just generate on the fly.

4:01Creator Tyler Ganges says, it's so simple, so easy, and will save me so much time. Theodore Mitu connected the dots between Genie 3, which we'll talk about later in the main episode, and said between this, the GPT Onslaught, and the new 11 Labs music tool, the entire legacy creative industry is cooked. For my part, I'm going to be super interested to see and watching very closely how much 11 Music actually gets used for this commercial use case, given that theoretically that's been possible with things like Suno before, but hasn't really happened. Maybe the way that they train this model will open up that use case, but we will have to wait and see.

4:33Now, another model launch, which actually happened all the way back in the olden times of Monday, was Lindy 3.0. Lindy 3.0, they're pitching as their biggest step ever towards their vision of the AI employee. Now, if you haven't tried Lindy, they offer an AI agent platform that aims to make the agent building process simpler. Their last big release was Agent Swarms, which allowed use case-like sales management agents to carry out dozens of tasks in parallel. Lindy 3 has three big new features that bring the platform up to speed and push the state-of-the-art forward. CEO Flo Cravello wrote, Our vision has always been the AI employee.

5:06As capable as humans, can do anything on a computer, and as easy to use. Just ask. 3.0, he says, takes three giant steps in this direction. Agent Builder, Autopilot, and Team Collaboration. Agent Builder is exactly what it sounds like and is, I think, very much the future of how agent models are going to work. Flo writes, just type what you want and it builds it for you in minutes. Now, this is a huge UX improvement for non-technical people. If you've ever fiddled with N8N or Lindy and immediately closed it because you were confused about nodes and about these charts and drawing lines from one function to another, this is a lot closer to what you probably are looking for, which is more akin to vibe coding for agents.

5:45A vibe coding for agents platform would be one where you didn't need to know the details of agent architecture, you just had to describe what you wanted to automate. Now what is a little bit interesting about the UX for Lindy's agent builder is that it still exposes the chain of steps even though it's creating them for you, which gives you more granular fine-grained control to modify it to the extent that the agent builder gets it wrong. That could end up being the right combination, especially for power users who want a little bit more control. Autopilot is Lindy's version of computer use. Flo again writes, Autopilot takes us closer than ever to can-do-anything.

6:18Lindy agents can now work with their own computers in the cloud. He also continued, When we say anything, we mean it. We found out after building Autopilot that now Lindy could also build fully functional websites, deploy them, and even QA them using its browser. We accidentally built Lovable. In dogfooding the product, Lindy found that it was useful for the repetitive work that could be automated in the background. Flo told TBPN, One of the most insane things it's automated for us is replacing the QA engineer. Every hour, a Lindy agent wakes up, tests the entire core flow, and if anything goes wrong, it pings the on-call engineer.

6:49Now lastly, team collaboration does what you'd expect, allowing your team members to natively share and iterate on their agents. I think what's most interesting to me in some ways, aside from just the utility of this, is the move to the vibe coding for agents kind of UX. We've seen some of this already with Emergence recently releasing their agents creating agents platform, but it's hard for me to imagine that this doesn't just become totally standard in de rigueur very soon. Now, lastly, one more little model announcement before we get to our main episode. Alongside Genie 3, Google also announced Storybook.

7:18Their Gemini app account writes, it's storytime reimagined. Now you can create personalized illustrated storybooks about anything complete with read aloud narration. And this is kind of one of those things where the capabilities aren't new, but it's just an interface designed to specifically get at a particular use case. Now, one of the things that's very notable to me is that when parents experiment with AI for the first time, a huge default use case is around creating stories for their kids, illustrating their kids' visions with image generation tools, basically bringing the magical things that happen in kids' brains to life with technology.

7:52It's the convergence of the magic of childhood with the magic of technology. And that definitely seems to be what was going on with the storybook. Joel, who's a PM at DeepMind, wrote,

8:20Now, of course, in a week of crazy open-source releases and world simulation models, maybe this seems a little bit small, but I wouldn't be surprised if for a lot of you parents out there, The coolest new thing that got released in the short term might just be Google's Gemini storybook. With that, though, we are going to close out the headlines. Next up, the main episode. This episode is brought to you by Blitzy, the enterprise autonomous software development platform with infinite code context. Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise scale code bases with millions of lines of code.

8:53Enterprise engineering leaders start every development sprint with the Blitzy platform, bringing in their development requirements. The Blitzy platform provides a plan, then generates and precompiles code for each task. Blitzy delivers 80 % plus of the development work autonomously while providing a guide for the final 20 % of human development work required to complete the sprint. Public companies are achieving a 5x engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding co-pilot of choice to bring an AI-native SDLC into their org.

9:22Blitzy is providing a limited-time, 30-day free proof-of-concept for qualifying enterprises. The team will provide a 5x velocity increase on a real development project in your org. Visit Blitzy.com and press Book Demo to learn how Blitzy transforms your SDLC from AI-assisted to AI-native. That's B-L-I-T-Z-Y dot com. As a founder, you're moving fast towards product market fit, your next round, or your first big enterprise deal. But with AI accelerating how quickly startups build and ship, security expectations are higher earlier than ever. Getting security and compliance right can unlock growth or stall it if you wait too long.

9:59With deep integrations and automated workflows built for fast-moving teams, Vanta gets you audit-ready fast and keeps you secure with continuous monitoring as your models, infra, and customers evolve. Fast-growing customers like Langchain, Writer, and Cursor trusted Vanta to build a scalable foundation from the start. And look, as someone who lives in the world of enterprise procurement, I love how Vanta makes it easy to get compliance right. The last thing you need when you're trying to win that big deal is to have it scuttled by something that Vanta has solved for over 10 ,000 companies. Go to vanta.com slash NLW to save$1 ,000 today through the Vanta for Startups program and join over 10 ,000 ambitious companies already scaling with Vanta.

10:36That's V-A-N-T-A dot com slash NLW to save$1 ,000 for a limited time. Today's episode is brought to you by Plum. Are you building with AI? Plum noticed that every technical creator tends to hit the same wall. You've got AI workflows people want, but monetizing them feels impossible because client work doesn't scale. Selling copies gives away your IP. And building your own platform? That's becoming a software company. It's a hard gap to bridge, and that's why they built Plum. Plum helps creators build an audience of paid subscribers for their AI workflows, all on a single platform. Think Substack for automations.

11:15There's no need to build extra infrastructure just to get paid for your expertise. Plum handles that so creators can do what they do best, solving problems with AI. Ready to turn your expertise into passive income? Visit useplum.com. That's Plum with a B. Welcome back to the AI Daily Brief. Yesterday, there were many breathless posts about the incredible progression and performance of OpenAI's new open source model. Twitter was absolutely flooded with posts like this one from Dharmesh, the founder of HubSpot, screaming excitedly about how they were running this model on their MacBook Pros. And of course, as you heard yesterday, there was a ton of excitement about the reported benchmarks.

11:55But inevitably, what happens is in the first 24 hours or so after a model gets released, people actually dig in. They start putting it through its paces, running their own benchmarks, and generally trying to get a vibe off the thing to understand how good it is in practice. Sometimes that leads to a model that outperforms its benchmarks. Other times, in fact, I would say probably more often, the shine gets a little bit worn off. Now, in the context of this model, where it's open weights and there's more information to play with, there's even more hacking to be done. So the question we're exploring on today's show is how people are responding and what vibe they're getting from it now that it's out in the wild.

12:32Initially, a lot of the responses I saw were somewhat similar to this one from Victor Talin. They wrote, My initial impression of OpenAI's OSS model is aligned with what they advertised. It does feel closer to O3 than to other open models, except it is much faster and cheaper. It's definitely smarter than KimiK2, R1, and Quen3. I tested all models for a bit and got very decisive results in favor of OpenAI OSS 120b. The independent benchmarkers didn't find quite as high performance as OpenAI reported, but it was still pretty good. Artificial Analysis writes, Independent benchmarks of OpenAI's GPT-OSS models, GPT-OSS-120b is the most intelligent American open weights model, comes behind DeepSeek R1 and QEN3-235b in intelligence, but offers efficiency benefits.

13:16You can see here that across the Artificial Analysis Intelligence Index, the 120b model scores a 58, as opposed to R1's 59 and QEN3's 64. Now, O3 is up at 67, so there's a fairly big jump from 120b to 03. This question of how much efficiency matters is something we'll come back to, however. And yet, if some of the initial impressions were positive, it didn't take long before you started seeing posts like this one. Ahmad Moustak, formerly of Stability AI, writes, Is GPT OSS entirely mid-trained or something? It's good for its size and MacBook, but kind of feels fried and odd at times. Mark Kretschmann responded, with a theme that we'll see over and over again, Very jagged model.

13:57Feels uneven. Celeste writes, It feels like synthetic data. It's been safety maxed. Dwayne writes, I was excited for it until I really started to use it, and it's weird. There's something off about it. Almost like they trained it on safe responses from 03, or maybe early checkpoint of GPT-5. Eevee writes, Yeah, it has strange vibes. Reminds of Microsoft crappy models like Fee 4. OpenCode's Dax wrote, Everyone legit I know is having a not good time with GPT-OSS so far. It's useful because now when I see popular accounts say, it's so good, wow, I know they are BSing. Now, when people tried to dig in with DAX, it sounded like part of the issue was specifically with tool calling, which obviously really matters because as we talked about yesterday, to the extent that this is going to be useful as an alternative for agent builders who want a higher level of security, privacy, and controllability, tool calling functionality is going to be extremely important.

14:45And pretty soon this became the popular narrative. Nikhil Chandak wrote, What it is? State-of-the-art at its size. What it is not, O4 Mini, as many people have been claiming. Sam Pace writes,

15:06And indeed, this is one of the biggest themes that's going around AI Twitter right now, is that this model seems to have been really focused on some particular use cases and not on others. Bjorn Pluster wrote a long thread about how GPT OSS 120b is, quote, very blatantly incapable of producing linguistically correct German text. He writes, I see this as an exceptional release highlighting OpenAI's willingness to contribute to the open model space and showing how strong they actually are on model training. But it is also very clearly a model not up to their usual standards with regard to multilinguality or output quality.

15:38Umbilicus writes, I hate to be the bringer of bad news, but this new OpenAI model is brutally locked down. Danny Aziz from Every writes simply, OSS 120B is not O3 level in my opinion. Kyle Corbett writes, GPT-OSS may have been trained primarily on synthetic data, similar to Microsoft's fee models. As a result, it's extremely spiky. Great at the tasks it's trained on, really bad at everything else. This was almost certainly to avoid copyright lawsuits, sadly. Lisano Guy writes, GPT-OSS models seem to be slot maxed on math, coding, and reasoning. They're great at that, but they completely lack taste in common sense.

16:10At least that's my vibe so far. Phil111 on Hugging Face left a comment called this model is unbelievably ignorant. He said this model has about an order of magnitude less broad knowledge than comparably sized models like Gemma 3.27b and Mistral Small 24b. This model, including its larger brethren, are absurdly ignorant of wildly popular information across most popular domains of knowledge for their respective sizes. What's really confusing is all of OpenAI's proprietary models, including their tiny mini versions, have vastly more general and popular knowledge than these open models. So they deliberately stripped the corpus of broad knowledge to create OS models that can only possibly function in a handful of select domains, mainly coding, math, and STEM, that over 95 % of the general population doesn't give a rat's ass about.

16:51Conveniently making it unusable to the general population and in so doing, protecting their paid ChatGPT service from competition. Now, let's say for a moment that there were specific decisions that went into this that made it optimized for those fields. Phil in the comments here is suggesting that that's them trying to protect the integrity of their business, which as an aside, would be a reasonable thing to do anyways. But I'm not exactly sure that the analysis is correct, even if the interpretation of what's going on under the surface is. As we discussed yesterday, the most likely users of this are people who have perceived data security and privacy needs that are significant enough that they want to use a less convenient, not fully state-of-the-art open source model over something that has any sort of controller interaction from a third party.

17:32And if that is the case, those folks almost certainly aren't using this for writing poetry or for getting access to general domain knowledge. They're using it, in other words, for coding math and STEM. Now, we have no confirmation yet from OpenAI that it is optimized for those fields. But even if it were, it would kind of make sense to me just on that use case level alone. But what about this question of really how it compares to the Chinese models? Just a day before this was released, A16Z's Martin Casato wrote, It's just remarkable how many US startups are being built on Chinese OSSAI models.

18:03I'd say the majority that are building custom models via post-training. The U.S. needs to step up, make it a national priority, and back with a huge investment. Which made it all the more interesting that the White House's Michael Kratios popped in yesterday, retweeting Didi Das from Menlo, who was talking about these three big model releases, and saying people ask what we mean by unquestioned and unchallenged global technological dominance. Simon Willison made this comparison directly in his tests. He wrote, I've been writing a lot about the flurry of excellent open-weight models released by Chinese AI Labs over the past few months.

18:35All of them very capable and most of them under Apache 2 or MIT licenses. Just last week I said, Something that has become more undeniable this month is that the best available open-weight models now come from the Chinese AI Labs. I continue to have a lot of love for Mistral, Gemma, and Llama, but my feeling is that Kwen, Moonshot, and Z.ai have positively smoked them over the course of July. I can't help but wonder if part of the reason for the delay in release of OpenAI's OpenWeights model comes from a desire to be notably better than this truly impressive lineup of Chinese models. Simon continued, With the release of the GPT-OSS models, that statement no longer holds true.

19:07I'm waiting for the dust to settle in the independent benchmarks that are more credible than my ridiculous tests to roll out, but I think it's likely that OpenAI now offer the best available OpenWeights model. Ramjad didn't find that, though. He writes, I tested OpenAI's brand new open source model against KimiK2 and Qen3 Coder. It seems to have performed worse than the Chinese models in one-shot tasks. Somewhat disappointing to see, but I think over the coming days, people will find good uses for it within coding. Dax again writes, for coding, they're not anywhere close to the Chinese models in the last few months.

19:35We'll give it time for dust to settle, but simple evaluations aren't enough. The Sun Al-Gaib writes, there is no Western open source model that beats or ties the best Chinese open source models. And adding even more questions to this is Ethan Malek, who retweeted Simon sharing the Chinese competition section of his post with Ethan writing, the US now likely has the leading open weights model or close to it, but the real question is whether this is a one-off situation from OpenAI, in which case the lead will evaporate quickly as others catch up. Unclear what their incentives are to keep updating.

20:06And this, of course, is a challenge. Let's say that the model isn't quite as good. Even this week, this is not the main announcement from OpenAI. This is just the amuse-bouche for GPT-5, which we think is coming tomorrow. So if we really want to compete when it comes to Open Weights models, it's not clear that OpenAI alone is going to do that. Now for a little bit more of an optimistic take, which is really just realism as optimism, Nathan Lambert wrote, seems like while this launch had the vibes right and OpenAI can jazz up a crowd, they're still going to go through a lot of the pains of why open models are hard.

20:39Just so many weird little failures I'm seeing people mention. May take a few weeks to get the best out of GPT-OSS. MatrixMemories responded, the pain and beauty of open source. You can't run away from the edge cases or control for them. So of course, the optimistic take there is that part of what makes these open models valuable is what the community can do with them. And so we may not want to judge it based on these first blush impressions. Nathan actually wrote a very long post on his Interconnects.ai blog, where he argued that OpenAI validates the open ecosystem, and that, quote, open models from the US labs were in such a dire spot that we need any step back in the right direction.

21:15But however, when the question becomes, is OpenAI the new Open champion, his answer is not quite. Nathan writes, it's a phenomenal step for the open ecosystem, especially for the West and its allies, that the most known brand in the AI space has returned to openly releasing models. This is momentum and could be the start of the turning point of adoption and impact of open models relative to China. But he writes, there's a lot of uncertainty in the incentives for open models. Some of the best China analysts I know share how China is sensing that releasing open models is a successful strategy for them and they are doubling down.

21:44OpenAI's release is a step in the right direction, but it's still a precarious position. Many people are making noise about creating open models, from the AI action plan to venture capitalists and academics. What all of these parties have in common is that it is not their number one goal. Now, Nathan does have a new initiative, which does have that it is number one goal called the Atom Project, and which we'll likely get into later in the week. But this sort of validates the point that Ethan Mollick was making as well. There is also lastly this question of speed efficiency and cost. One of the less talked about aspects of this thing so far is that these models are extraordinarily fast and extraordinarily cheap.

22:18Most of the research has suggested that at this stage, everyone's just using the best model, whatever it costs, but that won't necessarily be the case forever. And it won't necessarily be the case as we get more complex workloads that just consume a boatload of tokens. So all in all, I would say that the shine is slightly diminished from where it was yesterday, but that still most people are very excited to have OpenAI back in the open game, going to continue to dig in and see how much can be wrenched out of these models before they're abandoned in any sort of hugging face model junk heap. As you know, however, GPT-OSS was not the only model launch yesterday.