GPT-5: Everything You Need to Know

0:00After literally years now of waiting, GPT-5 is here. Is it a GI? Is it a disappointment? We are going to get into all of that on this special episode of the AI Daily Brief. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

0:24All right, friends, back with a very big episode of the AI Daily Brief here. Before we get into that, quick announcements as always. First, thank you to the sponsors of today's show, KPMG, Blitzy, and Superintelligent. to get an ad-free version of the show, go to patreon.com slash ai daily brief. And if you are interested in sponsoring the show, you can reach out to sponsors at ai daily brief dot ai. But with that, let's dive right into the meat of this, because it was a big day. It is not hyperbole to say that this has been years in the making. ChatGPT4 was the dominant model for so long. I mean, almost throughout basically all of 2023, it was the undisputed champion.

1:03And throughout most of the first three quarters of 2024, it was really just everyone catching up and putting out their GPT-4 class model. Now, things started to change a little bit, of course, in the fall of 24 when we got 01 in the advent of reasoning models. This year, we've seen continued progress on reasoning models. For many tasks, OpenAI's 03 model is the main model they use, but then we've also got the insurgency of Chinese open models, DeepSeq's R1, for example. And in the realm of coding, which is very germane to our story today, we've had the incredible dominance of Anthropic. Ever since Claude 3.5 Sonnet through 3.7 Sonnet, Opus 4, and now Opus 4.1, Anthropic has had the go-to coding model for a huge number of people.

1:47Now, Gemini 2.5 has also been up there in contention, particularly because Google has done a really good job competing on cost as well as performance. And while some people used O3 for coding, it really has been the Anthropic models that have been completely dominant. From what we heard coming into today's announcement, one of the big leaps for GPT-5 was going to be around coding. And certainly that emphasis was borne out in the presentation. In fact, I would go so far as to say that if you were just watching this presentation to get up to speed on where the big labs thought the AI competition was, I'm not sure that you would believe that there is any use case that OpenAI cared about except coding.

2:26We're going to talk a lot more about that, but first let's go quickly through what they announced, get some first impressions, talk about some other non-coding use cases, look at the benchmarks, and then we'll get into the meat, which is, surprise, surprise, all about the advent of vibe coding, the democratization of code-based creation, and the GPT-5 future that OpenAI is laying out for us. As you would expect, there are actually a few different models that were announced. GPT-5, GPT-5 Mini, GPT-5 Nano. They all have 400k context lengths and really competitive costs, which is something we'll get into in a little bit as well.

2:57From an availability standpoint, these models are rolling out right now to top end users and will be available to education and business users next week. The presentation was about an hour and 20 minutes long. And to some, the energy was a little underwhelming. Signal writes, This launch felt like attending a funeral hosted by minimalists. They're unveiling tech that should feel magical, real breakthroughs, but the whole vibe was grayscale grief. Even the storytelling arc, chart styles, then eulogy tributes, then closing on someone's health battles. What exactly are we as the audience mourning?

3:26Potentially great product, sure, but the emotional tone was so d***y away. Incredibly strange all around. Now, of course, Elon waded right into that conversation, adding underwhelming to it. And while I might not have felt the same as Signal, it definitely felt less energetic than it should have given the magnitude of what was being launched. Still, if there was one thing that the denizens of Twitter slash X really latched onto, it was this particular chart showing off their performance on the sweet bench. I don't think I've ever seen a chart from an AI launch be shared so many times with so much gasping disbelief.

3:58For those of you who are just listening, not watching, it's a chart of GPT-5's performance on the SweetBench Verified compared to 03 and 40. And the chart has 40 and 03 at exactly the same height in the chart, but with one scoring 30.8 % accuracy and one scoring 69.1 % accuracy. And then for GPT-5, it has a with thinking and without thinking score, 52.8 % without thinking, 74.9 % with thinking. But for some reason, the 52.8 % percent is above the 69.1 percent of 03. It's just a real aneurysm-inducing chart. Now, that said, we're sharing these things for the sake of completeness of the reaction.

4:36We're in the moment. Relatively speaking, if you've got a couple people who say that the mood was more dreary than it might have been, and a lot of people sharing a single chart they have a problem with, and that's the biggest critique that you're getting, you're in pretty good shape. Now, as you know, when it comes to understanding how good a model is, I tend not to care all that much about the benchmarks, especially the ones that are pretty saturated up near the top. But for the sake of completeness, we should talk about them at least a little bit. Now, one really interesting note is that on their research website and in the presentations, OpenAI chose not to compare their performance to any other models except OpenAI models.

5:13Whereas every other company is comparing themselves to Grok and to Gemini and to OpenAI and to Anthropic, OpenAI is only comparing GPT-5 to past OpenAI models, which I get as a statement to keep the focus on OpenAI, but it does sort of have the feel of not wanting people to see how it compared to other models, which doesn't really make sense because as expected, GPT-5 crushed a lot of these benchmarks. For example, on Humanity's last exam, GPT-5 with no tools got 24.8 % compared to O3's 14.7 % with no tools, while its GPT-5 Pro with full access to tools got 42%. On coding, you heard about the Sweebench verified.

5:55On instruction following and agentic tool use, it scored really well, which is something that's going to come up in a little bit. And overall, across the stats that they share, GPT-5 just performed really, really well. You remember I've talked a little bit about their internal benchmark for economically important tasks. The really interesting stat that they had shared previously is that ChatGPT Agent with full access to tools did comparable or better to humans in roughly half the use cases, and GPT-5 was right around that as well. In fact, GPT-5 did even a little bit better than ChatGPT Agent.

6:26But what about independent benchmarks? One of the more comprehensive independent benchmarks comes from artificial analysis, and the TLDR is that their high-end version of GPT-5 is now the highest performing model across their tests. You can see here that GPT-5 High and GPT-5 Medium are at 69 and 68, both very slightly above Grok 4. For those of you who are interested in going deeper, you can go on Artificial Analysis website to see where GPT-5 fit across all their eight different measures. They were right at the top of many of them, including the MMLU, Humanities Last Exam, the AIME, and the AALCR, or Long Context Reasoning, which is a benchmark directly from Artificial Analysis themselves.

7:05In fact, Long context reasoning stands out as one of the areas of the biggest gains for GPT-5, which matters because that test is really all about what a model can do in an agentic context. Speaking of that, Meter, whose chart I've shared numerous times on how long a model can successfully complete tasks for, and GPT-5 is now at the top of the pack. Now, this is the metric that maps how long of a task a model can be successful at with a 50 % success rate, and GPT-5 pushes that up to about 2 hours and 15 minutes. On LM Arena, GPT-5 debuts as number one. This comes from its testing period when it was under the codename Summit, and it now is above Gemini 2.5 Pro, O3, GroK 4, Quen 3, and a number of other models.

7:47One area where it kind of seemed to underperform is on the ARC AGI test. On both ARC AGI 1 and especially on ARC AGI 2, GPT-5 did well, but it was meaningfully behind GroK 4. At the same time, when you start to factor in efficiency, GPT-5 Mini does really well. In fact, Greg Kamrat, the president at ArcPrize, says that it appears to break the current ArcGIS frontier. A couple areas where OpenAI also clearly tried to put some emphasis was on hallucinations and sycophancy. On hallucinations, at least according to their own reporting, GPT-5 is way, way lower than O3 and O4. Simon Willison called out that when it comes to sycophancy, they've tried to build solves into the core model.

8:29He quoted their research post, which said, For GPT-5, we post-trained our models to reduce sycophancy. Using conversations representative of production data, we evaluated model responses that assigned a score reflecting the level of sycophancy which was used as a reward signal in training. So a bunch of sort of day-to-day improvements as well. But what about when it comes to the big use cases? There were three that OpenAI mentioned in their presentation. And really, as we'll discuss, only one that felt like it really mattered. So, the three use cases that OpenAI mentioned were health, writing, and coding.

9:01On the health front, they talked about a number of benchmarks, but they actually humanized this one and made it about much more than benchmarks and much more instead about a bigger change. Wrote one account on Twitter, OpenAI has brought out a cancer survivor to share her story that she had to use ChachiBT to help her advocate for herself and fight for her cancer and be able to challenge the doctor's opinions to make a better decision. This is bound to ruffle the medical industry and doctor's feathers. We foresee a battle incoming here between medicine and AI. Doctors were already annoyed by patients googling their symptoms before they came in.

9:31Imagine now when patients say, but my AI said, and don't trust their doctor's judgment. There's going to be a tension. And yes, of course, doctors can be wrong and AI can be useful and save lives. This isn't a value judgment, it's an observation of an incoming tension. Now Elon waded into that one as well and said, AI is already better than most doctors, that's the honest truth. And it will become far better. Same for all jobs, to be honest, including mine. Now, my strong perspective on this one is that while, yes, undercutting and undermining every decision that a doctor makes because what an AI said isn't the right approach, in general, a more informed patient base, while something that doctors will have to adapt to, is going to be a net positive.

10:10People should have more information. They should have access to more resources. And so often when it comes to medical decisions, all doctors can do is give you the best set of information and leave the decision to you. That was the situation in this particular case, where the patient wasn't sure what choice to make about a particular course of treatment that had a bunch of severe consequences and potential side effects. Now, we're not going to dwell too much on this show, but it was notable that this is a use case that they are really emphasizing as something that a lot of people are turning to ChatGPT for.

10:40The next two use cases bring it back, of course, to the business world. And the first we're going to talk about is writing. Dan Shipper and the crew of Every had a chance to test GPT-5 for a few weeks and made this handy-dandy chart where they gave a simple yes or no for a handful of key tasks, including day-to-day tasks. That got a yes. Pair programming. That also got a yes. Agentic engineering. That got a no. We'll come back to that. And writing was almost a split decision. For writing, it got a yes, but for editing, it got a no. So on the positive side, when it came to writing, they said, GPT-5 has a good voice, nuanced and expressive.

11:13It's less likely to output obvious AI idioms. So it's the first thing we turn to when we have a sentence we need to polish or a paragraph we need to draft. We sometimes return to GPT 4.5 for questions that require more thought. Interestingly, however, they say when it comes to editing, it's a no-go. For editing, they write, GPT-5 cannot determine whether writing is good. We have benchmarks to test AI's ability to judge writing, and GPT-5 consistently fails on tasks that Opus 4 passes. Latent Space also published a first look that we'll come back to, and they actually had a much more negative opinion.

11:44They wrote, While GPT-5 continues to work its way up the software engineering ladder, it's not really a great writer. GPT-4.5 and DeepSeek R1 are still much better. They shared a couple of examples of using GPT-5 to rewrite some LinkedIn posts, and they felt like the GPT-5 answer was more sloppy than the GPT-4.5 answer. In my life, I have a pretty strict dividing line between types of writing that I'll use AI for and types of writing that I won't. And pretty much in the column I won't is everything other than podcast descriptions and really basic stuff like that. So as I get my hands on GPT-5 in the coming weeks, I will certainly be testing to see whether this breaks out of that, but at least for right now, I'm not holding my breath.

12:23And yet, if that all feels like prelude, you're not wrong. During the presentation, I tweeted, With GPT-5, OpenAI is making an extremely loud argument that there is one singular AI use case that matters. Can you guess which one? Yes. When push comes to shove, this presentation was entirely about coding. OpenAI certainly tried to make the argument that GPT-5 was now the undisputed king for coding. They had, for example, Michael Truel, the CEO of Cursor, come up during the presentation and say in no uncertain terms that GPT-5 was the smartest coding model they've tried. But coding-like writing is about so much more than benchmarks.

13:02It's about vibes and experience. And as we've seen so many times over the last few months, There will be a model that technically has better benchmarks than one of the Anthropic models, and it just doesn't displace them. AI agents are the buzzword that everyone's talking about. But do you truly understand their significance? KPMG's agent framework demystifies the concept, offering practical steps to unlock AI agents' immense potential. Think of it as your GPS for AI strategy. KPMG partners with clients to harness the benefits of AI agents, guiding you from strategy to execution with a secure architecture and a plan for workforce evolution.

13:42Check out their comprehensive insights on scaling agent power within your enterprise. This isn't just about tech. It's a leadership imperative. Go to www.kpmg.us slash agents to learn more. That's www.kpmg.us slash agents. This episode is brought to you by Blitzy, the enterprise autonomous software development platform with infinite code context. Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise-scale code bases with millions of lines of code. Enterprise engineering leaders start every development sprint with the Blitzy platform, bringing in their development requirements.

14:19The Blitzy platform provides a plan, then generates and precompiles code for each task. Blitzy delivers 80 % plus of the development work autonomously while providing a guide for the final 20 % of human development work required to complete the sprint. Public companies are achieving a 5x engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding co-pilot of choice to bring an AI-native SDLC into their org. Blitzy is providing a limited-time, 30-day free proof-of-concept for qualifying enterprises. The team will provide a 5x velocity increase on a real development project in your org.

14:52Visit blitzy.com and press book demo to learn how Blitzy transforms your SDLC from AI-assisted to AI-native. That's B-L-I-T-Z-Y dot com. If you are a regular listener, you will have heard about Superintelligence agent readiness audits at this point. But I wanted to tell you today about the full suite of agent readiness products that go beyond just the initial readiness report. Over the last six months, Superintelligence has built out an entire agent planning suite. We help you move from discovery to planning to implementation. After you've completed your agent readiness audits, we help you double-click on your most important use cases with what we call our use case planning reports.

15:31These reports are going to help you understand what sort of technical preparation you need to do to be ready for a use case, what challenges you might face in implementation, and whether you should be thinking about building, buying, partnering, or some combination. After that, you can even get a spec document in what we call our technical blueprint that gives either your developers or the developers of the partner you work with what they need to build exactly the agent that you're looking for. If you want to learn more about Superintelligence Agent Planning Suite, we built a custom GPT to answer your questions.

15:59Just go to bit.ly slash super agent. That's bit.ly slash super agent, all one word. And if you have any questions, the agent can even help you book an appointment with our team. Now it's early, but a lot of the early reviews are about coding. So let's go see what people had to say. Turning back to every one more time, they again had a mixed review. They basically said that GPT-5 was a great pair programmer. Dan Schipper wrote, don't get me wrong, GPT-5 is a very good programmer. It's incredibly useful as a pair programmer, especially in AI-powered integrated development environments, IDEs like Cursor.

16:37It's great for engineers from traditional backgrounds who want an AI to help collaborate on code. And it excels at research and debugging complex issues. What Dan argues, however, is that, quote, the discipline of programming has fundamentally changed this summer. The benchmarks don't show it, but if you know how to YOLO four agents at once in Cloud Code, GPT-5 feels like a step backward. That's partially because of the model's current personality. It's more cautious than Opus 4.1 and isn't as comfortable working independently for long periods in our testing. But it's also due to the app you use to interact with it.

17:08Both Cursor and OpenAI's command line interface tool, Codex CLI, are not on the same level as Claude Code. Both were built for programmer AI pair programming, not true delegation. I bet this will change. The model is extremely smart, just not yet built for this use case. But for now, OpenAI seems to have missed the paradigm shift in programming caused by Claude Code over the last two months. Now, I wanted to present that one first, because so far at least, they're kind of the only ones I've seen saying that. Most of the other results are incredibly impressed. Like the team at Cursor, the team at Lovable was incredibly excited about it.

17:42YouTuber and AI educator Matthew Berman wrote, GPT-5 is here, and spoiler, it's a coding master. I've had the privilege of testing GPT-5 for about a week now, and I gave it the most difficult tests I could come up with. He gave it a Rubik's Cube test that only previously Gemini 2.5 Pro had cracked. He asked it to make an Excel clone and a Microsoft Word clone. He asked for a more complex version of the classic 90s phone game Snake, asked it to solve some physics, gave it the now-famous hexagon-with-balls-bouncing-inside test, and a number of others. and was basically blown away by the results.

18:13Now, one thing Berman noted was that GPT-5 is really good at front ends. And again, this is something that I've seen over and over again, is that while there is a general sort of feel and a flavor to a lot of the UI design that you see among AI and agentic coding in a similar way that you see common patterns across AI-based writing, it seems at least from first glance that GPT-5 often breaks out of those patterns. Pietro Serrano shared a compilation of different experiences that he had made with GPT-5 in one shot, coming away extremely enthusiastic. He said the Palm Camera app is particularly impressive because the model came up with all the details, like the way the photos stack in the gallery, the photo developing animation, etc.

18:50He actually made a bold pronouncement. In another tweet, he said, I had early access to GPT-5. It will do for coding what GPT-4 did for LLM adoption. It's fast, really smart, has great taste and aesthetic sensibility. This is electricity arriving in every home, a before and after moment in how we build. GPT-5 is the best coding model ever, but what really impressed me is how good of a collaborator it is. It tackles large implementations much better than before. At times, it was able to refactor thousands of lines of code at once, and also debugs large repos way faster and with precision. It's also, he writes, a really, really good agent that was able to run long agentic flows with no issues.

19:27It's also better at explaining why to run a certain tool versus another, and it has the best results on running parallel tools I've ever seen. This is particularly important for flows where, for instance, you need to create multiple files at the same time and consistency becomes a problem. GPT-5 has no issue with that. Matt Schumer had a really interesting take. He said, For those who will be testing and using GPT-5, you won't see much improvement if you're asking it to do things Sonnet or O3 can already do. To see how good it truly is, you have to ask it to do things that other AIs simply can't.

19:59In his longer review, he expanded this. His TLDR included, The GPT-5 is clearly a big leap from previous models, but you have to push it hard to get the most out of it. The ceiling for what can be vibe-coded is now much higher than it was with previous models. Matt wrote, I was granted access to GPT-5 on July 21st, and honestly, when I started testing it, I wasn't blown away. In fact, I felt quite let down, especially given all the hype and expectations around it. The model felt like GPT-4.2 at best. Faster, definitely sharper than 4.1, but not some huge leap. I tried to use it for my day-to-day work, which in my opinion is the best way to evaluate any new model, and while it handled the tasks I was giving it very well, I wasn't noticing anything dramatically better than GPT 4.1, Cloud4 Opus, or any of the other models I've been using.

20:43I got myself thinking, is this really it? I settled into a routine of using GPT 5 for pretty much everything I would use existing LLMs for, and this went on for about a week. Was it better than Cloud4 Opus, my previous daily driver? Yes, undoubtedly, but only marginally. It felt like a small incremental improvement. But then things took an unexpected turn. Josh, my lead engineer at HyperWrite, and I had spent an afternoon discussing a complex new product idea. One we'd estimate would take weeks, maybe months of dedicated engineering work to even get a proof of concept together. The idea was intricate, involving a sophisticated front end with tightly integrated components and a complex backend infrastructure for managing GPUs, autoscaling resources, and lifecycle management.

21:21This wasn't the kind of thing you just vibe code, even with the help of AI. It required deliberate human oversight at every step. Or so we thought. Josh and I already decided we'd need at least a full month of discovery just to figure out if a buildout was worth attempting. That night, purely out of curiosity, I fed GPT-5 a product spec, fully expecting it to stumble immediately. An hour later, I sent Josh a fully working prototype. His immediate reply, WTF. That moment completely flipped how I thought about GPT-5. We literally skipped a month of upfront customer discovery and planning. We could just immediately go test with real users.

21:55From there, things got interesting fast. I started probing deeper, trying more ambitious tasks that I never even bothered asking previous models. The more I did, the clearer it became that GPT-5 wasn't incremental. Now Matt also talked about this whole front-end piece. He said if you've used AI for front-end before, you probably know what I mean when I say it usually feels made by AI. The designs are typically a bit clumsy, predictable, obviously machine-generated. With GPT-5, though, the UIs felt way closer to convincingly human. 80 % indistinguishable at a glance. On backend and infrastructure, GPT-5 was just as good, maybe even more impressive.

22:28The deeper I went, the more clearly I saw just how different GPT-5 was. Now, he did find that there were specific tasks that he still prefers other models for. He said, for example, for explicit search tasks, I still prefer O3. GPT-5 stops digging sooner. For example, I was trying to have GPT-5 find the hometown of a public figure. It only found the city and stopped there. He also said on emotional or sensitive tasks like crafting difficult emails, I still strongly prefer GPT-4.5. Ultimately, he says GPT-5 is a true leap. Bottom line, GPT-5 isn't just going to improve vibe coding. It will fundamentally change the kinds of projects I consider doable without serious human intervention and steering.

23:03This past week, it turned what I confidently thought was a multi-month engineering challenge into a casual one-hour sprint. This is serious, real, autonomous software engineering. He summed it up on Twitter as well, saying, You can now vibe code real software. Not just simple SaaS apps, but real technical software. And by the way, the Vibe coders seem to agree. Felix from Lovable wrote, GPT-4 was build me a to-do app. GPT-5 is build me a SaaS with user auth, payments, admin dashboard, and email automation. We're not improving code generation. We're eliminating the need to code. Now, one of the most interesting reviews came from Ben Hiluck and the team at Layton Space.