Is OpenAI Secretly Training GPT-5?

0:00Today on the AI Breakdown, we're looking at recent comments from an AI industry leader about whether or not open AI is secretly training GPT5. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown.net for more information about our newsletter, our Discord, and our YouTube. Welcome back to the AI Breakdown. Well, it is a holiday here in America. It's Labor Day for those of you who are not in the U.S. And so at first I was thinking maybe I'll finally take a day off. But there has been this point of speculation, of rumor, of innuendo that has been bubbling for a while, but got a big update just before the weekend.

0:40And I think that given the significance to the industry of this particular question, it was worth doing a little bit of an exploration of, and a long holiday weekend seems like the perfect time to do it. The question I'm referring to, of course, is whether OpenAI is training GPT-5. And before we get into the new context for that question, let's go back and give a little bit of context for it. Now, almost as soon as GPT-4 was launched, rumors came out that OpenAI was already developing GPT-5. Sam Altman, the CEO of OpenAI, went to pains to say that that just wasn't true. The Verge writes about Altman's appearance at an event in MIT at April, where he was queried about the open letter that asked for a six-month pause for all AI development of systems that were more powerful than GPT-4.

1:25From The Verge, Altman said the letter was, quote, missing some technical nuance about where we need the pause, and noted that an earlier version claimed that OpenAI is currently training GPT-5. Quote, we are not and won't for some time, so in that sense it was sort of silly. Okay, so that was April. The next month in May, Sam Altman appeared before the Senate for the first hearing on AI post-ChatGPT, and once again reiterated that OpenAI was not training GPT-5 at that time. Now at that point, we started to get a sense that this might be not just for technological reasons, but also for regulatory reasons.

1:57One of the ideas that seemed to be coming to the fore in Altman's testimony was the idea that for models that were more advanced than GPT-4, there might need to be some sort of licensing regime or approval process. And so given that that was sort of where the winds were blowing, perhaps OpenAI was thinking that it didn't really make sense to walk down the GPT-5 path without that process becoming a little clearer. In June, Altman once again reiterated that OpenAI was still not training GPT-5. At a conference hosted by the Economic Times of India, he said, We have a lot of work to do before we start that model.

2:30We're working on the new ideas that we think we need for it, but we are certainly not close to it to start. Now, since then, we haven't heard anything directly from Altman. But of course, no update doesn't necessarily mean that they have now switched and started working on it. It could just mean that we haven't had an update or that Altman hasn't wanted to repeat himself yet again. What did happen in the meantime is that ChatGPT released its Code Interpreter feature, which they've subsequently renamed because the name never really made sense. But basically, Code Interpreter was one of two plugins that OpenAI was working on themselves, the other being an internet browser.

3:01And in many ways, both of these new features, one, the ability to browse the web, and two, the ability to write code, which in many ways allowed GPT to expand the capacity of things that it could be good at by giving it the ability to write scripts to help it with certain problems. Many thought that by and large, when you combine these two things, or even independently, they add up to a model that is significantly more advanced than GPT-4. Swix from Latent Space Podcast very prominently called code interpreter GPT-4.5, and others have had a similar analysis. What's more, people speculated that the reason that OpenAI wouldn't just call this GPT-4.5 was that regulatory concern.

3:37Another update that happened a little bit after Code Interpreter was released was the release of ChatGPT Enterprise. In addition to having Code Interpreter integrated, one of the things that ChatGPT Enterprise offered was unlimited higher-speed GPT-4 access. Some, such as Jan Peleg here, took that as an indication that they no longer needed to constrain their compute, and that the most logical reason that they would no longer need to restrain their compute was if they had already finished training GPT-5. So this is where the state of the conversation was heading into the end of last week. But then we got interesting comments from Mustafa Suleiman.

4:14Now, Suleiman is, of course, the previous founder of Google DeepMind, and now the CEO of Inflection AI, but he's also promoting his forthcoming book about AI, and so has been on something of a media tour. In a conversation with Robin Wiblin on the 80 ,000 Hours podcast, this is what he had to say about GPT-5. When asked if he believed that they were secretly training GPT-5, he said, I don't know, you have to ask them. I like them very much and I have huge respect for them, so I don't want to say anything bad if that's what they've said. But also, I think Sam Altman recently said they're not training GPT-5.

4:44Come on, I don't know. I think it's better that we're all just straight about it. That's why we disclose the total amount of compute that we've got. And obviously you can work out from that, roughly speaking, what order of magnitude of flops we're using. It's much better that we're just transparent about it. We're training models that are bigger than GPT-4, right? We have 6 ,000 H100s in operation today, training models. By December, we will have 22 ,000 H100s fully operational. And every month between now and then, we're adding 1 ,000 to 2 ,000 H100s. So people can work out what that enables us to train by spring, by summer of next year, and we'll continue training larger models.

5:15And I think that's the right way to go about it. Just be super open and transparent. I think Google DeepMind should do the same thing. They should declare how many flops Gemini has trained on. So a lot that is packed in there, right? One, he basically intimated that it feels very unlikely to him that OpenAI wouldn't be training GPT-5. Two, he's basically saying that even if they're not, everyone else who has the compute is trying to get out ahead of where the models are now. In other words, to the extent that OpenAI has voluntarily paused, their competitors certainly are not. The AI safety memes account also pulled some other related quotes from the conversation.

5:50One, they wrote, we're going to be training models that are 1000x larger in the next three years. Even inflection will be 100x larger than current frontier models in the next 18 months. Now, the response on Twitter to people posting about this was basically incredulous. I did a poll asking if people thought that OpenAI was training GPT-5, and while far from scientific, 65.5 % said absolutely, 26.7 % said most likely, and only 7.8 % said not yet. And reinforcing the other part of Mustafa Salimun's statement, basically that competitors were training GPT-4 plus size models even if OpenAI wasn't. On August 25th, Jason at AGI Koala wrote, Overheard at a Meta Gen AI social.

6:29We have the compute to train Llama 3 and 4. The plan is for Llama 3 to be as good as GPT-4. Response, wow, if Llama 3 is as good as GPT-4, will you guys still open source it? Yeah, we will. Sorry, alignment people. Altman Sam tweets, Meta wants to open source a GPT-5 level model and seems dead set on open sourcing right up until AGI. I want to be clear about what this means. One, there is no kill switch. If something goes wrong, an agent gets out of control, or a bad actor weaponizes it, there's no easy way to turn it off. Two, safety research becomes meaningless. All the work people have done into making AI systems honest, aligned, ethical, etc.

7:03becomes mostly moot. The population of AIs out in the world will evolve towards whichever systems produce the most economic output, irrespective of what values or motives they have. Now, I'll save the open source debate for another conversation. I think the point here is this sort of affirmation, at least anecdotally, and we should have the caveat that this is very anecdotal, of another competitor that is racing to get ahead of GPT-4 level capacities and is, at least from that anecdotal evidence, still planning to release it publicly as open source. Now, just to put a fine point on how this competition could lead companies to act differently than they might otherwise, another sub-theme from last week was a little bit of a back and forth between a blog that claimed that Google's Gemini would smash GPT-4 and Sam Altman, who intimated that that blog was just doing Google's PR work for them.

7:49Summing it up, Insider writes, AI bros are at war over declarations that Google's upcoming Gemini AI model smashes OpenAI's GPT-4. Insider writes, It's never fun for a CEO to hear their product might get thrashed by a competitor. That might explain OpenAI boss Sam Altman's defensive response to a post published over the weekend titled, Google Gemini eats the world. Gemini smashes GPT-4 by 5x, the GPU pours. Altman's tweet said, Incredible Google got the semi-analysis guys to publish their internal marketing and recruiting chart, lol. Insider continues, Much of the analysis, which the author said was based on data crunch from a Google supplier, boils down to Google having access to infinitely more top-flight chips and its model outdoing GPT-4 on a performance measure relating to computer calculations known as flops.

8:30Now, as many people pointed out on Twitter, it is up for debate around how much just having more chips means better models. Ex-user Technuium said, I hope someone dethrones OpenAI soon, but to say Gemini smashes GPT-4 by 5x makes it sound like it's 5x better than GPT-4. It's not, it's 5x compute. Not necessarily correlated to quality. A Hacker News user wrote, Computational power alone is not the only resource. It is also the training process itself and obviously the data and its quality. I will be convinced only after Google demonstrates that Gemini is better than GPT-4 in some or all tasks. So where does this all net out?

9:04A few things. First of all, it's very clear that whatever OpenAI is doing, their competitors are not standing still. Meta appears to be proceeding with Llama 3 and 4. Google is certainly banking on Gemini to put them back into this race in a big way. And I think that that word race is really relevant here. Remember, one of the reasons that Jeffrey Hinton left Google was that he was concerned that the economic incentives of competition around these foundation models were already starting to make companies act differently and with less responsibility than they might otherwise have. To the extent that OpenAI isn't training GPT-5, they might find themselves in the very unenviable position of being behind where they once were ahead.

9:46Will that make them think differently about safety and alignment issues? It's hard to say. But the pressure is certainly going to be there. Now the flip side, of course, is that as interest in AI has felt on a lower ebb over the course of the summer, one of the things that people anticipate bringing lots of people back is expanded capabilities of future models. Avi Schiffman posted a chart of what people use ChatGPT for, 29 % programming questions, 23 % education questions, 21 % content questions, 13 % sales and marketing questions, and asks, I'm curious how a multimodal GPT-5 will change this.

10:20As with anything in this space, there is a lot to be excited about as well as trepidatious about, but for now, that's the story from here. Thanks for listening or watching as always, and until next time, peace.

10:34Thank you.