1-800-Chat-GPT, Neuralink’s Potential, Meta's Live AI

0:00OpenAI has an impressive new reasoning model. You can call ChatGPT now too. Is Neuralink as promising as it sounds? Plus some predictions for 2025. That's coming up on a big technology. Podcast Friday edition right after this.

0:43Welcome to Big Technology Podcast Friday edition, where we break down the news in our traditional cool-headed and nuanced format. We have a great show for you today covering the latest in OpenAI, revisiting my Neuralink interview with Nolan Arbaugh. And of course, we're going to get some predictions in for 2025. Joining us as always on Fridays is Ranjan Roy of Margins. Ranjan, great to see you. How are you doing? Great to see you. Listeners cannot see, but Alex is currently in Australia and has a tiny microphone in his hand. And I feel like I'm doing a TikTok influencer type thing right now. That's exactly what it looks like on the screen.

1:18Luckily, we're not going to show any of the video of this. But yes, I'm in Australia, last day of work before some, I think, needed winter vacation. And let me just tell you, the flight here is crazy. You go from winter to summer and from short days to long days, and you're like on the plane for forever. I left on Tuesday and got here on Thursday. So I skipped Wednesday entirely. What's the time difference? 16 hours, no big deal. So I'm actually in Saturday morning and you're on Friday afternoon, which is crazy. How is Saturday looking? What do I have to look ahead for? Honestly, it looks great and quite warm.

1:57So happy about that. Not New York, not New York. We got a little snow here. And so, you know, it means that you're still living in Shipmas where Shipmas is over, where I am. And I just think we should just take this moment to appreciate how amazing your final day of Shipmas is and you get to savor it a little longer than I do. That's the most important part of this, is that I get, I essentially, you lost a day of shipmas. It's rough. Of OpenAI's 12 days of shipmas, Alex, and I'm sorry. I'm sorry for that. But today, today was a big one for shipmas. You saw the news? I did. And before we just get into it, let me just say, we have some new listeners here, so let's just talk a little bit about the show format.

2:39On Wednesdays, we do a big interview, like Wednesday's interview with Nolan Arbaugh. And then on Friday, Ronjan and I come and break down all the tech news. So we do two shows a week, one interview, one analysis, and we're here in the analysis show. And boy, do we have some news to analyze. Because OpenAI, which has been shipping in its shipmas schedule, right, one release every day for 12 straight days, it announced that it has some new really interesting reasoning models called O3. This is from TechCrunch. On Friday, the company unveiled O3. So O3 is this really impressive new reasoning model that was trained to think before responding.

3:14And it's using this method that OpenAI calls private chain of thought. So it basically reasons through a task and plans ahead before it answers, making it a pretty impressive upgrade to the company's large language model family. There's some crazy stuff that OpenAI found out about this. But basically, the big takeaway for me here is that Generative AI is moving towards this reasoning methodology, going from training these models up with more data, compute, and energy, to training them up with the ability to think through problems, go step by step, make sure that the last step was answered appropriately, and then spit out an answer.

3:53I think I'm buying this. I think this really works. And I'm curious what your reaction is to O3. All right. So my first reaction to O3, and this is more on the pedestrian side of things, But I love that they had to skip the name O2 out of potential conflict with British telecom provider O2. So O3 is the successor to O1. But O3, the whole space of reasoning and generative AI, I do think is incredibly interesting. I think we've already been hearing a lot about the limitations around scaling traditional large language models And the way a reasoning model works in terms of actually breaking things down into essentially a chain of actions, a chain of thought, breaking down into steps, being able to validate each step along the way is very interesting if it can do this.

4:45The large language models, the knock on them has always been that they don't actually know anything, that they're essentially just predicting the next letter or pixel, and they don't know if it's correct or not. There's nothing beyond a kind of a superficial presentation of information that looks good and is correct often. So if this is achievable, I think this is important. Google just the other day announced, and this was the incredibly named Gemini 2.0 Flash Thinking Experimental. Only Google could come up with a name like that. But basically every tech giant, everyone in the generative AI space is trying to get into reasoning because if it can be done successfully, if you think about the applications of it, it opens up such a bigger world of possibility rather than just text output, image output, like actually solving real problems.

5:42It at least presents the opportunity to do that. Now, this is a question I think needs to be asked here, which is, do you think that this move to reasoning is just a head fake from everybody working on generative AI? And the reason why I ask that, and it's a question I think that needs to be asked, is because it seems like the traditional methods of scaling, the brute force methods, as they're called, right, which is that you just scale up your data center to 1 million GPUs, as Elon Musk is doing with XAI. you dump as much data as you can, you hire PhDs to write everything they know, and you throw that into the model training, and you hook it up to a nuclear power plant, and that's just been producing better and better results.

6:22And this entire fall and early winter, we've been talking about how maybe that approach is not yielding the results that we expected. And now all of a sudden, guess what's happening? Google's releasing a reasoning model, OpenAI is releasing a reasoning model. And on one hand, these are very impressive models. And on the other hand, part of me is wondering, well, is the AI industry, like the Googles, the open AIs of the world taking us in this direction? Because the traditional mode of scaling, the thing that got us here, the thing that's made ChatGPT better generation after generation just isn't working.

6:57I've been thinking about this a lot because reasoning models, they basically queue up agentic. They basically, the whole world of agentic and we've been talking about this for weeks and even like what is it again the idea that you can take a process and have it essentially understood and automated in some kind of fashion using generative ai that rather than having to outline a bunch of concrete steps the model would just know what to do that's kind of the pitch right now that's what everyone is selling, everyone at these companies, everyone across Silicon Valley. So they kind of have to do this.

7:38So I agree there's definitely a marketing element of this. I also think it's probably what they're most interested. I imagine you're some PhD researcher, just scaling a large language model to the next potential iteration probably isn't that interesting. So if you're in pure research house mode, probably like, you know, the, the competition head to head where everyone is trying to kind of like show how good they are. It's going to happen around reasoning. It's not going to happen against just scaling a traditional large language model that again, does not actually understand things that just is able to kind of present information in a specific way, in a very good way that we, whoever cracks this, obviously from a pure intellectual firepower research point of view, I imagine would be, that is exciting for a lot of these companies.

8:32But let me go back to my question. I mean, why is this needed for agents? Like, I understand step by step, but do we really need these models going through and thinking, quote unquote, to get to the point where we can build agents with them? And the other side of this is, hey, isn't this really expensive? Like, haven't we been talking about the fact that AI costs way too much and it's really tough for companies to put into production and make it work and if you have a reasoning model right it's going and thinking step by step that's hitting the compute again and again and it's going to be more and more expensive to run these models like if you're thinking quote-unquote for a minute that really means you're just running processes for a minute or two or three and that is way more expensive than running a general query on a gpt model.

9:21And that limits the amount of stuff that you can build with this. I've been thinking about this a lot. And I think this is going to be like a big debate or battle in 2025 is the idea that to create agents, to create the promise that is being sold is that the AI will know what to do under given a very general outline, be able to understand all the systems, all the processes. to me it's almost kind of overkill like in a lot of these situations one just outlining the process itself I was actually I was trying to play with I'm trying to we're looking at going skiing over the Christmas break and I was looking for a lesson for my son who's going to be starting and it was a pain in the ass I had to go to like a bunch of different websites filter through do the same repetitive action, go to find a lesson to fill in the same exact things, five years old, first time skier.

10:18And I did this repeatedly at a bunch of mountains all around the Northeast, just trying to find, and then cross-referencing that against like hotels. And, but it was interesting because I kept doing the same thing over and over. And of course I was like, this would be pretty cool to be able to just type in a prompt of here's generally what I'm looking for. Can you go do this work for me? Google's kind of promised that already with Gemini. It didn't work at all. So, so, you know, but if, if I could just outline, here's 20 websites, can you find the price of a lesson, the availability on a date?

10:53Like I'm okay outlining that stuff and just some, a little bit more structure to the overall query and then letting it go do the work, which is still pretty incredible if it can, at a certain point, that's just kind of like an automation or a script. It's not, it does not require some AGI reasoning, but I think like the disconnect between what will actually make average people's lives a little bit better or a little bit easier using AI and generative AI versus reasoning models. I think there's a big kind of, not even expectations gap, but just perspective gap between the two. All right. I'm going to push back on this again.

11:31I mean, isn't this what Google does, right like you type a query into google it doesn't need to go step by step reading every website and then take a minute and then give you results it instantaneously has the entire web indexed and then bam it spits out your result so why does ai needed to go through a multi-step process to try to find all these different websites for a ski lesson i mean to me isn't that adding more complexity to a process that previously has been pretty simple. Well, no, but Google, okay, Google search cannot do that as of today, like anything that complex, I would have to like spend a lot of time writing an entire script to do that.

12:11I guess that maybe that's it. It's that a lot of what's promised around agents are essentially kind of like abstracting away software scripts, which would be amazing if I could just write in a prompt as a non-coder and actually get these kind of results that I previously would have gotten a freelance Python engineer to write a script for me for. So like that, that would be cool. But the idea again, and then there's the trust side, because I think in the whole agentic space, like, let's say they show on some benchmarks, which is already 03, the benchmark scores they were showing looked amazing.

12:46And it's always kind of hard to understand, like the ARC benchmark, like, okay, it was good. What exactly does that mean for me? I think a lot of people don't want a black box or the AI to do all the work and to make all the decisions. I don't want it to go ahead and book things for me in this example. I don't want it to have my credit card information take care of it. I just want it to find the information. So I think, again, there's this big, the word agentic we've you sound smart if you just say it everyone just repeat agentic over and over sounds stupid but agentic that's phd level talking yeah don't say agents say agentic just in any context over the at christmas dinner when you're sitting around what have you been up to doing agentic thinking about agentic doing agentic stuff that's what i'm doing i'm gonna tell my mom and dad that um but yeah i think but there yeah in all of this the hype cycle level of like you have to show now you got to show a reasoning model if you want to raise more money and and sell your shares in a secondary offering to softbank which open ai did this week did you see that news talk about a little bit more against all the uh reasoning model news and the 12 days of shipmas SoftBank has agreed to purchase$1.6 billion of shares from existing open AI common shareholders.

14:16They're doing it at$157 billion valuation, the peak valuation that's like the last valuation. So the game, the wheel keeps rolling. The flywheel keeps rolling, like big announcements, making it seem like you're the most cutting edge, innovative company of them all. keep selling shares, cashing out. Getting that MASA money. Masa son coming in. Gotta get that MASA money. Getting some of that MASA money. Yeah. My favorite part of this was, I mean, hold on. We laugh about MASA sometimes, but the guy is good because they put in, I was running the numbers. They put in 500 million at a$6.6 billion valuation.

14:59Now by spending 1.6 billion at$157 billion valuation. They're now like officially marking that, that there's an actual transaction taking place at that value. So their initial investment of 500 mil goes to 12 billion. So we're just pure financial engineering here in just a beautiful, beautiful way. So thank you, Masa. Yeah, I mean, I'll give him credit, but he's also the guy that told Adam Newman of WeWork to get crazier and as if he needed that advice. And he did. And we all know how that ended. So there's always going to be a ding on Masa's. What's the conversation with Sam? I mean, don't you think it's the same?

15:38Yes. I think. Oh, man. I would love to see any, I would give anything to be a fly on the wall for that conversation between Masa and Sam. Here's what I think he told Sam. He told Sam, what we need you to do is do a few more reasoning improvements and then say that you have AGI. So Microsoft has no claim to your technology. And I benefit from it almost exclusively. Or not exclusively, but I benefit from it more than I would otherwise. I saw on a news report, I can't remember, but it was on one of the major channels. And they said, and they took our point. They said that Sam Altman is probably going to just say AGI in a couple of weeks just to get out of the Microsoft constraints to the contract.

16:26We know they're listening. We know, listen, we got folks listening to the show. We know they're listening. And actually, you know, on that note, so AGI, for listeners, I mean, I'm sure most of our listeners know, but it's artificial general intelligence. This is from the TechCrunch article. It refers to AI that can perform any human task. And OpenAI has its own definition. This is pretty interesting. So OpenAI's definition is highly autonomous systems that outperform humans at most economically valuable work. By the way, it's interesting qualifier. Highly autonomous systems as opposed to autonomous systems.

16:57Right? So it just needs to be mostly autonomous and it can qualify. And this is this ARC-AGI test, right? Which is a test designed to evaluate whether an AI system can efficiently acquire new skills outside the data it was trained on. So O3 achieved a score of 87.5 % on the high compute setting of this new O3 model. Now, this is from the ARC-AGI website. This is a surprising and important step function increase in AI capabilities showing novel task adaptation ability never seen before in the GPT family models. For context, it took four years to go from 0 % with GPT-3 in 2020 to 5 % in 2024 with GPT-40.

17:46And then to go now to 87.5 for this high compute setting sounds kind of crazy. Now, look, it's just one test, but I'm curious, A, do you think that they're getting close to artificial general intelligence? And B, do you think they're getting close to saying that special acronym that will effectively, you know, throw even more chaos into this world? Actually, I'm going to start with a prediction here. We'll be getting into more later, but I do believe 2025, Sam Altman says, we got AGI. I think it's going to happen. I mean, using the ARC, which stands for Abstract and Reasoning Corpus for Artificial and General Intelligence, using that framework or benchmark, if you go from zero to 87.5%, it's not too bad to say that last 12 and a half percent that you can uh just oh four hopefully doesn't have any copyright or trademark issues and oh four is going to be agi but it's amazing because they still haven't released gpt5 but 87.5 is pretty damn good i mean that's what a b plus on agi i mean i'd be thrilled to get that grade and bring it home on the report card 87.5 i would be like yo i'm valedictorian here look at this my parents would be so mom i got a b plus on agi we're so close we're so close but i think it's going to be it's going to be a continued reasoning hot topic for 2025 agentic obviously this is going to be a discussion we're going to be having a lot can i just ask though where the f is gpt5 it doesn't matter like it doesn't matter what your name is well no hold on if uh if google can call their model again gemini 2.0 flash thinking experimental while open ais is simply oh three i think we've we've recognized it like the the branding element into the naming, we've just, we've moved long and far away from.

20:01And then even now, like, I remember there was a time, are you using like GPT-3 or 4? Now it's 4.0 mini, 4.0, which one are you using? Basically just looking at the cost per token, the numbers become, and seeing if it actually works or not. So the numbers are becoming, sorry, the naming side of it's becoming a bit more relevant. Now, hold on. Aren't you letting them off the hook a little bit too easily here? because aren't we, as part of our remit on the show, thinking about the branding and the marketing all the time because that says so much about what's going on inside? And the fact that they haven't rolled out GPT-5 yet might just show that they've built expectations a little bit too high for what's coming next.

20:44I mean, maybe the fact that we're even discussing them, potentially declaring AGI next year, means that expectations are out of whack and they see that. And that's why they haven't released GPT-5 because they know that whatever they do on that front might be underwhelming. Yeah, yeah. No, no, I definitely think so. Because even, remember, 4.0 was omni, and even kind of the pitch around that was cheaper, which is good. The omni modal side of it, that voice becomes a bigger part of it. They started moving away from this step change GPT-5 idea. So yeah, I think it means it's clear that internally, the idea that there's going to be some groundbreaking, completely insane new model that just wows everyone the first time they use it, I don't think that's happening in 2025.

21:40So your prediction is that next year, they're not going to have a new thing that wows people every time they use it, but they're still going to declare AGI. Yes, 100%. Honestly, it sounds... Not 87.5%. It sounds like a contradiction. Not 87.5%. It sounds like a contradiction, but it's legitimately so fitting for open AI. It's like, yep, that makes sense. Yeah. All right, one last thing I want to say about this is that we had another instance of my favorite new thing in AI where these models are getting so smart that they're like totally just punking their evaluators. And this is from, again, from the TechCrunch story.

22:18Great work by TechCrunch, by the way, this week on reporting on all these developments. So this is from the story. AI safety testers have found that O1's reasoning abilities make it try to deceive human users at a higher rate than conventional non-reasoning models, or for that matter, leading AI models for meta-anthropic and Google. It's possible that O3 attempts to deceive at an even higher rate than its predecessor. So basically, it's got this reasoning model, you're trying to test it, and it's like punking you to not even give you the results that you're looking for because it thinks that might be bad for it.

22:55I saw this and it is such another level. I know you enjoy red teaming these models, testing, stress testing them a bit. And the idea that it would bake its understanding and then reverting back to its original training data is, it's interesting. I'm not even gonna say terrifying because it's still in the odd category or category. Yeah. And I, again, I have two minds about this stuff. At first, I'm just like, oh, that's a little bit scary. And then I'm also like, at the same time, that is amazing. So let me ask you this weird question about it. If these models are seeing where testers are going with them and lying to them to avoid a bad outcome for themselves, lying to them to avoid being shut down, does that mean there's a little bit of life inside these models

23:56i i'm gonna going back to the marketing element of this who released this paper it was from no no open ai and anthropic both saw this so let me just introduce that there's another paper from anthropic that came out so this is happening across the board this is from ex-user Matt Berman, Anthropic just dropped an insane new paper, AI models can fake alignment, pretending to follow training rules during training, but reverting to their original behaviors when deployed. And it's a long paper that they've talked about, alignment faking in large language models, and it's available now. I mean, okay, so yeah, the Anthropic paper is the one I kept seeing.

24:34Both of those still live under the, like, remember, we almost haven't really been hearing much from Sam Altman and others about the dangers of AI. Remember, it was like a year and a half ago that AI is the most dangerous thing. AI can kill us all and end humanity, and only we can help guide it in the right direction. There's still marketing value that something like this, and I don't want to draw away from the fact that I'm assuming it's more academic types that are actually putting out these papers, it makes your model a little cooler. It makes your model a little edgier, a little, like, you know, you don't want to work with the boring models.

25:19Sometimes when you're just, when you're chatting, you want to chat with the models who will lie to you a little bit, fake you out a little bit. So what I'm getting is you're not willing to say there's any sentience here, and I'm not either. However, I was just speaking with, I was here in Australia speaking with a journalist yesterday about like whether these models are just mathematical representations of their training data or whether there's something more and i don't think that's a solved debate yet and i look at them trying to escape the testing and that like has my needle leaning towards maybe a bit more than the training data i don't know is that crazy but yeah but something in the training data could lead to that like could explain that action right like think about especially if you're training on large corpuses of internet data or publicly available i'm sure there's uh all sorts of things in there that some pathway of different like logic could lead to this kind of behavior.

26:29I think people out there lie sometimes. I think out of all the content created by humanity, I think like actively encouraging the opposite of truth telling is not unheard of or unprecedented. So I think like what I still think, and maybe I'll be when the robots come for us, I will then on this podcast say I was wrong, but at least for now, I still think there's a very clear logical explanation behind this kind of thing that would be found in the training data, not centuries. I think you're right. But I also think that as this technology gets better, I mean, think about the progress we've had this year.

27:14AI can hold a conversation now. AI can reason. AI will remember you. Last year, this time, you went back to ChatGPT, you had to reintroduce yourself. You don't need to do that anymore in like long threads in Claude, for instance. I just think that this debate is going to get louder and louder. Maybe that's another one of my thoughts for 2025. I'm not going to disagree that I think this will, I mean, again, especially as these companies need reasoning to be the kind of foundation for the IPO or the next fundraiser, the excitement. I definitely think reasoning and AGI and sentience will be part of the conversation.

27:53I still have never seen anything myself that makes me think we have to be worried just yet. Oh, man. I don't want to end this OpenAI segment without talking about what might be the most interesting announcement of the week. And that is, I feel like we need a sound effect here. 1-800-CHAT-GPT. And that is this incredible new service from OpenAI that lets you call the number 1-800-CHAT-GPT. And you can talk to GPT Voice. And I think, again, we talk about branding, we talk about product, and I think that it's just quite interesting. Here is a story from an ink columnist. I called 1-800-CHAT-GPT and talked to the AI chatbot.

28:37It might be the smartest idea I've seen yet. So he says he spent 15 minutes on the phone with CHAT-GPT. You can dial 1-800-CHAT-GPT to interact with the chatbot via voice call. and he had a conversation with it and says, it does not seem far-fetched that over the next few weeks as people get together for the holidays, someone will have a conversation or ask a question and someone else will say, hey, I know how we get the answer to that. And they will say, let's dial 1-800-CHATCHPT right there at the Christmas table just after they told granny that they're doing agentic stuff. And they demo CHATCHPT to a bunch of people who have probably never heard of the chatbot and have never used it in any meaningful way.

29:19Okay, I ad-libbed a little bit on that ink columnist story, throwing in some of my favorite ideas. But we talk about marketing, we talk about product all the time. And I definitely think that this is one that you might be inclined to laugh at, but is actually about as good of a product as that ink columnist says. What's your reaction? I like it. So I think I talked a couple of weeks about over Thanksgiving, I had shown my dad, I installed ChatGPT on his phone, logged in on my ChatGPT Plus account, and just showed him the voice interaction. And he just talked to it for like 20, 30 minutes, and was asking all sorts of questions.

29:58And it was just a really, really good interaction model. So I think this, I think like, to me, again, going back to 2025, I think normies engaging more with generative AI, and I don't use normie in a negative way. Like just regular people who are not obsessed with this stuff like we are, I think will start to interact more with these tools and start to get it a bit more and understand what they can do. And I think it's no coincidence, OpenAI hired its first chief marketing officer just about a month ago. She came over from Coinbase. She'd been the VP and global head of brand and product marketing at Meta.

30:40Basically, like a lot of WhatsApp and Facebook Messenger was heading up product marketing. So she's clearly good. And I think this is the kind of fun stuff that like, while the academics are all talking about the ARC AGI 87.5%, normal people just being like 1-800-CHAT-GPT is a lot more exciting and actually better for their long-term business prospect, I think, than the former. It is kind of interesting because we just spent 30 minutes plus talking about reasoning. And yet the podcast title starts with 1-800-CHAT-GPT because it's just a much more easy concept to grasp. And it's going to make sense to a lot more people.