Is Generative AI Plateauing?, Booming Bluesky, Apple’s Smart Glasses Play

0:00Is generative AI plateauing as training methods top out? Blue Sky is booming as an alternative social network, and Apple looks into smart glasses. All that and more is coming up on a Big Technology Podcast Friday edition right after this.

0:16You're used to hearing my voice on the world bringing you interviews from around the globe. And you hear me reporting environment and climate news. I'm Carolyn Buehler. And I'm Marco Werman. We're now with you hosting The World Together. More global journalism with a fresh new sound. Listen to the world on your local public radio station and wherever you find your podcasts.

0:43Welcome to Big Technology Podcast Friday edition, where we break down the news in our traditional cool-headed and nuanced format. We have a great show for you today covering everything happening in the world of AI. Very big news. There's concern that training methods that have gotten the generative AI field to here are not going to continue to scale. And that's really coming to the fore right now. We're going to talk about that. We're also going to talk about the rise of blue sky, whether it will be sustainable, and Apple smart glasses play, which is quite interesting. Joining us as always on Fridays to break it all down is Ranjan Roy of Margins.

1:15Ranjan, welcome to the show. Great to see you. Scaling laws are here, Alex. They finally come for the industry. I know you're excited about what's going to happen with this, but I know you're even more excited about the Jake Paul, Mike Tyson fight. Who you got? Neither. Neither. I think Netflix is genius in promoting their live programming by just bringing out two people that no one wants to see win. But I'll still take Tyson if I have to. All right, I'm taking Paul. We can put it on the prediction markets and see what happens. My friend group has been saying that this is an exhibition match, basically.

1:50And a lot of betting sites are looking at it the same way. The folks are saying that it's rigged. Do you think this is rigged or a real fight? I think this is a real fight. I don't think Netflix would go into this fully rigged or making it reality TV, but I think it's a good reminder of the blurred lines between reality TV and actual live programming. But I think it's real. I'm going to go rigged. Okay. We got a lot of palm market contracts to set up right now. Absolutely. And now we can talk a little bit about what's happening in the AI world, where, shall we say, there's another fight going on between the purists that believe that large language models will continue to scale if you add more data and compute and power to the mix.

2:32And those that say eventually these models are going to hit a wall. There's been a long brewing battle between these two factions. And this week, I think, has been the week where both sides have started to put their positions in the ground and say, you know what? I've won. And that's really come on the back of this great information report that talks about how OpenAI has basically found that it has hit the limit of improvement when it trains with more data, more compute, and more power. Here's from the story. The number of people using chat GPT and artificial intelligence products is soaring. The rate of improvement for basic building blocks underpinning these products is slowing down.

3:16The challenge that OpenAI is experiencing with its upcoming flagship model, codenamed Orion, shows what the company is up against. While Orion's performance ended up exceeding that of prior models, the increase in quality was smaller compared to the jump between its last models, GPT-3 and GPT-4. Some researchers at the company believe Orion isn't reliably better than its predecessor in handling certain tasks. This could be a problem as Orion may be more expensive for OpenAI to run in its data centers compared to other models. Basically, the idea here is that OpenAI has been training subsequent foundational models with more data, more compute.

4:01And it's reached the point of diminishing returns to the point that this grand next model that it's supposed to release. We don't even have a sniff of GPT-5 yet. It's calling this Orion. is going to be just a bit better, but more expensive. What do you make of this battle? I am happy about this. I think I've talked a lot about how I don't need GPT-5 just yet. I think the amount of opportunity there is around actually productizing the current models is so massive right now. Again, there's so many little magical moments, even with Claude, with ChatGPT, with any of these tools that you see so much potential, but then actually translating that into helping you do your job better or create certain things better.

4:48I think that there's just so much work to be done there that everyone competing to kind of create this next massive foundational model has never made a ton of sense to me before they actually just got a GPT-4 or a Claude 3.5 Opus working, you know, kind of pushing it to its limits and making it work as well as possible. So I'm kind of hoping that this actually moves people towards making tools people use rather than just saying AGI and GPT-5 and whatever else. Now, look, I hear you, but I also have to disagree. I mean, the field's potential is so much more if these models continue to improve. And while they're good today, they're not where they've been promised to be.

5:33And if this is the limit, then it severely diminishes sort of the potential of these models to change everything we do as the AI industry has been promised. And it's by the way, it's not just this information report, lots of people have been saying this. So this is Elias Sitzkevr has talked about it. He says to Reuters results from scaling up pre-training the phase of training and AI model that uses a vast amount of unlabeled data to understand language patterns and structures, it's plateaued. Ben Horowitz and Mark Andreessen talking on their podcast. Horowitz says we're increasing the number of graphics processing units used to train AI, but we're not getting the intelligent improvements at all out of it.

6:16And Andreessen has been saying that lots of smart people are working on breaking through the asymptote, figuring out how to get to higher levels of reasoning capability. I mean, shouldn't we just put your concern about building practical applications aside for a moment? I don't think anyone's going to disagree that it's a time to build practical applications, but isn't this like fairly concerning for the progress of the AI industry if let's say this is about as smart as they're going to get? No, I think first of all, I love Ilya Unleashed right now, going on, going to Reuters and now being able to say things like the vast amounts of unlabeled data to understand language and patterns and structures have plateaued.

6:59But I, yeah, I think the focus being kind of distracting by focusing only on these step changes in quality of the models has distracted from practical applications. Yes, we should be able to have both, but you even see it in the way that an open AI is structured as a company and where they invest their resources. We talked a lot about this with Corey Weinberg from the information that the cost structure of open AI is still much more heavily towards the R &D and improving the actual models versus building out a good sales force and a sales enablement and customer success team. And these things might sound boring, but if you're actually want these technologies to be adapted by corporations and companies and just everyday people, they have to be easier to use and more practical.

7:53Like I get the idea that there's a lot of times that if you're using one of these tools, it doesn't work perfectly the first time. And everyone, the kind of natural reaction is, okay, I guess it's not good enough. But then you learn, the better you prompt it, the different you structure your workflow, you can get it to do what you want it to do. But instead, I think a lot of these companies are promising the model will get so smart in the next iteration that you don't even have to do that work around prompting and workflow building that it'll just figure it out and it'll be okay. And AGI will be here, etc.

8:30But isn't that what they're trying to do? I mean, aren't you ignoring the business story here that OpenAI just raised the largest VC around in history,$6 billion. Is it Microsoft or Amazon are hooking up to nuclear power plants? Anthropic is out in the market trying to raise billions of its own. Just from a business standpoint, if these companies cannot advance this anymore, isn't all that money going to come do and sort of crumble the industry? I'm not ignoring the business story at all. That is the business story to me. I think like over raising for the R &D side of things rather than the actual like operationalization and building out businesses on top of the existing technology.

9:11I mean, again, we've debated this plenty. I think that is a huge mistake and that it actually, you know, potentially hampers the long term development of the industry. So if it actually means this slowdown, you know, puts a little cold water on the promises of GPT-5 and whatever else, and people just get back to work in terms of actually building things that solve problems, I'm happy about that. I'm trying to pin you down here a little bit on the technology question and you keep wiggling your way out, which I respect. But I have to ask, like, isn't there just a tad of disappointment on your end if this is sort of the end here, the end of the road in terms of where this is?

9:53Not at all. I mean, the things I've already been able to do, I just made a little game in Claude the other day. I saw some video of like, it's kind of like a Space Invaders type of game. I coded a Space Invaders type game with these like custom images myself in an hour and then hit the cloud limit, which a lot of listeners probably do. And it's kind of annoying even as a paying customer, but like that was magical to me. And that exists on the existing technology. It's possible. And there's so many other applications I can imagine if I'm able to do that for fun in an hour that are not being properly explored because all the attention hype is on the much, much bigger thing.

10:39So if the Claude business gets built as, you know, actually teaching people how to use the existing technology well, I think that has, again, much better longer term potential than the entire bet on the entire industry is the technology will get step change better in the next year or two. Yeah. I don't know about that. I mean, they have, that has been the bet though. It has, it has. And I don't think it's the right one. And I think something that pushes us away from that kind of strategy is going to be a good, it'll shake things up. It'll definitely shake things up, but I think it's, it's healthy for the longer term, uh, world of AI.

11:23You know, you've really not played into my game today where I wanted to evoke feeling of one single feeling of disappointment or sadness from you. And you say, yeah, it would be tough if this is how I'm feeling. Gosh, like if this promised, you know, AI revolution ends here, then I don't know how far we get. And then I come in and say, well, actually maybe we're not done, you know, sort of like that Walter White give, we're done when we say I'm done. All right. Well, I'll give you video generation is the one area that I do think we are severely, we're not even close to anything interesting. And we've been promised things that are interesting, i.e.

12:05Sora. But we're very, very far away. Even the runway MLs and other tools that I've tried, we're so, that's one area where I see a huge need for technological improvement. But for any content generation, any coding, even data analysis right now, I think the models are pretty damn good at doing what most people need them to do. We just don't know. Most people just don't know how to use them correctly. Well, Ronjan, thank you for playing along. And I have to inform you, we're not done. We're done when we say I'm done. And that is That is because, yes, these research houses might have hit some sort of wall.

12:47And the reason why they're hitting the wall is obvious that they're using synthetic data because they've run out of data and it's offering less good results. And this has sort of been the issue with these training, these new models. However, in recent times, there has been a development of a new discipline here, which is reasoning. And we talked about it back in the day. And that's what sort of freaked Ilya out. And he left OpenAI. And that really might be the near future of this field where the models now, such as OpenAI's 01, are prompted and they think. And the more they think, the better they get.

13:27And this is, again, from the information. In OpenAI's case, researchers have developed a type of reasoning model, 01, that takes more time to think about the data that LLM trained on before spitting out an answer. This means the quality of O1's responses can continue to improve when the model is provided with additional computing resources while it's answering user questions, even without making changes to the underlying model. And Casey Newton from Platformer cited one example from one OpenAI researcher talking about it. this open AI researcher says, and this was out of Ted AI Talk, it turns out that having the bot think for just 20 seconds in a hand of poker, step by step, got the same boosting performance as scaling up the model by 100 ,000 times and training it for 100 ,000 times longer.

14:17So I think what we're about to see is a pivot in the AI research field, where yes, they might be applying practically some of the models that exist today. But it seems to me like everybody is going to go completely in on this reasoning format. And that is going to be where we see the improvements. And that's why I want to highlight this post from Dan Shipper that I saw this week. He says, the message that the information headlines conveys is at odds with what people inside the big labs are actually feeling and saying. It is technically correct, but the takeaway for the casual reader that AI progress is slowing is the exact opposite of what I'm hearing.

14:55So this might be a combination of spin and reality, but I'm curious how much stake you're putting into reasoning when it comes to being able to advance the status quo. Yeah, no, I think both reasoning and how synthetic data is used matter. And I think actually are an almost more promising direction for the industry than just raw processing power and size. I think first on the synthetic data, like we're going to be talking about a company writer.com in just a little bit, but they, one of the things they did was like create their own foundation models. And they apparently trained them for$700 ,000 total by using really targeted synthetic data to create different models for different kinds of problems.

15:41And I think in the coming months and years, we're going to start to see some awkward headlines around size matters because smaller will be better in terms of actual models being used. And like, again, one model should not be reliable to solve every problem for everyone at all times versus maybe there is a model focused on financial data analysis. And it's actually much better at solving problems around that versus writing poetry or generating images. So I think using really targeted synthetic data for more targeted models is actually a really interesting space. In terms of the actual reasoning side, I think that's really interesting.

16:25Like, like rather than coming up with new ways of actually generating the answers using the existing information that could be incredibly promising and solve so many of the ideas, so many, so many of the challenges facing the industry, like cost for any kind of new foundation model, like just, you know, viability of these things actually succeeding. So I think, again, today, today I'm positive. Today, these are all good things for me. Right. Right. And the cost really matters because if you're using a reasoning model, a lot of that can happen in inference versus in the training, which is, I think, less expensive.

17:03Before we move on from this, I just want to talk quickly about this AGI thing that we talk about so often, but rarely define and rarely talk about in context, right? That all these labs are trying to push toward artificial general intelligence or human level intelligence. And it seems like some inside these organizations are like full fledged trying to get there. and others, I don't know, probably like see it as useful marketing so they can sell products today. Actually, I think that a lot of the productization that happened has kind of been an accident in places like OpenAI as they've pushed, you know, the research forward.

17:36But why don't we just take this point in time to just talk a little bit about AGI? Do you think A, it all along has just been this marketing term? And do you think that if we're not going to get there through these current methods, that the magic of that marketing falls away a little bit, making it harder to sell into companies, making it harder to fundraise if all these companies are doing are just sort of productizing what they have today. And I guess, B, do you think we'll get there? I think I'm going to go with A, and it's because, I guess, how would you define AGI or artificial artificial general intelligence?

18:19I think Jan LeCun's definition is really good, which is basically that it's human level intelligence. It can handle a variety of things just the way that a human can. What is human level intelligence though? Because there's a, I don't know. I think ChatGPT can already do a lot of things better than plenty of humans. It's almost like, yes, it can answer questions about philosophy the way that a philosophy professor could, but it's almost like the little more nimble things that it really struggles at. ChatGPT, you can't tell ChatGPT to like, you know, go, you know, write a bunch of emails to people you need to communicate and it does it for you well.

18:58It's not really able to do that. It's not really able to switch very well between tasks. It's not very well, it's never really able to, you know, learn in the context and get things right the next time. These are all things that I think make human intelligence special as sort of the adaptability and the ability to be, as we say, general. And I don't think AI is there yet. Okay. That's a fair definition. And using that definition, I actually think it's fine for the industry to not be on the direction of getting there. Because even what you said, writing a bunch of emails to different people, I think that problem could and should be solved soon in really targeted manners.

19:40Like, you know, take your entire existing email history, train something on that, use that to generate new emails and build like a process or workflow where you actually validate them. Like, I mean, really practically, I think solving that problem could be possible pretty soon. And it's just not getting solved because we're still all trying to chase the dreams of AGI. And I think for me, and what exactly it is, The human level reasoning makes some sense, but it's amazing to me that it's always brought up, but there's not one like clear accepted definition or one clear vision that's communicated by the biggest people in the field, the Sam Altman's and everything else.

20:23So it remains this murky kind of like dystopian robots taking over who knows what it is, will be a line item in a contract with Microsoft's investment in order to like change the profit structure. I mean, it's such a nebulous term that that's why I think it does represent a distraction from progress. I don't know what it says about my life that you're like, imagine the strongest form of artificial intelligence possible. What does it do? And I'm like, yeah, just write a bunch of emails. Like, oh, my God. Imagine a world where I'm worried about robot takeovers and you're just trying to go to inbox zero here.

21:08Honestly, if an AI could get me to inbox zero, it would be a true a true miracle. I would really believe in the power of science. It would have to get through 12 ,000 unreads. But it is interesting that they have built AGI. It initially was like sort of like what I was describing, human level intelligence, adept, able to generalize. And now I think it's talked about really in a way that's akin to super intelligence, something that's smarter than humans in almost all fields and can perform things that humans can't. And that's when you hear like the messaging coming out of open AI that it can lead scientific discoveries and these type of things.

21:43And it's like, okay, that's not really general. That's super intelligence. And, you know, I think that that has led a lot of the investment and a lot of the hype around this that will eventually get there. But it just doesn't seem like it's going to be through the traditional scaling of LLMs. I guess that's my point here. Yeah, I agree on that. And you're like, I don't care. That's good. I don't care. That's good. That's my new philosophy on scaling laws with LLMs. It's a good tagline. Okay, last thing about this. Did you hear that Google? I think this is worth watching. They have a new experimental Gemini model.

22:17It's called Gemini EXP1114. And do you know about Chatbot Arena where they test which is the best LLMs? It's currently sitting at the top of Chatbot Arena and kicking the butt of ChatGPT4.0, O1 Preview, O1 Mini, previous Geminis, all the clods. maybe Google's got it figured out. Whatever they're doing there seems to be working. That it's, and this, by the way, folks, this arena is where people compare responses to different models and pick the best one. And it's been voted on by 6 ,000 folks at this point and is at the very top. So it's quite a moment for Google that I don't want to glance over.

22:58And I think we'll probably be coming back to it when we talk about Google's prowess in the field. Yeah, I think, and for listeners, I mean, check out Chatbot Arena. It's honestly a fun thing. And it's a blind comparison test. So you don't know, you're given two answers, you select which one you think is better, and then you find out what's the actual model behind it. So it is essentially an unbiased test. Gemini, I have a question. Are you using Gemini in day to day life? No, I'm still all in on Claude. And I'm looking at Chatbot Arena and I'm like, I am not doing it right. I mean, it's interesting because maybe these models can give a better response, but it's also just like it matters.

23:44Like UI matters. I mean, this is sort of me agreeing with your argument that user interface matters. Are you agreeing with me? Personality matters. Usefulness matters, even if the model is smarter. But this is making me think it's time to give Gemini another shot. How about you? I don't use it very often. I have it bookmarked, but I mean, still, Claude Perplexity, ChatGPT, steady rotation, all for different use cases. I think Perplexity is almost one of the best examples of UI completely transforming how nice it is to use. and by this point I really thought Gemini should be my entire travel booking given it's connected to Google travel Google flights and everything Google Maps like it should be my starting point and it still isn't and may I still think the answers in the existing form just are not very good.

24:41It gets a lot of stuff wrong. It doesn't answer a lot of things as well. And I get trying to be a little bit more conservative and risk averse. I still think Google is incredibly well positioned just given their ecosystem, but it still has not gotten there yet. And I mean, it's only maybe the next experimental model, once it becomes reality, we'll kind of cross that chasm, but we're not there yet. Yes. And I have to say, I have become a bit of a perplexity guy. I'll admit it. Perplexity is pretty good. It's so good. It's my, it's become my kind of like companion for other things. Like I think chat GPT is more when I have like, I'm sitting and doing something focused.

25:25Claude is for a lot of work, a lot of like more on the coding side and kind of like really the art using artifacts perplexity is when I'm watching a movie a sports game of any sort like like it just is so good and just giving you quick information in a really nice format with additional links to keep exploring and questions that uh I think for that like and which makes me think it's almost it and it still is the biggest competitor to real search yeah I find it to be really good for research imagine trying to sort through a bunch of programs and figuring out what they offer. Yeah, I am just making the decision on the Epic Pass versus the Icon Pass for winter skiing, if other if listeners are making the same decision and all done in perplexity and asking like specific questions, which resorts in Vermont, which resorts in Colorado are there?

26:19And like it was so, so good in doing that. I do hope that perplexity finds a way to get me into China this winter, which I'm hoping to stop in on the way back from Australia. So fingers crossed. Where are you looking to go? Beijing. I want to see that wall. See that wall? I was there in 2009. I went to the Great Wall. It was definitely, it was a good time. It was lived up to the billing. Nice. So, okay. So let's just take a minute and go through three stories that the two of us kind of found this week that are talking about what you really are interested in, Ranjan, which is the practical application of AI at this stage and how even if we stop right now, which I don't think we will, we're going to have an extremely powerful technology that's going to disrupt industries and really be practically useful.

27:10So why don't you kick it off with this Rider fundraising that you talked about right before? Yes. So Rider is a generative AI startup. They just raised 200 million at a nearly$2 billion valuation. What's interesting about them is they built their own foundation models. We had just talked a little earlier about how they build more kind of targeted models that are really focused on solving enterprise business problems. And the entire kind of differentiation that they're focused on is kind of exactly what I've been talking about. They have a lot of big name enterprise customers going in. And remember, these companies have messy data.