New ROI Questions For AI, Microsoft’s Empire Plans, Jassy’s Amazon Comeback

0:00Tough questions about the AI business are starting to be aired loudly. Microsoft's Satya Nadella is seeking to build an AI empire beyond open AI. And Andy Jassy's Amazon is thriving after a bumpy start. All that and more is coming up right after this.

0:18You're used to hearing my voice on the world bringing you interviews from around the globe. And you hear me reporting environment and climate news. I'm Carolyn Buehler. And I'm Marco Werman. We're now with you hosting The World Together. More global journalism with a fresh new sound. Listen to the world on your local public radio station and wherever you find your podcasts.

0:46Welcome to Big Technology Podcast Friday Edition, where we break down the news in our traditional cool-headed and nuanced format. We have a great guest for you today. Helping me anchor the show is Tom Doton. He's a reporter for The Wall Street Journal covering Microsoft and all things AI. Tom, great to see you. Welcome back to the show. Thanks. It's a big honor. I'm feeling cool-headed. All right. Well, let's see if we can maintain it through the course of this episode. One of the things that has my head a little bit in a place where I'm scratching it is this disconnect between the actual fruits of the AI moment and the promise.

1:25And I spoke about it with Matt Wood from AWS on Wednesday. And it just so happens that many posts are starting to show up with these similar themes. So Bendik Devins, the analyst, also a friend of the show, has just put out this post called AI Summer, making the point pretty well that we've been promised a lot. But is it adding up? So he said about ChatGPT, for consumers, ChatGPT is just a website or an app, and it could ride all the infrastructure built over the last 25 years. So a huge number of people went off to try it last year. The problem is that most haven't been back. If you ask what they used it for, it turns out that most people played with it once or twice, only to go back every couple of weeks.

2:12And here's his bottom line. If this is the amazing, magical thing that will change everything, why do most people say, in effect, very clever, but not for me, and wander off with the shrug? And why hasn't there been much growth in the active users as opposed to the vaguely curious in the last nine to 12 months? I think it's a really good point. I think he really nails sort of the big question about this moment in the AI, quote unquote, boom. And I'm curious what you think. Does he say in this post what the latest active user numbers are? I have not kept up with it. And I know that last year they were talking about 100 million and they kept changing 100 million.

2:51What? Like it was 100 million weekly active users. Initially, it was daily active users. I haven't seen a new figure. Does Benedict have a new one there? He doesn't have a new one in the post. The most recent number for chat GPT from OpenAI is still 100 million users. Yeah. So people say like they're whispering, at least what I've heard in sort of the back rooms is it's sort of up to 200 million now. or maybe a little bit more, but we haven't seen the sort of exponential growth that you might expect if it is, is like Benedict is saying, this like unbelievable invention that's about to like mirror human reasoning.

3:27If it's 200 million, they should have put it out by now because this 100 million figure has been it for a while. And I remember last year, they had a developer day conference where I think a lot of people were expecting, a few days after that, Sam Altman got fired, by the way. Maybe it's because he didn't have that 200 million. number. If only it was that clear, if only that's what it were about, I would have, that would have saved my Thanksgiving. Uh, if it were that simple, I apologize to you, man, that was a rough one, right? Yeah. Real low point personally for me, let alone Sam, I don't care, but for me, it sucked.

3:59Um, yeah, there hasn't been a new number in a while. And I think that to a degree is a red flag because if this is supposed to be, and this was initially touted as the fastest growing consumer app of all time and the fastest to get to 100 million, which it means something. But if the gap between 100 million and 200 million is starting to get longer and longer, and we're not seeing the same sort of growth we saw at first, I think it's reasonable to temper expectations and say, you can't just map it alongside every other consumer app like Facebook or Uber or something like that, where you saw this exponential growth that continued past 100 million and say like, oh, it's going to be the next that.

4:38Like if it really has reached a ceiling, doesn't mean it's a disappointment, but it's certainly not the thing that people initially thought it was going to be. Right. And maybe it's just because the models need to get a little bit better. I don't know if you saw this, but Bloomberg had this report that OpenAI had a bunch of different levels on the stages of artificial intelligence as we make our way towards superintelligence. So I shared this and a lot of people got mad at me because they were like, oh, you're just sharing OpenAI propaganda. First of all, I'm just putting it out there, not like, you know, saying that this is to be believed or, you know, sort of standing by what OpenAI is saying.

5:11But I think we should talk about the different levels. So for them, they say level one is chatbots, AI with conversational language. Level two is reasoners, human level problem solving. Three is agents, systems that can take action. Level four is innovators, AI that can aid an invention. And level five is organizations, AI that can do the work of an organization. And they say that we are past level one, approaching level two, at least that's what they say internally at OpenAI. Is this a matter of like, we're not using these chatbots as much because they're just not useful to the point where like, maybe when they get to level two as reasoners or level three as agents, those active user numbers will go through the roof.

5:57yeah i guess it's all a matter of utility right the other things that they're trying to achieve here are more difficult but also more useful to people so i actually kind of question their strategy in releasing this kind of scale publicly because if you were not able to actually hit these new levels over the next year or a few because the technology hasn't grown in sophistication I mean, it's all very arbitrary, right? So like there's an argument to be made that like they can claim they've hit level three, level four, whatever that means, because they've decided they've hit these levels. but if the technology that underpins all this stuff isn't going to get more powerful and sophisticated over the next couple of years then you've basically just told the world yeah we don't really have it like there is promise with this technology but we're not going to be able to deliver on it because we can't you know the the scaffolding that needs to exist in order for this to get there is not achievable yet so i actually i mean i imagine they put it out because they think they will but if they don't they kind of dug a hole for themselves doesn't it remind you a little bit of like the five stages of self-driving cars self-driving cars yeah maybe that too but like but really like we've been waiting to get to sort of like level five autonomy and just kind of stuck on level two forever what did that mean again what did level five autonomy mean i know what you're talking about but i can never remember what that meant i'm just going to give you directionally what it is like level one is driving a car with a steering wheel and level five is like fully autonomous vehicles.

7:28Right. Well, what's interesting with that is I remember covering tech back when people were talking about that. That was like 2015, 16. Um, and, and Uber or, well, Uber was very big into it and Google and, um, self-driving cars seemed like it was kind of a, kind of a mistake. It's just, just kind of a dud. They never really were able to get it on the streets. And, you know, now in San Francisco, the Waymos are everywhere. And I got to say, it's pretty great. Um, they're amazing. Yeah. It's a, it's a very good experience. I'm embarrassed to say how much I like them. Uh, I'm with you on that. It's the one, I mean, it's, I think it's the coolest tech I've tried in recent, in the past few years.

8:09Yeah. Yeah. And so, I mean, I, there's probably still many levels to go for it to be what the promise was, you know, a decade ago or so, but like they've really achieved it. Uh, it's at least more than I thought they were going to. But yeah, there is definitely a parallel between the two. And, you know, I guess if you want to give open AI or AI in general, the benefit of the doubt, like self-driving, it got there or it got there more than a lot of skeptics thought it was going to. And so, you know, maybe just give them more time is the smartest way to think about it. I don't know. Right. And then the question is, okay, is this just a patient's game?

8:42Right. Because the first group of entities that are going to start deploying this stuff is going to be the consultants really, or the B2B partners of these companies that are trying to put it into play in an enterprise situation. And I spoke with a bunch of Amazon folks this week and Bendik highlights this in his post, that it really is sort of a moment where the rubber is meeting the road for AI and it's going to take longer than a lot of people thought. So here's just a couple of stats that I'm pulling out from his post about the consultants. So he said, Bain tried to split pilots, experiments, and trials.

9:20And he said, everyone had a bunch of tests as far as AI goes, but far fewer people are trusting something in their business to this yet. Accenture, he said, proudly announced that it had already done 300 million of generative AI work for clients and that it had done 300 products. And he says, even an LLM can divide 300 by 300. That's a lot of pilots, not deployment. And I just published this in Big Technology Today, citing from a Gartner poll of organizations interested in artificial intelligence. And the Gartner poll had more than a thousand organizations responded. Only 21 of the companies surveyed by Gartner said they had Gen.AI in production.

10:00That's this year. And the rest were either piloting or exploring the technology. And this is what Bendix says about this. What happens when the utopian dreams of AI maximalism meet the messy reality of consumer behavior and enterprise IT budgets? It takes longer than you think, and it's complicated. Yeah. Yeah, I think that's spot on. And it's something I've been thinking about a lot reporting on this space because I cover enterprise software companies for the journal. And that's ground zero, right, for adoption of this stuff. Like these are the guys with the big budgets. you know, Microsoft is pushing their co-pilot, their agents, uh, as hard as they can to all their customers.

10:43And, you know, I don't have data yet on how well that's doing on its own, but I can tell you on the startup level, there've been a lot of companies that were building enterprise software, AI enterprise software that are in a tough spot right now because they're just not, there's not a lot of business and a lot of revenue yet. People aren't buying it yet. And I feel confident enough saying that there was a miscalculation by a lot of investors in AI that there was going to be quicker adoption to this. I really think they saw the rise of chat GPT and the fact that it reached this hundred million user level within a record amount of time and just assume that everything was going to follow in that footstep, in those footsteps.

11:23And it was going to be, you know, not only they're going to fall in those footsteps, but the revenue is going to, that spigot's going to turn on crazily fast. And I think we've seen there's so much hesitancy and slowness and just frankly, lack of capability in the technology that has stopped a lot of big enterprises from throwing huge hundred million dollar type budget buys at this stuff. And the repercussions of that slowness are playing out right now. It's that simple. Like there are companies that had made projections on startups. I mean, like projections on what revenue would look like based on uptake of the software.

11:57And it didn't it didn't happen. and like in a sense i think it was arrogance and hubris on the part of investors uh to just assume that it was going to happen because i mean this is always the issue with silicon valley right it's like oh they're early adopters here so they just assume everything's going to be like that and um this tech takes a long time uh new new if you really believe this is the revolution that you know you're claiming it is it's just going to take longer than uh than you initially thought and uh You need to be able to, as an investor or like Silicon Valley denizen or just a general like advocate of technology, like swallow that pill and be honest about it, that like this is slower than it was expected.

12:39Right. And I definitely want to get into the numbers because that's another post that came out of Sequoia this week. But before we do, I think one of the last questions that Benedict poses that's quite interesting is sort of what this technology is. Is it a product? Is it a technology? So here's what he says. stepping back, the very speed of which ChatGPT went from a science project to 100 million users might have been a trap, which is basically what you're saying. A little like natural language processing was for Alexa. Large language models look like they work, and they look generalized, and they look like a product.

13:13A science of them delivers a chatbot, and a chatbot looks like a product. You type something in, and you get magic back. but the magic might not be useful in that form and it might be wrong. It looks like a product, but it isn't. And then he basically says there's two options that we're the two different paths that we might be on. He says, you could also suggest that these startups are a collective Silicon Valley bet that LLMs are a technology, not a product, and that we need to go through the conventional process of customer discovery towards product market fit. Or the other thing is, and this is, he says, it's the thing that really drives a bubble, is the idea that history is over and LLMs will be able to do everything.

13:54And that in that case, we wouldn't need any of these companies. So I'm curious what you think. I mean, it seems really like we're, we are going to be in one, the option one, which is that this is a process, it's going to take a long time to find product market fit. And that's why people like Matt Wood at Amazon Web Services this week is telling me the name of the game is incremental and patience as opposed to like what people within the you know sort of sphere of influence back in 2022 were talking about sea change revolution even like google for instance is i mean google's fine bing has with ai hasn't just taken it over right no it hasn't really made a dent so i'm curious what you think i think it's always funny to have big tech executives like matt would say the name of the game is incremental and slow because they can afford to be incremental and slow because they have giant profitable businesses to allow them the time and space to see something play out over a number of years, like a decade.

14:53Whereas a startup can't do that, right? Like you can't go into Andreessen Horowitz, Benedict's former employer or Sequoia or any of these guys and be like, we got a business plan and that business plan is incremental and slow. Like that's not the way the tech industry works. And so I think a lot of the hype around this technology was driven by the financial needs of investors to see fairly fast returns and uptake on this technology. And like, we've all sort of been a victim to that in a certain sense. But can I make a counterpoint here? Because hasn't this also like largely been a scale game? It's been a moment dominated by the Microsofts, which have invested in OpenAI and the Googles, which have their own, you know, AI research houses and Amazons with their investment in Anthropic and NVIDIA, which can make the chips and you throw it.

15:42Okay. So who's the upstart? Is it meta? Which is like, you know, sort of come from nowhere to make its own open source model. Like it does seem more than most moments in technology that this has been a moment that favors scale and the big guys. Oh yeah. No, it's been a huge benefit to, uh, the entrenched tech giants. They have gotten, I mean, Microsoft has gained more than a trillion in market cap. NVIDIA is now one of the largest companies in the world. Oracle, which is a company that you didn't think much about in the cloud game, has benefited tremendously all of a sudden because of this. So the spoils have absolutely gone to the incumbents, which is, again, kind of funny in this industry that's supposedly driven by disruption and a changing of the guards every couple of years because new technology comes out.

16:33But, you know, we also should be fair that like open AI is a real, they may not be profitable probably, but they have real revenue. They're in the billions because they do run like the most popular, you know, Chat Chat GBT and GBT4 and Anthropic is doing pretty well. Like there is real money that is going into a lot of these companies. Like I don't want to overlook that. but I think the sort of ecosystem-wide disruption that the startup and venture capital community sort of expects when there's a technological shift has been pretty slow and tricky and I think in the next couple of months we're going to see, we've already seen some flame outs of these startups like Inflection had this bizarre aqua hires type situation over at Microsoft but it's straight up a situation of a company that didn't work out.

17:28Adept, which was an enterprise software AI agent company, they sold to Amazon in a very similar way. And I think we're going to see more of that in the next couple of months. So as like a kind of startup ecosystem, it's not the healthiest. So there's something else here, which is sort of gets to this second half of what benedict was suggesting and i'd love to hear your thoughts on it this idea that like llms will inevitably scoop up whatever is built on top of them we've talked a lot about like how it's like a chat gpt is everything just a chat gpt wrapper right and so like the other side of this you know are we just waiting for product market fit is like do startups inevitably end up just being eaten by the big models as they get bigger And is that another advantage towards scale?

18:19Well, that's, I guess, something I've been writing about recently. I mean, this idea of scale and that's like just size of model, you know, like these giant models like GPT-4 or when Anthropic is built or Gemini, which is Google's large language model. These are very powerful, large technologies. They have to run in like, you know, massive cloud infrastructures, but they also do a lot more than what individual businesses really need. like if you're trying to build like a chat bot to give financial advice or something you don't need gpt4 uh to at the same time give financial advice and also give you like a recipe for flan or or something okay but and i brought this up with wood like okay bloomberg spent all this money to train bloomberg gpt with i think they use amazon technology to do it and then gpt4 ended up giving uh just as good financial answers as the custom bloomberg bot once it came out that's the that's this is sort of the balance here well yeah i mean i think i mean to go like into like the specifics of these small models and large models no one is should reasonably say a small model is better that's not true what a small model can do is just be attuned specifically to the needs of its user and so uh if you have like the small model plus trained on specific data sets it'll be effective and like more cheap to run it's really just like a an efficiency play uh but you could also train a large model on or tune a large model on specific data and it'll be just as good i don't i don't know this bloomberg example uh the bloomberg gpt it's like a chatbot where you can talk with Bloomberg data basically.

19:58Okay. Financial data. That's fun. Yeah. Um, yeah, I, I think like the large model is in one sense, a very cool demo technology and it can do lots of stuff and you know, all the things that Benedict Evans points out in terms of its capabilities and shortcomings, like it's all, it's all true in there. The one thing it is also is incredibly expensive to run and, uh, uh, kind of a loss leader for, um, for these companies, like just The cost to build is in the hundreds of millions of dollars. You know, each, some of these companies, like each inference, each time that you put a request into the API for them loses money for the company because it costs more to like in process, you know, cloud computing costs than it is, you know, than they charge customers to be able to do it.

20:47And so there's a real desire, I think, on a lot of businesses to find a way to do this more cheaply. and so that's why they kind of have been focusing on these smaller and dumber models that are good enough to pull off the task but are also uh efficient and and not going to lose you a bunch of money so is the idea that they can sort of outflank the bigger models by doing the same things cheaper or is there also is that what it is yeah i mean like the analogy that i used in the story that that we published a couple weeks ago was like you don't need to drive a tank to go pick up groceries at the store and there's like a million in some counties in the u.s that's sort of the way that people do it maybe it's recommended i don't know i guess like that humvee era people were doing that pretty actively in the suburbs uh you definitely didn't need to do that no though um but uh whatever you don't need a lamborghini to get the mail i mean you can you can think of a million different analogies but the point is like the cost of running these things when what you need the output from it is so hyper specific and doesn't need a huge model uh is leading people toward using these smaller models and um it it's a funny moment to me because it also runs in the face of like agi and this desire for these companies to build you know a replication of human level intelligence because if you think about it as like a researcher and you got into generative ai a lot of people were like like let's go dude let's go like replicate human level intelligence let's build the biggest possible model we can we'll run it on supercomputers the likes of which have never been seen before and we're gonna like we're gonna do it we're really gonna make like a robot brain uh we're gonna make her and uh that's fine they haven't done it yet clearly but the business imperatives are also saying yeah why don't you build a not human level intelligence why don't you build a small dumb thing relatively uh that can you know be give Bloomberg financial advice.

22:45And, and that is just not that inspiring to those people. I imagine like they kind of, they didn't get into this game because they were going to build like capable, competent chat bots. They really thought they were, they were building something, you know, transformative here. Yeah. And it's always the economics. That's what points the direction of the technology. Like as much as people would like to say, it's not about business or whatever businesses, what drives the development of this tech. I think so. I mean, if you can build AGI would probably be good for business but in the short term you're right like it's a lot of these change management incremental get it in the workflow type of thing that's not exactly as inspiring to somebody as you know building a human brain might be right and it's also probably harder to make those arguments to enterprises to like IT directors to CIOs and all those people to make you know huge investments when we're talking about is incremental productivity right incremental capabilities but do you think there's going to come a point where this technology is going to get good enough where like you won't have to like for instance spend all that time uh standardizing the data where it just sort of is good enough that it can sort of work much more easily than it does today uh yeah i do honestly i i don't i don't i'm not stupid enough to give a timeline for it right but i like and i guess that would be my slight push back to benedict evans i know he was is raising the question not really taking a side but chat gbt is pretty good at a lot of stuff like i i forget this from time to time because i don't use it all that much but when you ask it to do certain types of things very specific at times like um as i was writing a story i wanted like a a quick summarization of what uh ec2 is this like aws technology that is elastic computing thing, just like extremely, uh, inside baseball, almost esoteric thing.

24:41Um, you know, I Googled it and it was not finding a great, like one or two sentence summarization of what this product was. When I put it in a chat GPT, it gave me a pretty good thing. And I fact checked it cause I'm, you know, I'm not that stupid. Like I, I think we all should do that. But when it does things like that, you do sort of see a bit of the magic that got people excited. Coding, of course, is maybe a whole other category where it's shown capability. So I get kind of annoyed at times with the maximalists on the other side who just say, this is a dead end. This didn't do anything. This was a huge boondoggle waste of money on the part of Microsoft and everyone else going into large language models.

25:25I don't think that's true either. I think there's something really there. But the timeline, like we've been saying, is maybe going to be a lot longer than we initially thought. Yeah. And on the Bing front, I mean, you covered Microsoft. We brought up the Bing challenge to Google. What do you think has happened there that it hasn't been able to be successful? I mean, it didn't work. Like, it did not change the way people search online. I should say it's obviously caused effects. It completely messed up Google, the Bing chatbot. The chaos that the initial release of Bing chat costs within Google was tremendous, and all the reorgs and internal pressure and code reds and stuff that they've had to do because Microsoft has jumped ahead in this world has been real.

26:12But the market share did not shift. I'm sorry. You can show me any number of different third-party studies and claim that these things are noisy, But bottom line, this was not like a huge reordering of the guard in terms of search. And I think that was a surprise to Satya. I really think I was there in January, February of 2023 when they rolled out Bing chat or Bing with chat and Sam was there. And I think this was really positioned as like an iPhone level moment where we would all look back at this and think, oh, my God, how did we ever live before Bing chat existed? and it just didn't happen.

26:53And I think it's totally reasonable to call out Microsoft and Satya, who I know is like, and I've written about him, like, you know, considered one of the most effective CEOs of the current era and, you know, gets a lot of credit for turning Microsoft around and stuff. But like this should absolutely, for the time being, be categorized as not the win that he thought it was going to be. Totally. Yeah. And like, we'll see what happens. There's this new team in there that's supposed to be, you know, rebuilding it all and re-skinning it and seeing if there's going to finally get some consumer attraction.

27:26But, uh, it's just, we should be honest about that and say that this didn't work out. Right. Can I actually ask you a quick question on, on your usage of this stuff? I mean, you know, you put together this run of show thing beforehand, his listeners can get a little behind the scenes where you kind of listed out the different topics you wanted to cover. I mean, that's easily the kind of thing that you could have put into chat GPT, right? Or at least any of these, these large language models and spit out like, Hey, summarize Benedict Evans post and talk about the other things we're going to do.

27:59Like you could have gotten this thing to spit out a summarization of this outline of the, of the episode. You didn't look like you did. I mean, do you ever consider using this stuff? Like, is this, is this something that could change your workflow at all? No, for my workflow, like I need to be reading the stuff. Like I need to be in the documents and picking out the points. I mean, I'll do things like upload lots of interviews to Claude and then ask it like, you know, what am I missing? Or, you know, this week I took a bunch of interviews, uploaded it to Claude, and then I uploaded my story and I said, what did my story leave out?

28:34And like, you know, it's much more effective after the fact versus proactive because I just think that the human touch is still more important. Honestly, I think I can do a much better job than the AIs of picking out what we should be discussing versus what we shouldn't. It would obviously be faster. You don't mind spending the extra 20-30 minutes putting the outline together. I spend longer than that. I also think that without that work ahead of the time, the show wouldn't be as good as it is. or the show would be worse let me put it that way you know yeah well that's such a that's interesting because that's sometimes the argument from these guys is like well it's not as good as a human but it's only you know it's only it's 98 or it's 90 and if we're willing to like deal with that then you actually get increased efficiency and only a slight degradation in quality which i think is kind of cynical uh if you think about let me let me explain it this way progress like with podcasts, it is zero sum.

29:36Like if you have a choice of listening to a show, that's like a hundred percent of what it is or 90 % of what it is. And there's another show that comes in and it's 95%, like what you've just given up in efficiency, you've lost your entire audience because people will go to the show that's 95 % as good versus 90. So it's basically like the job is just to make it the best quality show possible and sort of, you know, that's why I like the branding for this tech like microsoft has always been like it's a co-pilot like it's something to sit alongside you it's not autonomous which is again ironic because the different levels of what open ai is going for is not just a co-pilot it's a pilot like they really want this stuff to be able to handle all tasks on their own um and so i mean the technology doesn't allow for that yet but we'll see like uh at this current moment it's a co-pilot that people aren't all that excited about at a grand scale.

30:31And there's an acknowledgement that there's also like a quality gap there. Yeah, definitely. It's a hard sell. It is a hard sell. And so it sort of is a great lead into our last segment of this half, which is talking about this post that David Kahn from Sequoia came out with AI's$600 billion question. And he basically said effectively that But AI is going to have to generate, you know, 500 billion dollars more in return to make up for the massive amount of investment that's going into it. And he cites some risks there here for the lack of pricing power. Basically, like if this is all going to be commoditized, you won't be able to make a lot of money off of the models that your investment incinerates, maybe because of training or because people will lose money in speculation.

31:25depreciation the models get worse over time and then the winners and losers like if you're not a winner you're in you're in rough shape and here's what he says he says speculative frenzy is a part of technology and so they should they are not something to be afraid of those who remain level headed through this moment have a chance to build extremely important companies but we need to make sure not to believe in the delusion that has now spread from silicon valley to the rest of the country and indeed the world the delusion says that we're all going to get rich quick because agi is coming tomorrow and so we need to stockpile the only valuable resource which is gpus in reality the road ahead is going to be a long one it will have ups and downs but almost certainly it will be worthwhile i think that's sort of like the most like cool-headed take on this entire moment that i've read so far yeah well that's the theme of the podcast indeed i i guess that is it's also the optimist take on it and it does still rely on an assumption or a null hypothesis that the technology will improve uh at some sort of a rate uh and you know that goes against what the real critics of it like gary marcus will say which is like deep learning large language models are going to hit a ceiling and you can't just throw more data and more scale more gpus at this to you know to reach the level of breakthroughs that need to occur for it to be as valuable as that extra, what was the number?