AI Adoption Challenges, Waymo’s Big Expansion, Amazon hits $2 Trillion

0:00Companies that spend big on AI are struggling to get it working reliably. Consultants, meanwhile, are cashing in. Waymo is ditching its waitlist in San Francisco and opening up to everyone. And Amazon hits$2 trillion in market cap as it teases a Timu and Shein competitor. All that more is coming up right after this. Welcome to Big Technology Podcast Friday edition, where we break down the news in our traditional cool-headed and nuanced fashion. We have such a good show for you today. We're going to talk about the practical issues that companies are running into as they try to implement AI in their workplaces.

0:34along with an important update on RoboTaxi expansion, and then this big moment for Amazon where it hits$2 trillion as almost nobody's actually thinking we're talking about it, but we will. All right, joining us all the way from Taiwan is Ranjan Roy. Ranjan, welcome back to the show. I'm in southern Taiwan. It's 97 degrees here, but thanks to the miracle of the modern internet, still able to do this podcast. Yeah, it's great to have you back, and we have a lot to talk about, including at the end of the show a little bit about how one of your posts has gone viral again hitting the top post on reddit but let's begin with this issue that companies are having with ai uh assistance right so this is from the wall street journal ai work assistants need a lot of handholding it says artificial intelligence assistants were designed to provide businesses with a relatively easy avenue into the cutting edge technology but it isn't quite turning out the way that a lot of people hoped, with chief information officers saying it requires a heavy internal lift to get the full value of the pricey tools.

1:38And there's some incredible examples here. So one executive said that if she asked a question related to 2024 data, an AI tool might deliver answers based on 2023 data. At Cargill, the Wall Street Journal says an AI tool failed to correctly answer a straightforward question about who is on the company's executive team. At Eli Lilly, the tool gave incorrect answers to questions about expense policies. Here's a great quote. I remain an AI optimist and I'm confident that we'll get there. It's just taking a little longer than perhaps we thought. So here's what's happening, right? We've said that the biggest amount of spend on AI is going to come at least initially in the enterprise.

2:15And now enterprises have spent that money. And one of two things are happening, or maybe there's three outcomes that are happening. One outcome is they just have this AI money allocated and they don't know how to use Two is they are using AI. It's not working very well. Three is they're spending the money and it's working incredibly well, but we just don't hear often about those stories. So I'm curious if you think we're going to head right into the trough of disillusionment here because we are seeing these companies. They bought into the hype. They spent the money and they're clearly not getting their money's worth when it comes to putting this stuff into production.

2:52I want that trough. Give me that trough of disillusionment. You might get it, man. No, no. I've been waiting. I've talked about this, and I've been wanting this to happen. And these exact examples in the Wall Street Journal story are so perfect, because this is what I've been saying. Anyone who has used any kind of generative AI tool, especially that does any kind of RAG or retrieval augmented generation, basically where you use a generative AI layer over a preset existing amount of data, it does not work perfectly. especially trying to do calculations and numbers, it works even less perfectly.

3:31So I think people are starting to understand if the data is not well-structured or very clear, and if the questions and the prompts are not clear, you're not going to get the right answer. I would say a year and a half ago when I started really doing a lot of work with generative AI, there was this idea and promise that you can just connect it to a BigQuery database. You could upload 100 PDFs and it will get it perfectly. And I remember uploading a bunch of bank analyst reports and it could not get anything at the time a year and a half ago, especially trying to be like add up the revenue across different reports.

4:07If it was in a PDF format, no shot. So I think it's good because the more I've learned to work with this technology, you start to understand, okay, come up with more finite sets of data and questions, understand the right questions to ask. Like once you start to use it and learn to use it properly, it will work. But these companies that have gone in headfirst at 100 miles an hour, I think there's going to be a lot of trough, a lot of trough. But these are not dumb companies. We were talking about Eli Lilly and others, right? They have smart people working inside them, presumably. If they can't get this right, are we going to start to see a point where for most companies using AI tooling, it's going to be harder to actually put this stuff into practice than to do things the quote unquote old way?

4:55And then what happens if that's the case? Yeah, I think this is going to be a really interesting next six months or so, because as you said, they've invested the money. But what's actually happening in a lot of these companies is you'll have some innovation team or it'll actually live outside the business unit itself that's creating the product. Then they go to the business user and say, hey, do you want to ask it some questions? Are you happy with the output? And then they will ask it something and then get something that's not perfect. And out of maybe 20 questions, one is wrong. Send it back to the innovation team and not be happy about it.

5:33So I think it's less about a company is dumb, though I think massive organizations have inefficiencies. But I do think it's the way companies have approached setting up these projects from everything I've heard and seen is it's almost asking for these kind of challenges. And it's going to take a lot of will to overcome them because I can see a lot of groups will just kind of write it off and be like, OK, we tried, we invested, I checked off that box and now I'm good. Right, exactly. And so this is sort of the sort of obvious business question that is just begging to be asked, right? A lot of the we've saw ChatGPT come out November 2022.

6:10We saw a lot of companies develop their AI strategies in 2023. We're heading into summer 2024. We're coming up real close to like, a lot of the windows of where renewals are going to be on the table for companies. hey, like you have spent this money on our AI tools and internally, how's it working for you? Do you want to spend more? Do you want to renew? And this is going to go directly both to the bottom lines of research houses like OpenAI and Anthropic, but also like these big cloud companies, right? Like AWS, for instance, and the Azure's and the Google Cloud, which have all basically turned in better than expected results because of the AI sales.

6:53So do you think in the short term, we could actually see a dip in these companies' revenues because renewals just aren't going according to plan, or at the very least, a slowdown of growth. I think a slowdown of growth is definitely in the cards. And it's going to be interesting because, again, Microsoft was able to, I think, attribute around a billion to AI services. Google did not break it out for their cloud, but still kind of said, we are growing our AI-related cloud services. So I think if it's a dip, that's going to be very, very bad. But I think a slowdown of growth is inevitable because you'll have certain – if you're a cash-rich large company, you will still be able to invest in this.

7:38But if you are in cost-cutting mode, one of the first things you'll cut, I think, will be, okay, here's this technology that we were promised. I'll be able to ask it any question. It'll get it perfectly right on day one. And when that doesn't work and it's renewal time, I think that'll definitely be an issue. But I think that idea of are companies dumb, I think I'd be curious to ask you, do you think this approach is dumb? Or do you think these companies have just like, what error have they made in the way they're approaching generative AI? Well, I just think that change management is difficult.

8:14It's very difficult to implement change inside an organization. and there's like you talked about there's innovation groups that are owning it there's no natural group that's going to own artificial intelligence within a company because on one hand it's very department specific but on the other hand it's very strategic for a broader company so do you have that broader like innovation group that's sort of handing it top down and saying oh by the way that database that you've been querying for 10 years now you're going to type it in and it's going to get it wrong sometimes like trying to get somebody to switch over like that's not very easy at all.

8:46So I think that it's just a change management sort of issue slamming into a hype issue. And we've talked about this a lot. Some people are already calling AI human level intelligence. We've had people on the podcast who've worked at these big tech companies who believe it's sentient, right? So I'm just saying it is what it is, right? And so you're going to have this hype and change management run into each other. And here's where I think we could really come to a head. And, you know, if there's ever going to be a moment where the air comes out of this AI movement, it might be this. When GPT-5 is released, because it is going to come, you know, whether it's later this year or sometime next year, when GPT-5 is released, it's going to be an improvement, but I don't think it's going to be, and we've talked about, it's not going to be the human level AI that some people are believing it's going to be.

9:32It's just going to be like a more improved GPT-4, right? The next level of models are going to be an incremental advancement, not a revolutionary advancement to what we have, even though they're going to be quite impressive. And when people see that and they're going to be like, oh, I have the same issues with GPT-5 and let's say chat GPT that I did with GPT-4 or 4.0. And it's not really working as well as it could be within the business. That's when you might see some of the questions come out. And I spoke about this with Michael Batnick on Wednesday. Where's the ROI on this? We haven't really heard that because, you know, there's so much promise, but there's going to come a point where that promise sort of tops out if these factors continue to go on their current trajectory.

10:15And that's where you can see some of the big questions here. Well, one thing I've been noticing, though, and as someone who has been saying the real promises in the enterprise this whole time, is actually the increased utilization of all these tools in just the day-to-day individual work people are doing. Like so many more conversations. I hear people talking smartly about how they're using ChatGPT and uploading CSVs to do some basic data analysis, or a lot more people I know are using perplexity to just answer questions. So I do think maybe there is a world where, as an individual, you throw in a CSV and ask it to do some data analysis, and it's not some massive, multi-billion-dollar, 12-month project that's going to give you the answer.

11:03And then if it's not perfect, you're like, okay, maybe I'll try another way versus Cargill. Someone asks, it gets the wrong answer. And you can imagine the, the shit storm that follows. So, so I think it's almost maybe a positive name. But sorry, go ahead. No, no. I mean, but, but exactly. It's like, but if, if it's just you sitting there, you're like, okay, that was kind of funny or ridiculous. Let me ask again, because this is going to make my life easier and I don't want to have to do this all from scratch. And that's actually good. And I think it's so maybe it will be more at the individual level within companies that this needs to be pushed.

11:45Because I do think this model of having an innovation group to a six month proof of concept, it's just it's rough. And it's going to end up in a lot. It's almost going to, the chance or the likelihood of good outcomes almost gets lowered if this is the way people approach it. And that's a crucial bit of context. And I'm really glad you brought it up, thinking about the individual level, because the one thing that you can't take away from this technology is that it is extremely powerful. And it is extremely useful if you know how to use it in the right way. and so no matter how quote-unquote dumb companies might be on an organizational level there are going to be people within companies that are going to find out ways to use it in ways that are productive and that's going to make a very big difference and so like when i talk about the air coming out of the of this moment you know notice how i don't refer to it as like a bubble bursting because it's clearly not like it's just you know potentially that some of the aura of it will deflate and maybe this is going exactly to what you want to happen is that people will look at it way more realistically.

12:48Just rein it in just a little bit. That's all I ask. But you know, I mean, if this is a change management question, do you know who is great at selling solutions to change management, Alex? Well, Ronjan, you just nailed the segue. I think that might be the best segue in show history. Now I'm ruining it by talking about the segue, which is rule number one about what you don't do to them. You broke the fourth wall of segues. Lord almighty. But let's talk about consultants because they are so good at change management or so they might say in 100 slide PowerPoints that talk about the change that they're going to bring to your organization.

13:22The New York Times has a story talking about how wonky consultants are the unlikely early winner of the AI boom. Now, I wouldn't call them unlikely. And this has sort of been shared on X by more than one person. But like consultants always win in this situation, but they are winning so big. And I know we have a lot of consultant listeners, so I'm going to try to be gracious here. But it is interesting how much of the early winnings are accruing to the Boston Consulting Group, the McKinsey and companies, the IBM and the Accenture. This is from the Times. Companies are in desperate need of technology Sherpas who can help them figure out what generative AI means and how it can help their businesses.

14:04By the way, if you are in a meeting and you say we need a technology Sherpa, you're in trouble. I'm just going to say that's one thing. OK, you're in trouble. OK, now here's from the story. While the tech industry is casting about for ways to make money off generative AI, the consultants have begun cashing in. IBM, which has 160 ,000 consultants, has secured more than one billion in sales commitments related to generative AI for consulting work and its Watson X system, which can be used to build and maintain AI models. Accenture, which provides consulting and technology services, booked$300 million in sales last year.

14:41About 40 % of McKinsey's business this year will be generative AI related. That's insane. 40 % of McKinsey's business generative AI related. And KPMG International, which has a global advisory division, went from making no money a year ago from generative AI related work to targeting more than$650 million in business opportunities in the U.S. tied to this technology over the past six months. the management consulting industry in the United States is expected to collect$392 billion in sales this year, up 2 % from a year ago. Okay, so obviously the amount of money that they're making pales in comparison to the amount of money that the big tech companies that we talk about on this show make in a quarter or a year, whatever it might be.

15:23But it's still clear that AI has been just this massive, massive boom for these consultants. Is this a good sign, Ranjan, or is this a sign of concern? I think this is a terrible sign. I think this is actually the exact worst possible outcome of this, because if anyone can remember, and I say this as, again, to one of our consultant listeners, I think there's a great deal of value generated by large consulting companies, but the scale and speed to which they have pivoted to generative AI, If you just look at, and I have like RSS feeds of McKinsey and BCG and Bain and everyone, the amount, I mean, that would be an interesting analysis to do with generative AI.

16:08What percentage of headlines have determined it? Because it's skyrocketed. And I mean, the most fascinating part of this is if there was an industry that would be under threat from generative AI, it should be management consulting. Because taking a lot of kind of, you know, diverse concepts and data numbers and ideas and being able to condense it into a smart sounding, short, concise argument is the business of management consulting. And that is literally the thing that an LLM does better. It might get some stuff wrong and a consultant can get some stuff wrong. So it's amazing to me because I remember two, two and a half years ago, McKinsey is under threat like consultings.

16:57The entire industry is going to get killed by LLMs. And kudos to them for actually turning it into an opportunity for themselves and making – and you said it pales in comparison. But remember, if KPMG is making$650 million, I think Microsoft, it was only$1 billion that was attributed to AI services in the last quarter. And that was in a quarter versus a year. But still, it's a significant amount of money in terms of actual cash being exchanged for any generative AI in the current market. Yeah, I'd be curious to see how many of every dollar spent on generative AI is going to consultants. It must be a lot.

17:38And it's a very interesting perspective that you bring up about how this might eventually replace some of the work that consultants do. I haven't fully, I mean, I haven't thought about it that way, but it is interesting because I have like at times taken like all of big technologies, you know, data from like the back end of like Spotify and Megaphone and like plugged in, like it comes in spreadsheets, right? Like the amount of downloads and the headlines that we use for the shows. And I've also plugged in like big technologies, the newsletters data and been like, what should I do differently? What's working really well?

18:11Give me some takeaways. And it is able to do that. And it's like, yeah, that's something you might hire a consultant for previously. It's pretty interesting. So maybe this generation is not going to replace them. But maybe they are kind of on like the easy pickings in terms of like where you'd see AI start to cut in on their margins. Well, because if you think about it, like the true value of the management consulting industry is being able to have basically, you know, an insider look or just a very smart understanding across different companies where most companies only are looking at themselves myopically.

18:46so the management consultant can come in and give them a greater perspective about the world and their business challenges, understanding those challenges from other companies. That is, again, what does generative AI do great? It knows the whole internet and is able to kind of pull all that information together and parse it and analyze it. So I think, again, I would have thought already the management consulting industry would be seeing some genuine threat from this technology. And they've completely turned it around the other way. Right. And it is interesting that they're only going to grow 2 % this year, which is pretty anemic.

19:25And if you took away AI, they would be contracting. Yep. But if you remember, actually, the most surprising name in this for me was IBM. And to hear that they have more than$1 billion in sales commitments related to generative AI. And to even hear that, and it was related to its consulting work and its Watson X system. I did not, we've not talked about Watson X in this show. I know. I kind of think maybe we need to get into Watson X a little bit because I need to know more. I remember going to a conference in 2015 for Watson and they had this booth set up and it was these chocolates were invented by AI.

20:16And it was like we fed a bunch of ingredients into AI and gave us these unique recipes that we've created this chocolate off of. So they've been pushing this for years now, for a decade almost. So I was surprised to hear that they're still going strong. Yeah, I couldn't believe that. I mean, maybe this is going to reveal a level of ignorance in me that I'm not happy to reveal. But I couldn't even believe the name Watson is still in use. and maybe it just goes to show you about like the relevance of IBM but if yeah why don't we do some digging and come back and talk about it on a future show maybe do we need a if listeners would like an entire Watson X episode hit us up in the comments five stars ask for the Watson episode we'll do the Watson episode this is going to be the show of revived formerly dead tech companies Yahoo episode I think the Yahoo episode was a hit we'll do a Watson episode I think and the The Yahoo episode not only was great, I mean, Yahoo's getting more interesting right now.

21:16So maybe this will bring Watson X back into the conversation, at least for us. That's what we're here for. The one interesting one, just to wrap this with a bow, we talked a little bit about how consultants are good at doing this stuff in our beautiful segue, but actually it turns out that they're also having problems. So this is also from the story. The results have been mixed. Generative AI is prone to giving people incorrect, irrelevant or nonsensical information known as hallucinations. It is difficult to ensure that it provides this accurate information. It can also be slower to respond than a person which can confuse customers about whether their questions will be answered.

21:53And this is from IBM, which has this$20 billion consulting business, ran into some of those issues on its work with McDonald's. The company developed an AI-powered voice system to take drive-through orders, but after customers reported that the system made mistakes like adding nine iced teas to an order Instead of one diet Coke, McDonald's ended the project. Maybe that's what's going on with Watson X, Ronjan. Maybe we don't need to do it. Watson X just likes iced tea. We just want you to have iced teas. Actually, AI is smarter than us. It's like, you know, all the chemicals in that diet Coke. You know you need some caffeine to have these nine iced teas.

22:30AI is sentient in trying to tell you what to do. Yep. It actually did a calculation. the amount of caffeine in one Diet Coke perhaps correlates to that amount of iced teas. Can you imagine the amount of sugar in nine McDonald's iced teas? No, I cannot. But I can ask Perplexity to make that calculation for me. You can, but you might not want to because Perplexity, your favorite of the Gen AI companies, has come under fire. Wired had this great story called Perplexity is a Bullshit Machine. basically it makes two arguments one is that perplexity is ignoring robots.txt which basically is a signal from website developers to not crawl and they are actually able to find the secret ip address associated with perplexity that they were able to test by effectively like creating discrete web pages and then asking perplexity what's going on uh all these pages and then perplexity's bot uh pinged it so like sort of gave up its secret ip address it's a very smart investigation.

23:34And so sometimes perplexity is actually going to these pages that have instructions specifically for crawlers not to crawl and crawling anyway. And sometimes it's not crawling and just making things up out of thin air. And this is continuing on with our sort of like copyright questions about like our generative AI company is going to start to hit serious copyright issues. And it's been a major rolling story. We obviously talked about their issues with Forbes. Rajan, I'm curious what you think about this latest round of reporting on perplexity and whether it concerns you as a true Perplexa fan.

24:11As a true Perplexa fan, I think it's going to get interesting. And I've been wondering about why it was Forbes, now it's Wired. Everyone is choosing perplexity to go after this publicly. And the more I thought about it is, and for anyone who uses perplexity, it's still the best at kind of presenting the information in almost an original article format. Whereas ChatGPT, there's no images. It's just kind of like very raw text. Perplexity really formats it nicely and makes you feel like you're reading a news article. So I think that's why publishers are probably taking it almost visually as more of a threat.

24:54But I mean, this did not look good for perplexity if they're actually still trying to circumvent kind of standard practices of the web and basically hack their way to that content. And when we had talked about this the other week, to me, it was, and again, as a perplexa fan, they should not be scraping paywalled content. Unfortunately, maybe you should not answer that story around Eric Schmidt and his drone company. You just don't get that one. And if someone asks... How long did they not answer it? They featured it. Yeah, they featured the image that was actually from the original article. So I think it's a reminder, but remember OpenAI, there's the whole little kerfuffle around how they basically scraped all of YouTube and transcribed it.

25:38And so every, I mean, every one of these companies has essentially trained or been, you know, like conditioned with the behavior to be as aggressive as fast as possible. And I do think it's going to cause a lot of problems because perplexity, especially because I ask it a lot about more current events, what's happening in The French election that's going to like the special election. I've actually I've learned a lot about what Macron is trying to do. And it was very good at explaining these things. If they lose access to all news, then anything new is less interesting. So I think there's going to have to be more and more at least humbleness in the way they approach things, if not coordination.

26:18And we talked about how there are vague promises around paying publishers or something like that. But I think this is going to be, for perplexity, a major issue. Now, Ranjan, you talked about how every one of these generative AI companies, and maybe this is painting a bit too broad of a brush, but most of them are really scraping extremely aggressively and kind of being opaque about where they got the data from. And they're all training on copyrighted material and telling us basically like, shut up about your copyright questions. I mean, that's basically the sentiment. yet they're all talking about how they want to develop AI ethically and they're very concerned about its impact on the world would you say that their disregard for intellectual property law and norms is ethically compromised and if so can we trust them on the ethics that would apply across the board because if they're ethically compromised in one area why should we believe that they're going to be ethically on the straight and narrow in the other areas.

27:25So the whole publisher issue here, the one argument I have found a bit, I don't know, that, you know, that is in their favor is the idea that still, and we discussed this the other week, that many publishers still built a lot of their own businesses off summarizing and not copying, but rewriting other original content, which is essentially what Perplexity is doing and all these other AI services faster and at scale, rather than having some junior writer be tasked with, okay, the Wall Street Journal has published something. Now go write it in a few different bullet points or a slideshow and let's put it on our website.

28:05Not naming names there, but I think – That sounds like Business Insider or maybe even a publication that I once worked for. But sorry, go ahead. I think maybe. And that's why publishers have operated. So just the idea of taking something that is out on the internet and reframing it in a different way, obviously copying verbatim is always going to be bad. So it is not the most unethical thing. I think it's problematic for the long term kind of viability for the business of perplexity. But I do think like, yeah, out of all things, it's not necessarily the most problematic in terms of where these companies could operate unethically.