In short
The episode breaks down big tech earnings and AI-related developments. Topic: Meta raises 2026 CapEx guidance to up to $145B; Microsoft reports accelerating cloud and discloses $9B+ AI revenue and 20M M365 Copilot users; Google Cloud and AWS backlog acceleration; Elon Musk’s OpenAI trial cross-examination; Starlink revenue per user trends; and GPU utilization inefficiencies.
Guests
Brent Thill (Senior Analyst, Jefferies) covers Meta/Google/Amazon earnings. Rishi Jaloria (Managing Director, Software Equity Research, RBC Capital Markets) covers Microsoft. Rocket Drew/Elon Musk reporter (courtroom updates). Theo Waite (Starlink revenue reporting). Stephanie Palazzolo (GPU utilization reporting).
Key claims/examples
Meta’s ad growth guides (high-20s next quarter) and “opaque” ROI/wearables (glasses) drive investor worry; Google Cloud growth ~70% and $219B sequential backlog add; AWS signs a $100B Anthropic deal and improves margins; Microsoft AI revenue is likely conservative and Copilot adoption is early (pilots/POCs). Musk appears “not looking so hot” on AI safety basics (system/safety cards, preparedness framework). Starlink ARPU falls $99 (2023) to $81 (last year) as customers quadruple; price cuts drop cheapest plan from ~$120 to ~$50; Amazon Leo could pressure pricing. GPU utilization: XAI model flops utilization ~11% despite chip scarcity; reasons include bursty training, data/memory bottlenecks, cluster network weak points, and “gaming” by rerunning/deleting experiments; Cursor may help fill workload gaps.
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Chapters
Tap a time to open that second in VOAnalyzing Meta's Earnings and CapEx Forecast
0:54 to 2:46
Discussion on Meta's increased CapEx forecast and investor concerns.
“We had a slew of big tech earnings Wednesday afternoon, including Alphabet, Amazon, and Meta.”
Exploring Google's Impressive Growth Metrics
2:46 to 4:51
Insights on Google's cloud growth, market share, and ad revenue.
“I think Mark Zuckerberg didn't do himself any favors when he said they don't have a precise plan for monetization.”
Amazon's Strong Performance and Future Outlook
4:51 to 9:37
Overview of Amazon's earnings, cloud growth, and advertising business.
“And ultimately, the real ROI is still really opaque.”
Microsoft's AI Revenue and Market Position
9:37 to 14:00
Analysis of Microsoft's AI revenue growth and its impact on the market.
“That was Brent Thill, Senior Analyst at Jefferies.”
Microsoft's AI Adoption Strategies
14:00 to 17:40
Learn about the metrics driving Microsoft's growth with M365 Co-pilot.
“This gets right to the growth of the company, right?”
Elon Musk's Cross-Examination Updates
17:40 to 22:20
Get insights on the intense questioning of Elon Musk during the OpenAI trial.
“That was Rishi Jaloria from RBC Capital Markets.”
Starlink Revenue Insights and Future
22:20 to 28:00
Explore the trends in Starlink's revenue per user and its implications.
“Theo joins me now to share what he knows.”
Starlink's Valuation and Future Prospects
28:00 to 29:25
Explore the significance of Starlink in SpaceX's valuation and growth strategy.
“And, you know, for what it's worth, you know, SpaceX says Starlink is getting more efficient.”
GPU Performance and Utilization Challenges
29:25 to 31:34
Learn about the challenges AI developers face in utilizing GPU resources effectively.
“That was Theo Waite, our Elon Musk reporter here at The Information.”
Strategies and Issues in AI Training with GPUs
31:34 to 36:24
Understand AI researchers' strategies for GPU utilization and the challenges involved.
“out of those chips but if there's such a scramble for chips how can companies be both short on GPUs and still not fully using them.”
Transcript
Automatic transcript. May contain errors.0:13Welcome, everyone, to the Informations TITV. My name is Ken Brown. It is Thursday, April 30th. We have a great show lined up for you. We're talking to analysts about the big tech earnings, Microsoft, Meta, Alphabet, and Amazon. We'll also check in with our AI reporter, Rocket Drew, who is covering the Musk OpenAI trial. Plus, the information published exclusive reporting about Starlink revenue. We'll be joined by our Elon Musk reporter to discuss that. And we'll wrap up the show with a look at the NVIDIA chip crunch and how AI developers are utilizing GPUs that they do have. It's going to be a fun show, so let's get into it.
0:54We had a slew of big tech earnings Wednesday afternoon, including Alphabet, Amazon, and Meta. We're going to break it all down with Brent Thill, a senior analyst at Jeffries. Brent, let's start with Meta, raising its 2026 CapEx forecast to as much as$145 billion. That appeared to worry investors, the stock's way down. What did you make of the number? Yeah, the worry is really the guide they gave on the ad business for next quarter growing high, high 20s and the guide into the mid 20s. And then to your point on the CapEx, the higher guide and ultimately where's the return going to come? I think we saw the first wave of AI investments in Meta improve the core properties of Facebook, Instagram, WhatsApp.
1:40But I think investors are questioning, you know, this next leg of investments. are the glasses going to pay off? Are the new AI investments in some of the core properties going to really take this to the next level? Or is this Zuckerberg wasting money on projects that aren't going to produce the type of ROI? And so I think when you match Meta up against Amazon, Google, Microsoft, and you see the magnitude of their AI investments, and then the associated bookings and return we're starting to see, this is also being sold off because of the relative outperformance and excitement that is going on in other areas of tax.
2:27So I think the numbers itself weren't that bad. It's just when you put it relatively to the rest of the group, whether it's memory, semis, what Google just said, and their blowout backlog number, it basically weakened the meta case for investors in the short term. Got it. I think Mark Zuckerberg didn't do himself any favors when he said they don't have a precise plan for monetization. That seemed to freak a few people out. Never good to say that. Yeah. That was hurtful. So what is the path for them to make money? What do you look for in their investments where you're going to see signs that things are working?
3:11Well, I think there is signs that are working. I mean, they have the fastest growing ad business that we cover. They grew faster than Google. They grew faster than Amazon. Amazon grew low 20s. And obviously, Google's executing well on their ad business. But the growth was the highest at meta. So I'd say they are executing. I think investors have just lost effectively focus on this story, given all the other excitement that's going on in the rest of our coverage of tech. And so I think that's part of it. And I think the second part to your point is, I think everyone looks at Meta and says, like, what is the next big return that can happen?
3:53Is this going to come in ad-driven revenue? Is this going to happen where they can open source an AI model and sell it to enterprises? I think we've said, like, I think it's unclear exactly what the next revenue engine is. Our belief based on CIOs is that they would love to see Meta open source and launch their own wrapper of their own AI model. There's financial services companies in New York that have said to us that they would love to embrace Meta. They don't have that business today. Is that something that they could do as a revenue engine over time? The answer is yes. What's going to happen with wearable AI?
4:32We're big fans of what we might be an outlier, but we're fans of what they're doing with the glasses. I think there's a handful of other initiatives. I just don't think they're quite as big and as in front of us as what you see with Google Cloud, AWS, what's happening in Microsoft with some of their properties. So I think this, again, is just it's further down the road. And ultimately, the real ROI is still really opaque. And when Zuck says, you know, trust us, I think everyone is saying, well, you know, we trust you in the first wave and it's showing up in the ad business. But ultimately, what's the next property that's going to do this?
5:11And I think everyone's kind of scratching their head a bit on that. Right. Well, let's move on to the good news here. So let's talk about Google for a minute. So what stood out to you most? There was a lot of nice numbers there. But what was the thing that really, like, jumped out at you? Yeah, what jumped out is the acceleration of Google Cloud. So 70 % growth versus Azure, you know, growing close to 40 and AWS growing close to 28. So their growth rates are incredible. Second is market share gains. So they gained market share for two quarters in a row against Amazon and Microsoft. I think the backlog was mind-blowing.
5:48I mean, the type of cloud backlog, they added$219 billion of backlog sequentially from Q4 to Q1. And their cloud backlog is up 400 % year over year. I mean, these numbers are insane. And part of what they're doing now is selling TPUs to third parties. Most of this is actually their AI infrastructure they're selling to corporations. So I think what's happening is this kind of vertically integrated model from the data center all the way to the apps into Gemini is really resonating. And we have said this repeatedly that every CIO survey, Google's gaming, Steam, every independent conversation with executives in Silicon Valley, that they're all using Gemini more and more.
6:38Their corporations are using their infrastructure more. and what's even wild to see beyond just the Google Cloud is that in their ad business, search queries were at an all-time high and their search revenue grew close to 20%. So while I thought AI was going to kill search, they're telling you queries are at an all-time high. So I think there's several factors that are driving Google up 8%, And it's one of the best performing stocks in the internet this year. And it's for several reasons. It's a subscription business. It's the ad business. It's the Google Cloud backlog. And I think last year, everyone thought, you know, OpenAI and Q1 was going to crush Google.
7:28And now it's reversed. Everyone thinks Google now is leading. And so the sentiment on Wall Street got twisted. And we think that the true power and the underlying engine of Google is becoming apparent. Yeah. Sorry. It's really untwisted here. I want to get to Amazon just because we have a lot to cover. And so give me your top-level take on what stood out there. Yeah. Top-level. Well, AWS was in line at 20%, but their backlog is, again, similar to Google. They're blowing the doors off on the backlog. They signed a$100 billion deal with Anthropik. Their overall backlog will grow over 100 % year over year.
8:16Their Amazon AWS cloud margin improved. Everyone thought in the world of AI, margins were going to be down. Their margins are actually up. Their advertising business grew in the low 20s, which was really good. and then the consumer business there was no signals that the consumer consumer soft they had consumer revenue upside and margin upside so this is the first time uh that we have seen uh amazon grow 15 percent uh in basically almost four years so constant currency growth at 15 and then they posted a 13 margin which is one point above the high end of the guide and has showed under jassy that he cares not only about revenue growth, but about margin.
9:02And as I've said, I think having a software CEO in place running Amazon is important because in my history covering software, executives care about high margins, and I think Jassy really cares. So I think it goes back to discipline, operational efficiency. EDA US doing phenomenally well. The RPO, including Anthropic will be 464 billion or growing 146%. And these numbers are incredible. And so similar to Google, great backlog commitment that AI is working in these AI companies are going to their infrastructure. Wow, we covered a lot. Thanks Brent. That was Brent Thill, Senior Analyst at Jefferies.
9:47Please. Microsoft revenue grew 18 % in the first quarter, an acceleration from the prior quarter. And the company also disclosed total AI revenue for the first time ever, reporting it was 123 % higher compared to the same period last year. Joining me now to discuss the results is Rishi Jaloria, Managing Director of Software Equity Research at RBC Capital Markets. Hey, good morning. Thanks so much for having me. Pleasure. So what's your biggest takeaway for the quarter with Microsoft? Yeah, look, putting the stock action aside, I think what it's telling me is Microsoft is continuing to execute well on the core cloud, on AI, on their broader portfolio.
10:30They're setting themselves up for success in the long term with things like M365 Copilot, GitHub Copilot. They're definitely, I think, playing the long game here. And I think this is one of the best positioned companies, at least in my coverage universe, for AI because of the broad portfolio and the number of ways they can benefit from AI throughout the stack. So I was very pleased with the numbers. It was great to see Azure accelerate for them to talk about further acceleration the coming quarter and then the two quarters after that as well. That I think is very encouraging to see. Yeah, exactly.
11:08The market is not as upbeat as you are about the company, but we'll see. Microsoft disclosed more than$9 billion in AI revenue. How meaningful is that number? Look, I think it's incredibly meaningful. It's probably, by the way, a conservative definition because if you recall, Microsoft, for the longest time, and there's going to be changes with this, had not been recognizing revenue from training of open AI models in that, in anywhere in the P &L. So that excludes a lot of that as well. So it's probably a pretty conservative number. Now, is it mostly open AI and now maybe a little bit of anthropic?
11:46Yes. But then there's your own first party AI services in there. There's GitHub Copilot, M36Hop Copilot. And I think it's great to see, you know, this really strong trajectory on the AI side for Microsoft, and we have only improved from here. So on the AI side specifically, can we tell from what they disclosed exactly what's driving the AI growth, or is it too many pieces here? Yeah, it's too many pieces. We can do our own guessing and do some napkin math, and guaranteed a lot of that is going to be a function of OpenAI's growth. And we can draw a little bit of a line to that. But look, they did It also disclosed that they had 20 million M365 co-pilot users in the quarter, and that was up 5 million from 15 million a quarter ago.
12:33And, you know, we could probably make some assumptions around ARPU there and how does that translate. So we could do some napkin math in there. It's hard to do it with any high degree of certainty just given all the moving pieces here. That co-pilot number was interesting to me. What does that tell you about enterprise demand? Yeah, look, I think what it's telling us is we're still in the very early stages of enterprise AI usage. And this checks out with all the work that we've done out there. For all that we're seeing out of AI interest, and it's clearly unyielding, but at the same time, most AI usage today in the enterprise, putting aside the AI labs and Google, Meta, etc., putting that aside, most of the AI usage today is a lot of pilots.
13:19it's a lot of proof of concepts and when it is deployed live in organizations it's being used for software development and coding and that kind of stuff as well as customer support both internal and external but it really hasn't made its way to the domains of the knowledge worker to productivity to sales and marketing in a major way yet we obviously all see really great early success stories of that but in terms of widespread enterprise adoption it's not there yet but i I think it's telling us that Microsoft is probably one of the companies that's leading the charge on that. Got it. So that's interesting.
13:57So, I mean, we know about like the coding and all that and where the use is now.
14:04This gets right to the growth of the company, right? When it starts to hit knowledge workers, what do you think, what are you looking for in the next few quarters for that increased adoption? Yeah, look, I think what we'd want to see is really three things. Number one, which we can see in the numbers, is continued healthy growth in paid M365 co-pilot users. Now, this is going to get a little muddied because we're now seeing the introduction of Microsoft E7, which includes co-pilot. That goes GA tomorrow, actually. And I'm really encouraged by what we're seeing in there. But ultimately, is the amount of AI users within that M365 co-pilot seat growing at a healthy clip?
14:45Number two, does this show up as overall acceleration on the offers line? Because that's ultimately what matters. You don't want robbing Peter to PayPal type of situation where it's a game of allocation. You want this to lead to overall growth acceleration. So we'd like to see something like that. And the third, which is now a non-financial metric, but would love to see improving customer feedback to the actual technology around Office Copilot. I think no secret that initial customer feedback was pretty mixed to negative. And it has improved. Even over the past six months, I've seen it tick up.
15:22I'm excited to see how customer feedback with the launch of E7 and solutions like Critique and Council that levers multimodality, how those start to actually play out. If we start to see improved customer feedback, that makes me feel even more bullish that they will be able to drive usage and ultimately that drives revenue for Microsoft. Right. Because, yeah, we saw it too, the general unhappiness. So we had this pile of earnings last night, which was, I'm sure, a joy for all the analysts out there. Where does Microsoft sit versus the competitors? The stocks are all going in different directions today.
16:06Yeah, look, I think that all of the major competitors have different angles by which they're attacking this, right? And I think, and look, my colleague, Brad Erickson, does a fantastic job covering the the internet guys, I cover Oracle. So between us, we all cover the major hybrid skills. But what I would say puts Microsoft in a very unique position is, number one, the breadth of their portfolio that I think can benefit from AI, right? I mean, you obviously see Meta talking about how they're using AI to benefit user engagement and advertising. Google's obviously talking about advertising and what can this do for search.
16:41So everyone's attacking it from a different angle. So Oracle, definitely much more of a, I would say, more raw infrastructure provider. For Microsoft, it starts at infrastructure, but it goes into the developer layer, goes into the data layer. We've seen some pretty good early success with tools like Microsoft Fabric and Cosmos DB. It's going into the productivity layer, as we were just talking over at M365 Copilot. At some point, I think it starts to flow into the security layer with security Copilot. And I can even make a case that assets that get very ignored, like gaming or LinkedIn, can see major benefits from AI.
17:18In fact, Sethi was talking about some of the benefits from AI that LinkedIn is already seeing today. So that's what I think puts Microsoft in a unique position is leveraging the multi-model nature, leveraging the close early relationship they had with open AI, and then bringing out that throughout the entire stack of Microsoft Suite. I think that puts them in a very unique position in this post-AI world. That is really interesting. So thank you so much, Rishi. That was Rishi Jaloria from RBC Capital Markets. The information's Rocket Drew is covering the Elon Musk OpenAI trial from the courtroom all week long.
17:54Here's his latest update for us on Elon Musk's cross-examination. Elon Musk had a brutal cross-examination session from OpenAI's lawyer. Musk was on the stand for the second day in a row, but this time it was OpenAI's turn to question him. And they took that opportunity to ask some very hard-hitting questions of Musk. In fact, there were some heated moments between the two. Musk accused the lawyer of asking misleading questions, asking questions that were too vague. The lawyer said Musk was being uncooperative and repeatedly had to pull up old documents or even the transcript from Musk's own deposition to point out that Musk had answered questions before or to at least jog Musk's memory.
18:39Now, this was really crucial for OpenAI's side because it was their first chance to get Musk to speak to some of the points that OpenAI has been making all along, that they see Musk as a disgruntled co-founder, someone who was always content to have OpenAI operate in certain for-profit ways as long as Musk was in control of the company or was going to be in control of the technology. And in fact, Musk helped set up a for-profit subsidiary for OpenAI and went so far as to suggest that OpenAI could attach to Tesla as its cash cow. So OpenAI showed that Musk basically thought he was leaving OpenAI for dead, that it didn't stand a chance without him.
19:23And if OpenAI had never been such a success story, then Elon wouldn't have been so resentful and wouldn't have brought this lawsuit in the first place. So that's the portrait that they painted of Musk. Now, on the other hand, this brutal questioning was an opportunity for Musk to stress test the refined theory, the ways he's refined his own position in the case. So, for example, he's found a way to answer the question, why didn't you bring this lawsuit sooner, before there was a question about whether the statute of limitations had expired? And he says, well, I only became suspicious of OpenAI in degrees.
20:03There were phases to it. And yeah, sure, I had some suspicions back in 2017, but there were suspicions that a promise might be broken in the future. It wasn't until years later that a promise was actually broken, or I came to believe that that was the case. So that's a way that he's refined his case. Still, Musk came out of the questioning session not looking so good. For example, Musk has had a lot to say about AI safety during the case, that he thinks OpenAI is dropping the ball on AI safety, that they've been deprioritizing it and instead prioritizing profits. But when OpenAI's lawyer asked Musk some very basic questions about the industry's best practices around AI safety, Musk was clueless.
20:45He was asked about safety cards or system cards, which are the papers and documents that often accompany a new model and explain the testing that went into that model. And he didn't know what that even was. And similarly, he was asked about OpenAI's preparedness framework, which is a key safety document that governs a lot of OpenAI's safety work. Musk also said he had never heard of it or other protocols that OpenAI has in place. So Musk comes out of this not looking so hot, but he did have a chance to advance some of his positions, and he was also able to justify why he founded XAI as a for-profit company, which has been an awkward question hovering over this whole case.
21:24If operating, you know, if running AI as a non-profit is so important, why didn't Musk found a new non-profit when he came, when it came time to found his own, you know, organization in 2023? And his answer to that is he's okay with it being a non-profit or a for-profit, but you have to pick one. And the way he puts it, OpenAI was trying to have its cake and eat it too. So we should expect to hear more from Musk on these topics. He's due for a second round of cross-examination. After Musk, his associate Jared Birchall is expected to take the stand. After him is Stuart Russell, the UC Berkeley computer science professor, and finally, Greg Brockman.
22:04So all of that and more is coming up in the Musk v. Altman trial. The information published exclusive reporting about SpaceX's Starlink revenue per user relative to its growing customer base. My colleagues Theo Waite and Valida Pau wrote the story. Theo joins me now to share what he knows. Theo, what's the... Hi, Theo. What's the big takeaway?
22:31Theo Wayt:So Starlink is growing, but as it's growing, it's making less money from each user. So we reported based on confidential filings that SpaceX Starlink's revenue per user was$99 in 2023,$91 in 2024, and$81 last year. Over the same time, the number of customers quadrupled. So they're making a lot more money, but they're making less money from each customer. so so why is that why is the arpu going down so starlink has kind of gone from this niche service for rich people that with cabins and boats and and things being the average user to a more um you know mass market service that uh you know in suburban parts of the u.s um you know is now competitive sometimes with traditional wired internet from Comcast or AT &T.
23:32Theo Wayt:But to get there, Starlink has had to slash prices pretty aggressively. A couple of years ago, the cheapest plan in the US was something like$120 a month, and now the cheapest is$50. So it's changed the business quite significantly. You wrote about Morgan Stanley's estimates for Starlink revenue back in 2024. They missed the mark what did they get wrong about this story so they they were honestly insanely bullish it doesn't really make sense i mean there were other analysts at the time even that didn't think it made sense but they morgan stanley thought that starlink was bringing in more than 170 dollars per month per user um which just wasn't true at the time and and isn't true is even less true now um and i guess from that perspective they just thought the average star Starlink customer was a lot more affluent.
24:25Theo Wayt:And for some reason they thought that that would continue to be the case. But, you know, in the US, Starlink is trying to go, like I said, to this mass market offering and internationally, it's expanding in countries where people don't expect to pay that kind of money ever. Like, you know, internet service in Europe is cheaper, not to mention they're, you know, entering more countries in Africa, Southeast Asia, like consumers in those areas just have less money to spend and have different expectations. So I honestly don't understand why Morgan Stanley thought that it would stay the same. It doesn't make sense.
25:00So it seems like we've seen pricing pressure, and it seems from your story like it's going to get worse, right? There's a rival satellite service coming on later this year. What's happening with that?
25:13Theo Wayt:So Amazon Leo is supposed to launch later this year. Amazon hasn't detailed exactly how much they're going to charge for the terminals or for service. But, you know, they're framing it as a competitor to Starlink that, you know, will be just competitive on price. And Amazon has a big advantage in distribution. You know, they have Prime, they can bundle it with that. They have their e-commerce site where if you just put a Amazon Leo, you know, splash on the homepage, a million people are going to see it every minute. So if Amazon can offer a service that's competitive with Starlink, technically, if the speeds are okay and the service is okay, it'll lead to a lot of pressure on SpaceX.
26:00Theo Wayt:But that's still a big if because the service has yet to launch. So let's talk about Starlink's customer base, individuals versus enterprise. So how are they different and how is that evolving? So the individual business is both people that buy terminals and put them on their house and smaller businesses that have their own standardized subscription plans. And that's the majority of Starlink's customer base. It's over 60 % according to the numbers we saw. And it's also faster growing. And that's comparable to AT &T or Comcast. Like it's just individual subscriptions that are managed in kind of a, you know, generalized, standardized way.
26:46Theo Wayt:And then there's the enterprises, which are the U.S. government, foreign governments, the military, airlines, shipping companies, like all of the gigantic customers out there who have, you know, individually negotiated deals that are managed by SpaceX. And that's also a pretty big business, but it is not as big and not growing as fast. So Starlink is still kind of a primarily a consumer business. So we have this little IPO coming, potentially a trillion dollars for SpaceX. How does this declining revenue per user trend fit into the IPO? And is it something that an investor should be nervous about?
Read the full transcript
27:29Theo Wayt:You know, I think out of all the things that should make investors nervous or might, this is pretty normal. Like, you know, as a business like this gets more efficient, gets bigger, expands in countries, you know, internationally where people have less money to spend, like, you know, this is not totally shocking. And like revenue is going up because the service is growing quite quickly. Like, out of all the things at SpaceX that you could, you know, potentially find concerning, this is like not really one of them. And, you know, for what it's worth, you know, SpaceX says Starlink is getting more efficient.
28:06Theo Wayt:They're going to launch bigger satellites that can serve more people. And so, you know, if you're thinking about it as a standard business, it's not that unusual, but obviously the valuation is totally out of proportion to how other satellite companies and telecoms are valued. So it depends on what you're comparing it to. The interesting thing about the IPO with Starlink is Starlink is like the profit driver, right? It It is this business in there that people are looking at for, at least in the near term, the profits. So, I mean, is there an outsized sort of attention paid to this? I think so.
28:46I think that it's a segment that they're trying to grow.
28:51Theo Wayt:and they're putting a lot of money into Starlink Mobile, which is kind of a separate service that uses a different set of satellites to beam service directly to cell phones rather than terminals. And, you know, I think that will get a lot of scrutiny in the next few years because SpaceX is spending almost$20 billion on buying Spectrum for that. But as of now, it brings in very, very little revenue. So I think anything that they do to keep Starlink growing will be quite important to the company. Thanks, Theo. That was Theo Waite, our Elon Musk reporter here at The Information. Right now, there's a big scramble for NVIDIA server chips.
29:33But recent reporting for The Information shows that even when AI developers have GPUs, they might not be getting the most out of that expensive hardware. My colleagues Stephanie Palazzolo and Anissa Gardizi wrote about this in our AI Agenda column. Steph joins me now to share what she knows. Steph, to start with, how do developers judge whether they're getting enough performance out of their GPUs? So there's a couple ways to think about this. The first way is to think about whether the developer even has enough work to use all the GPUs in the first place. So theoretically, if this developer has 100 chips, maybe he only has enough work to use 60 of them.
30:13That's obviously not great. The second way to think about it is out of the GPUs that this developer is using to train a model, run a model, it's possible that those GPUs are not being fully utilized 100 % of the time. And so developers can measure this with different numbers like model flops utilization, which is a metric that we talked about in our newsletter from this week. So let's focus on XAI for a second. You reported that they may not be getting the most bang for their buck. What's going on there? Yeah, so as we wrote in our newsletter, we reported that XAI's model flops utilization rate is around 11%, or has been in recent weeks.
30:57And to kind of put that in perspective, you know, in a perfect theoretical world, you would want that metric to be at 100%, which means that the chips are being fully used. In reality, most labs do find it pretty difficult to get above 40 % from what we hear from different AI researchers. but you know even compared to 40 percent I think 11 percent is a pretty uh rough number to to have um so I think this this goes to show that even a company like XAI which has been in the news so much for you know getting these clusters up and running super quickly can you know face a lot of issues with getting those those chips you know getting the most bang bang from their buck basically out of those chips but if there's such a scramble for chips how can companies be both short on GPUs and still not fully using them.
31:43It just doesn't make sense, right? Yeah, yeah. It's almost like contradictory kind of thing going on here. I mean, there's a couple of reasons for this. I think the first reason is that training models can be very bursty, which is a phrase that a lot of AI researchers use whenever they're describing this to me, meaning that there can be this kind of sudden spikes of GPU usage where, you know, you are training that model running experiments followed by periods of lower activity where researchers are kind of sitting around looking at the results of the experiment they just ran and deciding what to do next.
32:17Another reason why the utilization might not be so high is due to the kind of special memory that AI chips need. And because of that, sometimes the chips spend a lot of time waiting for data and moving data around rather than doing the actual computing that they need to do. And I mean, just in general, whenever you think of these giant clusters of chips that can have, you know, thousands or hundreds of thousands of chips, any kind of weak point in the network can really snowball and cause trouble for the larger cluster. Got it. So is this a now thing in the sense that we're still early, you know, the training models, data center is just getting built.
32:57Is this something that people expect is going to be sort of worked through in the coming years? I mean, on one hand, obviously, all the AI labs are working on this problem. And I think, you know, we do hear that a lot of the kind of top AI labs have made a lot of progress here in terms of getting that utilization number up. But at the same time, I think this is going to continue to be a problem because, you know, every year or two years, there's these new chips coming out. And with each chip, it's kind of like you have to start over this process of, you know, figuring out how to connect them together, making sure that things work well without hitting any bumps in the road.
33:31And so it's almost like AI labs have to go through this kind of trial and error process every year or every two years whenever they get a new generation of chips in. So yes, I'm sure we'll make overall progress here, but like, I think this is just going to kind of continue being a problem that they're going to have to have to deal with with every new generation of chips that come out. Got it. Now you also report that there's a little bit of gaming going on, right? These researchers are trying to hold on to GPUs. So how are they doing that? Yeah, I think that was one of the most interesting parts of kind of doing the research for this.
34:04So, you know, as you can imagine, if you're an AI researcher, you are short on chips and you want to hold on to the chips that you do have. Because, you know, if you think about who, like, you know, if I'm an AI researcher, if my company realizes I'm not fully using the chips to their full potential, they might want to take some of those chips away from me and give them to another researcher who they believe can really, you know, eke out everything that they can from those chips. um so you know researchers have basically come up with these interesting ways to basically fake higher utilization of these chips so for instance uh some researchers at ai labs told us that they've seen others basically you know rerunning the same training experiments over and over again and then deleting the results and then rerunning it so that it basically looks like you're you know constantly using these chips even though you're basically just redoing the same task over and over again.
34:53And so by doing that, you can basically fake that you're using those chips more and that way kind of prevent your managers from taking those chips away and giving them to other folks on your team who might use them. Got it. I love it. People are always people, even when they're training AI models. Exactly. You mentioned that XAI's deal with Cursor. So how does that help solve this problem? So Cursor might help solve some of these problems that XAI is facing, but maybe not all of them. So if XAI, you know, kind of the first problem I was talking about earlier, if they just don't have enough work to, you know, use up all their chips, maybe they're only using like a certain fraction of them because they don't have enough, you know, training experiments going on.
35:36Cursor can help kind of fill in those gaps because Cursor is also training models itself, and it's also running models. So, you know, if XAI is using 60 % of its fleet, maybe the other 40 % can go to a cursor to help them run or train those models. But if XAI's issues are more on the kind of like chip-by-chip basis, or if it's because, you know, researchers are kind of hoarding chips whenever they're not supposed to, it's not clear whether a cursor is going to be able to help with that, because some of those issues are more on this kind of like very like low-level engineering infrastructure layer.
36:10So in that case, it's going to be more of like this engineering problem where they really have to get into the weeds and like work on the kind of coding surrounding the chip to make them work better. So in that case, cursor might not help as much, but overall, I think it kind of can help fill in those gaps where they do exist. Got it. Thank you, Steph. That was Stephanie Palazzolo, one of our AI reporters here at The Information. That does it for today's show. A reminder, we are on this stream Monday through Friday at 10 a.m. Pacific, 1 p.m. Eastern. If you can't make it then, episodes are available on theinformation.com, our YouTube channel, or wherever you get your podcasts.
36:51Make sure to follow us on social media on X, Instagram, and TikTok. I'm already excited for our next show. Have a great rest of your Thursday. Bye-bye for now.
From the publisher
Jefferies Senior Analyst Brent Thill joins Senior Finance Editor Ken Brown to break down Meta’s massive $145 billion CapEx forecast and why Mark Zuckerberg’s monetization plan is rattling Wall Street. We also talk with RBC Capital Markets’ Rishi Jaluria about Microsoft’s $9 billion in AI revenue and the enterprise adoption of Copilot, and we get into the "brutal" cross-examination of Elon Musk in the OpenAI trial with Rocket Drew. Lastly, we discuss Starlink’s declining revenue per user with Theo Wayt and the technical hurdles of GPU utilization at xAI with Stephanie Palazzolo.
Articles discussed on this episode:
https://www.theinformation.com/articles/musk-battles-openai-lawyer-claims-co-founders-stole-charity
https://www.theinformation.com/newsletters/ai-agenda/xai-shows-hard-use-lot-gpus
Subscribe:
The Information: https://www.theinformation.com/subscribe_h
Sign up for the AI Agenda newsletter: https://www.theinformation.com/features/ai-agenda
TITV airs weekdays on YouTube, X and LinkedIn at 10AM PT / 1PM ET. Or check us out wherever you get your podcasts.
Follow us:
X: https://x.com/theinformation
IG: https://www.instagram.com/theinformation/
