Big tech earnings and the current AI debates, with Sarah Guo and Elad Gil

0:05Today on No Priors, we're having a special episode of Sarah and me just talking. Hello Sarah, How are you? Hey, Alon. What's going on? I see you a lot. Not much. Good to see you. Let's talk about models. What's going on in the model world? Yeah, I guess there's a lot of hand models that are emerging. So I was thinking of maybe trying to do that eventually. It's almost as good of a business as investing. I know, right? Yeah, so there's been a lot that's happened in the model world recently. Obviously, Google launched Gemini, which I think had a few interesting characteristics, both in terms of performance, but also the huge context window, right?

0:40It was a million token context window. So companies like Magic, I think in the past, have actually put out like a 5 million token context window model and things like that. But it's really exciting to see that. And I think for certain application areas, like biology, longer context windows actually seem to be quite important. And so, for example, if you're doing a protein folding model and you have a short context window, you're often actually not encapsulating much of the protein, right? The average protein is, I think, something like 300 amino acids long, at least in the human genome. But there are things that are dramatically larger than that.

1:11and so you just can't capture it in some of the context when it was being used for biological models and so I do think this is going to be one of those areas that will end up being more important than people think at least in the short run but Gemini 1.5 seems to have some really interesting performance characteristics there's obviously Sora from OpenAI which was the video model that you know is just beautiful to watch you know there's other model companies like Pika and others that I think are doing exciting things as well and then Mistral or Les Mis launched Le Chat which is really the name of the product le big model le big model le big mac i believe they call it mistral large yes yes the large mistral large they launched that and the thing that's really really impressed me about mistral is just the velocity of shipping it's incredibly impressive they went from basically starting the company to almost gpt4 level in less than a year right Nine months, yep.

2:07It's amazing. And they have, you know, small performance models. They have the Big Mac or, you know, the large model. They have chat. They have multiple languages. It's just, it's very impressive execution. So, and then I think the other thing that they just launched or announced was that deal with Microsoft where, you know, they're now being licensed onto Azure. And so I think the main models in Azure now are OpenAI, Llama, Mistral, and then some of the Microsoft models. So again, that's striking as well. So just very impressive progress by that company so far. I think the design space for what you actually want from models is certainly going to include state of the art capability.

2:46And Mistral is very much going after that, and they've said so. But I think like from the beginning, the company has talked about efficiency and latency and the ability to serve different use cases with that. and also, you know, being long-term proponents of retrieval, right? Like one of the big debates in the research world right now, I don't know how much of it is a debate, but people are talking about it. I'm on one side of this, is that like rag and retrieval is dead with sufficient context. And I'm curious what you think here, but I'm more of the belief that it just opens up the set of trade-offs you can make between retrieval, more sophisticated retrieval and model reasoning by having a larger context window versus saying like, we don't need any ability to work with a specific data set versus just retrain or stuff something into context.

3:41Yeah, we're going to have both in my opinion. The other thing I think that's very under discussed and this could lead into agent stuff, but I'd like to also spend a little bit of time on Gemini before we move to agents is if you look at a lot of the optimations that are done for areas where you had human related sort of reasoning or other components pre-LLM based reasoning. A lot of it was happening at inference time, right? So when you were doing, when you're trying to build a better poker AI, a lot of what you did was, you know, certain types of tree searches or other things when you hit inference time, right?

4:14You built the model, but at inference, it did a lot of extra work. And I think that's also a little bit under discussed in terms of probably a lot of what's going to happen in the future, particularly we get into agents and reasoning, is stuff that's happening at that point of inference. And then it's used to sort of feedback over and sort of continuously train or retrain a model over time. Because I think that's the other piece of it is, you know, from a model perspective, you spin up a giant data center and you spend$100 million over 12 months overall between all the different works that you do and everything to launch your next model.

4:49and then you have a file and then you use that file for the next year as you train the next model versus saying you're going to use some sort of continuous upgrading or training. And so all these things are going to shift over time. I think it's early in the technology cycle. And so all these things are going to happen. You know, one of the companies that has a lot of capabilities to do interesting things over time, of course, is Google. So I'm a little bit curious if Gemini has changed your opinion of sort of the AI model race and what role Google plays in the future. Or, you know, has it not changed your mind much?

5:18I think the question on like whether or not Google has the ability to do the research work to have a competitive product has been answered. Right. Gemini is a very impressive model. I think the the capabilities that they have internally that they haven't released yet around additional like function calling and multimodality are also really, really impressive. And so the questions around Google are less about do they have like they have all of these extraordinary advantages. And you're you're the ex-Googler. Like, I want to hear your opinion. But they have the distribution. They have the customer behavior.

5:58They have all the data on like what the search behavior is. They have the data on what queries are valuable and which they would peel away and turn into like an answer. They know how to build like. advertising auction systems. And they have a great research team and enough GPUs and the model capabilities. Do you think it's progressive enough though? Do I think the models are progressive enough? Yeah. One might actually ask if they're perhaps a little too far in that direction. Right. And so I think like the question is actually, can they steer Google to like focus on being competitive versus the many other demands from their employee base and like different missions that are not brokering the world's information and like market cap.

6:52Yeah, it's interesting because the launch of 1.5 has made me more bullish on Google. And I was always actually quite positive on it, right? Like I think I read a blog post a year or a year and a half ago basically about the model world. And one of the things that I mentioned at the time was I felt like Google was kind of a sleeping giant And once it awoke, you know, it could really make enormous progress quickly. And just as Mastral is executed from scratch as a startup, which is extremely hard to do, right? You're literally building everything from the ground up. Although obviously there's open source to support you and all these other things.

7:23But fundamentally, you're just building an entire company. It's pretty amazing, right? Google has really accelerated its efforts. And it's had a series of launches over the last two, three months that have been quite impressive in terms of the velocity from cold start to having things that are externally accessible. And they have all the resources that one would need in order to do extremely well in AI, right? They have the compute, they have unique proprietary data as well as all the data from the web, all the data from YouTube. They have specialized data that you could potentially opt into, like all your emails and your Google Docs.

7:59And they have this immense corpus of really valuable information. and then they have amazing talent. And so really, I think the thing that was lacking until recently was the will. And it seems like now, because of the competitive dynamic, the will has been reborn, right? And so it really feels to me like they are going to make really big strides going forward. And, you know, it's always possible that the velocity only increases from here for them. If I think about the domains in which these general LLMs are still not as capable. I mean, it's every domain, but in particular, not as capable as we want.

8:43Like two of the areas, one you already mentioned that I'm excited about include like biology and then robotics. So maybe let's talk about that for a second. As a task, for example, if you ask ChatGPT to design a DNA sequence that can express CRISPR-Cas9. It can't do that yet, right? And if we think about cell design, protein design, protein optimization, a lot of these are areas where you have researchers showing like really exciting progress in use of transformers and diffusion models to get to much better predictions for, for example, drug discovery and target identification. And so I think, you know, I've seen a number of companies in this area of better understanding of biology that really feels like a different type of reasoning, a different type of data set.

9:38And as you said, even like specific context window constraints. And so I think that's an interesting one. And then on the I don't know if you wanted to mention the robotic side or if that's something you've been looking at, too. The robotic stuff seems super interesting. It's a little bit earlier on than some of the other models and part due to data constraints, but it seems like there's pretty reasonable ways to generate some of that data now. So it seems like the, you know, in general, I wouldn't be surprised if 2024 and 2025 is the year of proliferation of models, where we're going to start to see an expansion in terms of the different types that are covered, you know, chemistry and material sciences, et cetera, et cetera.

10:15Robotics will be part of that. Biology will be part of that. Maybe physics and math. I think maybe the last thing that is happening from a model perspective is I think the last few weeks have seen a lot of different sort of agent-centric companies get up and running and I think that's been a really interesting wave and some of them again are taking very different approaches from the traditional let's just build a giant LLM and they're looking at things like AlphaGo or some of the game-centric work that have been done in the past you know how do you build a better poker paper? How do you build diplomacy?

10:52How do you build Go? And there you have a very strong notion of acting sequentially based on changing information. You have some forms of what's known as self-play. You have the machine play itself a billion times ago, and it learns new patterns based on that. You have really interesting approaches and heuristics and algorithms at time of inference versus training. and so I think that that purpose of knowledge is about to hit the world in the context of new products and it'll take time for those products to emerge you know 6 months, 12 months, a year but it does feel like that's another wave that's coming where you're taking a fundamentally different approach that involves reinforcement learning but is just different in terms of how you think about what you're actually doing in architecting and what you're inferencing and all the rest So that's one other area on the model side that I think is very exciting.

11:46One thing that I've seen here is that people are getting much smarter about agents as part of systems versus expecting to simply like instruct an agent and have it work with compounding failure across a bunch of tasks. Right. In a general environment across any type of software. Right. And so if it is operating in an environment that supports reinforcement, well, like a game environment or even a web application environment, but one that is constrained to particular tasks or working agents working in domains that better support. a sampling and validation, like code generation. Like I'm really excited about that.

12:32And I feel like I've begun to see the glimpse of some of those things work. Whereas a very real question you could have asked in Q3, Q4 of last year would be like, is any of this stuff useful, right? Is it anything? And I think now it's like, it is. Yeah. Yeah, people went too broad too early versus just saying, I'm just going to focus on a handful of targeted use cases or domains. And I'm going to figure out how do you create feedback loops in those domains so I can actually train effectively? And so, you know, the very early versions of this, even predating this LLM wave was, you know, hey, we're going to have a browser plug in and it'll watch everything you do and then it'll do everything you do.

13:10Which is a very different problem from saying, hey, we're going to make our PA better. We're going to make code better. We're going to make customer support better. We're going to make, you know, XYZ thing better. So I think the targeted approach makes a lot of sense. Yeah. And I think some of the teams working on this have also, they've actually experimented with post-training in environments where you can pay for human feedback data, right? And if you do that, then you actually understand, like, the distribution of data you need, the scale of data you might pay for. And that's very exciting because it turns it like the agent problem from one that is like open ended, untenable to just like how much is it going to cost to make a particular task work?

13:57And I'm massively oversimplifying here, but that is a very different proposition when scoped than like, as you described, the initial set of forays into agents, which is like, you know, we'll try to do anything. Yeah, that makes sense. I think we'll still get there, but there is like rapid success on this front.

14:20Nvidia, everybody's talking about earnings. What do you make of it? I think earning money is an excellent idea. How about you? I think Jensen understands this better than everybody else. I think one thing that people have been talking about is whether or not this was a short-term phenomenon, right? Like, if there was only so much demand and once the supply chain caught up a little bit, There would be less insane growth. And I think now people are pretty confident, especially hearing Jensen's comment that they expect to continue to be supply constrained for the rest of the year. Demand is just like much, much larger than I think most people expect on the CapEx side.

15:03And I think it's worth understanding the upgrade cycle that drives that. Right. Because there's this huge efficiency incentive to upgrade from A100s to H100s to H200s to B100s. I was talking to one of my portfolio companies that's buying in the tens of thousands of GPU size and is skipping to B100s because they described it as like free money in terms of training efficiency. It's funny when somebody describes spending hundreds of millions of dollars as free money, but free money in terms of training efficiency, if you can actually get access to a cluster of a certain size. And so if others feel that way, it is wild how much this expands the server market.

15:49Yeah, it's probably a good time to run a hedge fund. I think in general, one thing that's a little bit under discussed is a lot of the emphasis on startups and startup rounds. and it'll look, the startup raised$100 million or whatever. And the reality is a lot of the spend is the big hyperscalers and then other clouds that are building out right now. And then I think the other thing is that if you were to look at at least enterprise adoption of AI, it's still really, really, really early days. And despite that, if you look at Microsoft Azure revenue in the last quarter, they mentioned that revenue grew by 5 % from AI-related products, which if I'm doing the math right, If it's a$25 billion a quarter Azure sort of revenue, then that means they're adding something like$1,$1.5 billion a quarter in new spend due to AI.

16:37So that's$5 or$6 billion annualized. And so one thing that is a little bit perhaps not talked about is there's a lot more stuff coming. and over the next two years, three years, etc. as enterprises really adopt this at scale we should anticipate as well that the need for compute will continue to grow so it's really interesting to see about this replacement cycle you're talking about the massive spend by big tech on LLMs because they're driving most of the spend on LLMs because they're the big rounds the big rounds aren't venture capitalists investing billions of dollars it's the big tech companies it's Amazon and Google and Microsoft and Salesforce and NVIDIA, actually, right?

17:22And then there's the enterprise adoption, which is still TBD. So yeah, there's a lot going on. On this point, if you look back a month, you know, AI years are like dog years. So a year to the meta earnings beat at the end of January. Did you see this article that David Kahn wrote at Sequoia? The 200, like AI's$200 billion question. Was this where he basically said, based on the spend, if you think of the ROI you need, then you need to generate hundreds of billions of dollars in return. Yeah. Or justify all the, yeah, all the spend that you had. Yeah. Yeah. Very succinct summary. And I was like, okay, yeah, that is the question.

18:00And I feel like the meta earnings beat was the like one day answer to that question. Right. So to your point, they're one of the large spenders. They said they're going to spend 30 to$37 billion on CapEx in 2024 driven by like AI driven by servers, right? Mark has this great like quote where he's talking about 600k H100 equivalent units of compute and saying like there's no room for other people. But the response to all of the investment that has in CapEx for training and inference at Meta over the prior years has been like a huge earnings beat from better targeting leading to better conversion, better recommendations, leading to better engagement, better advertising tools, leading to better ROI, as well as like the cost controls that the rest of the industry is doing.

18:58And so they had this one day, I thought it was really nice that the number was exactly this too, that this one day ad of 197 billion of market cap, biggest single session ad before NVIDIA. I forget where NVIDIA ended up landing after their beat. But like, that's the answer, right? Like, you know, 197 billion of increase in enterprise value on 25, 30 billion of CapEx. Like, you should keep doing it. Yeah, it's kind of amazing. It's kind of a related question because I remember Uri Milner showed me this chart, which basically he looked at the aggregate increase in startup market cap and the aggregate increase of what at the time was like FANG market cap.

19:38And obviously now there's like the magnificent seven or eight or whatever it is. And so if you looked at the top tech companies at the time, they added like, I don't remember it was five or 10 times the market cap of all the startup ecosystem combined during the same period of time. And to some extent, you could argue we're going into the same thing, at least in the short run for AI. And we still haven't seen the monster AI companies emerge from scratch. And undoubtedly, those will exist. But at least for the next few years, it seems like where we're going to see that really huge market cap incremental add, maybe companies like OpenAI and some of the model companies, but also it seems like increasingly it's just going to be existing companies adding huge amounts of revenue and earnings and compute and everything else along the way.

20:23So it's back to like maybe the right thing to do right now is just start a hedge fund. I think that also begs a question of how to think about like all of the other companies like tech and not in terms of amount of impact from AI. I actually think it would be like a really fun lens to run a hedge fund with because you can take a you can take a very long term view of something that feels very secular. Just classify companies this way and long short, like take that strategy as the only lens. Because like I do think that there are a number of services companies that are squarely in the sights of things that you will be able to significantly automate.

21:11And the only question is, which of these management teams is going to have the investment capability, technical talent, guts conviction to invest the way Mark did through, you know, people were really mad about the CapEx spend for a few years at Meta, right? And I think the answer is mostly, especially some of these services firms, like maybe they partnered to get there, but they mostly will not make the transition, I think. The other thing it isn't really discussed is the impact it's already having on some businesses. So obviously ServiceNow had like a blowout quarter in part due to AI. So we're starting to see a little bit of enterprise adoption.

21:47One of the folks from Klarna posted today that they built an AI assistant that's powered by OpenAI that in its first four weeks handled 2.3 million customer service chats for them. And so it ended up handling two thirds of all their customer service inquiries. It was on par with humans in terms of customer satisfaction. It was higher accuracy, so it led to a 25 % reduction in repeat queries. Customers resolved their errands in 2 minutes versus 11 minutes. It's live 24-7 in over 23 markets, communicating in over 35 languages. And it performed the equivalent job of 700 full-time agents. And so basically Klarna, in a few months or a year or however long it took him to build this, built this customer service chat product and it replaced 700 people's worth.

22:38And they say that at this point, they have something like 3 ,000 full-time agents. And so it cut the agents needed by about 25%, right? And so it's this really interesting post from Clarner where they announced this. And then one of the things they announced as part of that is longer-term society needs to think about what this means for society because this technology seems to be so good for certain human-level tasks. And this is back to that point of AI adoption in the enterprise is just starting. But how many years is it before every enterprise realizes that they can cut customer support dramatically, at least for certain types of products, just through adding, you know, simple apps, you know.

23:19And so I think that's the other thing that is kind of happening in the background that isn't talked about that much, but, you know, is already starting to really show its face in pretty interesting ways. Yeah, well, I do think you're going to get this accelerated adoption that goes use case by use case, right? Where like in any market, you have early adopters that build it in-house or go get these solutions and are willing to take the risk when you don't actually know like what the impact will be, how well it will work. but as soon as one payments company does that and it's a better experience for the customer or it has real like impact on operating cost I think like you switch very quickly over to the entire sector being like we have to adopt it in order to be competitive on both fronts.

24:05Oh yeah, yeah, this stuff tends to happen slowly and then suddenly all at once and I think we're in the slowly phase right now and I actually had my team go intake global services and look at that, right? And so if you look at spend on software in the US right now, it's about half trillion dollars in software spend a year. If you look at human-centric services, just payroll, for things where Gen AI can probably impact things, it's three and a half to five trillion. So if you convert just 10 % of that spend into AI revenue, you've effectively recreated the entire US market software industry and market cap, right?

24:45And so these are huge trends that are coming. And you can kind of imagine vertical by vertical, what are those things going to be? And then you can ask, is it going to be built as internal tools for companies? Is it going to be a new company that emerges that serves these things? Or is it going to be an incumbent who figures it out and adds it? And so this sort of customer support chatbot thing, you know, you would have thought that there's a company doing this for everyone and it looks like in this case they're um they just did it internally or in-house uh but you could also imagine an existing company like a zendesk or somebody adopting to this and the real question is which of those three scenarios is going to happen at least from a startup perspective but from a technology wave perspective this is massive right and you can build in the feedback loops really easily for this type of product right because you can have the customer rated or thumbs up thumbs down at the end of the session etc so you have a really good sort of RLHF or some sort of training support as well.

25:40So it's a product that should get better and better and better over time as you use it more. Yeah, I think one of the things that is an indicator of like where that services spend might be that gets externalized is actually like the big tech companies actually have, you know, they're tech companies, but they have broader businesses than I think sometimes they're given credit for, right? Like Facebook meta interacts with SMBs. as advertisers. If you look at anybody who has this like large commerce type customer base, so as you just mentioned, Klarna or Square or Meta or Shopify, like they've all done this now and it's working, right?

26:23And so I think the fact that these are the companies that have the technical teams that are capable of doing it in-house is a nice indicator for like, well, if it's that effective, everybody else should too. And the question is, I think not every segment of customer, like retailers with enough of a technical team to build an e-commerce presence may not build this themselves, then it's a more likely scenario that either an incumbent or a new company, be it Sierra or something else, ends up owning that customer service segment. Yeah, 100%. Yeah, we have a longness internally of like the companies that I think should exist in this space, right?

27:00because there's so many obvious ones. And very few companies exist for most of them, if any companies. And so I think it's back to this idea that there are these human capital waves happening in AI. And the very first wave we saw was researchers and they built early model companies and they built some of the early applications like Perplexity and Harvey and all these things were actually started by people who were working on models initially. And they were just closest to the technology so they knew what to do. And then the second wave of human capital was like infra people because they were the second closest to LLMs.

27:33And then the third wave, of course, is going to end up being application builders, but many of them were not aware that any of this stuff was important until ChatGPT came out 15 months ago. And they're just starting to show up, right? It takes them nine months to quit their job and a few months to figure out what to do and find a co-founder and a few months to build a prototype. And so we haven't seen anything yet really on the app wave. You know, all the apps or many of the apps so far were started by people who are very close to the research community. And then it's kind of permeated into other areas with some things growing really fast, right?