Listener Q&A: 2024 Tech Market Predictions, Long Term Implications of Today’s GPU Crunch, and Will AI Agents Bring Us Happiness?

0:05Hey, everyone. Welcome to KnowPriors. I'm Sarah Goa. I'm a lot, Gail. This week on No Priors, we're back with another episode where we answer your questions about tech, AI, and everything in between. I think we have a lot of different questions that people have brought up this week that they were hoping we could cover and some topics that we thought would be kind of interesting. I want to go to one of our listener questions and I think a topic that's really popular with many of the companies that you and I work with in terms of access to computing for much smaller scale experiments. What's going on with the GPU crunch?

0:37Yeah, the companies that you and I work with, many of them are companies that, you know, they need to use very specific infrastructure to train and serve large models, right? These work on GPUs. And the structure of the industry is like, it's just not very robust, right? So you have a very small number of producers, NVIDIA and AMD generally. And then NVIDIA is very far ahead on the high-end processors that are most efficient for large-scale training and infrareds. Then you have the pandemic supply disruption, which we haven't fully recovered for. If you actually look at the supply chain, you go from the actual designers to, you know, the reliance on a few major foundries like TSMC.

1:28You know, expansion of this capacity is not easy, right? New fabs are billions of dollars. Yield is a very complicated thing. You can think of it as a massive precision manufacturing problem where temperature, pressure, chemical concentration, tool imperfections, new processes, materials issues, like anything can make production have lower yield or lower quality. And so, like, if you think about the speed with which the industry, driven by both large and small players, has decided that they want to do AI, like the physical processes cannot keep up with that demand. It's as if, you know, half the companies in the world over a year long period decided like, yeah, we need supercomputers, not superconductors, but gigantic networked GPUs.

2:22So like, what is the actual gap? So to your point, it sounds like much of the AI world is dependent on GPUs in order to train and then do inference on these big AI models. and the big suppliers are basically NVIDIA, AMD, and then there's like a long tail of smaller folks. What is the delta between the amount of capacity that exists today and that's needed? Are we off by 2x, 10x, some other number? It's hard to say because right now there's no way to explore like the price elasticity of these things, right? So, you know, just very specifically, like the industry is kind of looking at deliveries in small quantity in September, larger quantities in December, January, most of the large cloud providers are sold out for any scale for at least through April of next year.

3:15And so you have like really interesting dynamics, like large cloud players who, you know, are the biggest consumers of these GPUs already, like a Microsoft going and buying from other providers for near-term supply, right? So I think one question that I ask you is like, hey, do you think this is a long-term thing? Do you think it's a very short-term thing? But I think it just goes back to like the fundamental dynamics are, do you expect the demand for these chips to continue increasing at a pace that outcreases the ability to scale a very physical, like real-world process, right? Just to even be more specific, one of the challenges, like I was talking to Jensen about this.

4:00And a bonder, like not part of the GPU itself, but like a critical tool in the manufacturing and assembly of GPUs is very specialized. And so the ability to build any of these tools as well to enable these processes is a blocker. If you look at the demand from large labs today to continue increasing model scale and training time by magnitudes, I think it's hard to see that dynamic going away. What do you think? I feel like there's a couple different sort of second order implications of the fact that we're seeing this giant GPU bottleneck. I think the first one is that we're seeing new sort of models that are dependent on GPU access or ownership as ways to create all sorts of really interesting monetization and potentially eventually cloud services.

4:48So that's things like CoreWeave or Foundry ML or other companies that are basically providing now GPUs in different ways, in some cases through aggregation or federating different sources of GPUs. In some cases, it's just having these large GPU clouds and being able to use them in really interesting ways. And one of the interesting, I think, side notes is that GPUs used to be very heavily used for crypto mining. And while crypto is down, it may actually be more economic to just use them to rent out for AI training purposes or inference purposes. So I think that's one really interesting, almost like sectoral shift in terms of existing GPU capacity.

5:23The second is that a lot of the different players that are startups who've built their own semiconductors specifically for AI training, I think are starting to see a lot of really strong pull. So for example, Cerebris, and I think we're going to have Andrew from Cerebris on our podcast in a couple of weeks. They just signed a hundred million dollar deal with UAE for building nine supercomputers using their chips, which are optimized for AI. And so I think they and Grok and other sort of semiconductor providers are going to find really strong pull during this period where people are desperate for any solution and they're willing to do take the extra steps to really be able to utilize other forms of silicon and so i think it creates a bit of an opening for other players in the market and so it does seem like it's going to have these really interesting sort of cascading effects on members of the startup ecosystem and you know new players that are working against all this two um sort of second order things are like what do you do when scaling is blocked on capacity?

6:22Like you try to be more efficient. It's not been an area of massive focus to date because people have been chasing the state of the art following chinchilla scaling as the simplest path forward. But there are really interesting lines of research that are undervalued today unless the hardware supply crunch continues, including in dynamically figuring out or routing to efficient models. So think of like the frugal GPT work or generally like distillation or even just a more intelligent choice of data for your pre-training or your fine-tuning training mix so you can use less compute, right, for the same or for improved quality.

7:07And I think like everybody's been on this one path and an interesting second order effect is like, does it spread people out into lots of different directions in terms of chasing performance? I personally don't think the supply crunch goes away immediately. And like, a part of the dynamic is just, you know, how much more people want to scale. And another part is like, you know, if this stuff is actually useful, then inference, like inference already dominates open AI compute usage, right? And so that demand will continue to go up. Yeah, I do think demand will only rocket from here, at least in the short run.

7:46And so the real question is the degree to which the semiconductor industry adjusts to that. And the reality is that people really view NVIDIA's chips as the most advanced on the market right now. And so that means that a lot of it is just a bottleneck in how much can NVIDIA scale up manufacturing. And there's other players like AMD, there's the startups we mentioned, Cerebra, Sgrok, and others. But a lot of the capacity is just going to be how much can NVIDIA and maybe AMD scale up in the short run, at least. And so that may just cause some ongoing bottlenecks, assuming, again, that we continue to see this very rapid growth in AI and AI applications.

8:23I'm working on a blog post right now actually about this because it feels to me that we're still in the very, very early innings of this wave of AI adoption, right? It's not a continuum where we had CNNs and RNNs and now suddenly we have Transformers. Transformers created a whole new capability set. And we're only, you know, eight months since ChatGPT and a few months since, five months I think, since GPT-4. And so the only people who've really adopted this technology yet are the AI native companies like OpenAI and Midjourney and a few other folks. And then we had the first wave of startups come, the perplexities and Harveys and characters of the world, as well as the first wave of incumbents adopting it, Notion and Zapier and sort of the very, very early founder driven adopters.

9:05And so we've had zero real enterprise adoption in terms of real products at scale or close to zero. and you know most enterprises big businesses take six months nine months to see their planning cycles and then they'll spend a year prototyping and then finally they'll launch these ai apps and so we're probably a year or two years before we really start to see large-scale ai applications by existing incumbent enterprises in real products live everywhere so from a ramp perspective one can imagine that a lot of the future ramp and ai is coming uh in about two years or you know one to two years, something like that.

9:41So there's still a lot of room, I think, for the hype cycle, for increasing ongoing excitement, sometimes irrationally so, and then also for sort of adoption of semiconductors and other underlying infrastructure. So there's still a lot to come, it feels like. I agree with you. And I still think we're really early in, let's say, like the collective exploration of applications and constraints, right? Like you had the people who were bleeding edge of just personal interest. Like I think ChatGPT is looked at correctly as the starting gun for people to begin developing these AI applications generally.

10:21But if you think about how long it takes to ship actual interesting products to market, and then the buildup of some collective understanding of like how to make these models more useful in different applications, and then turn them into workflows and then advance the state of the art given a particular workflow if you have a hypothesis on value. Like that all takes time. So I think we're in inning one. Yeah, it's all been demos so far. Yeah. So I guess related to that, a lot of the interest and excitement right now is around agents. You know, I spoke recently, there's a group called the AGI House, which hosts these different hackathons in the Bay Area and stuff like that.

11:00And they had me come and help kick off like an agent hackathon they had and things like that. What do you think happens with the agent world? Like what form does that take? And is it a handful of very broad agents? Is it highly specialized ones? Like what do you think is coming there? Yeah, it's such a like powerful, broad idea that I think both will happen, right? And so like the overall idea is you don't just talk to a chat bot or query an interface. You have some sort of planning mechanism that is model-driven that allows you to take asks autonomously, take actions autonomously, and, like, complete a more sophisticated task, often using other tools, and then return that result or report back on your work to an end user, right?

11:52And so, you know, I think that is going to range from the pure consumer applications. So things like inflection, which is going to, you know, have personalized that do more for you. Minion, which is working on like web agents. And then, you know, I think like there's been very recently more attention or just more understanding of how powerful it is to have agents that in some way write executable code. Right. Because you can programmatically use many more tools. You can call APIs. And I think if that is do a task that is not a single query but requires multiple steps in analytics or in enterprise automation or even, you know, within like, you know, companies that we work with like Harvey, like a single legal task is actually a composition of thoughts, planning, attempts of research.

12:52like writing that an associate might do. And so I think it's going to be a pretty dominant paradigm. Yeah, it's kind of interesting, because if you look at past technology waves, and you ask about specialization versus sort of broadness, you know, are you building a broad based platform that you can use for anything, or a vertical application that really helps you with one or two things? Well, most of the things that really work are these vertical applications that help you really well. Now, some of them broaden and grow into the broad based platform for everything, right? Even in consumer, that's true.

13:22Like Facebook started off as a college network. And in fact, it started with like five colleges and they added all colleges. And then later they added the ability to add your work email as a way to register. And then they opened it up to everybody. And then they started building the platforms on top of it and gaming and other things. Right. But it kind of happened sequentially. And there's kind of examples to that. You know, Google would be a very broad based thing from day one. It helped you discover information on the web, right? You needed a tool for that. But it feels like in the agent world, a lot of the people that I hear talking about ideas have these very broad sort of abstract ideas.

13:55And so an idea would be, I'm going to build an agent that is going to be your assistant. And you're like, okay, well, what is it going to help me with? And they say, everything. It's going to make you happy. And you say, well, I'd love to be happy. But at the same time, starting with a very targeted, focused initial use case tends to be the best way to build product, A, because you know who you're building it for, B, you can really nail the use case. And there's the old sort of YC-ism, which I think, which is really good, which is, it's better to delight a small number of people than to have a very large number of people indifferent to your product.

14:28And so I think my bias for the agent world is, if you're building an agent, start with something really targeted. If it's an assistant to help you, what exactly does the assistant do? Does it do background information searches on all the meetings you have that day? Does it specifically help with certain forms of scheduling? Does it help with other aspects of your day planning or synthesis of what you've done or follow-up action items or whatever it may be? But choose one or two things and do them very well versus do everything. And then eventually you may build the thing that you start off that does one thing very well, but then broadens into everything.

15:01But usually starting with everything means you're not really doing anything deeply or well. And so I think that to me is one of the main patterns, at least in terms of prior ways of technology development. I very much feel like this is like a very classic tension between what I consider to be like, I don't know, the like infrastructure, platform engineering, like even research agenda driven approach that is like, oh, you don't understand. Like the technology is general. We don't want to be taken off the research path that pollutes our data mix in a way that it is not a general purpose technology.

15:39anymore, right? Or, you know, it can do anything, why limit it? Or even getting feedback from users, because you release this stuff, it is broadly capable that they're doing everything with it, some things much more successfully than others. And I think more of a like a product engineering, like traditional, like startup mindset that is like actually complete the task, right? And I definitely think the overall exploration has been skewed to one side, not as productively today. And one of the, like, even if you think from the research agenda, one of the reasons it is interesting to think about the, like, the, you know, have more focus, everybody's thinking about, but have more focus on accomplishing the specific task is like, you want to be happy a lot.

16:27All I want to do is like never write boilerplate code again, right? And so if you think about... That's how I define happiness. Okay, great. Then we're still the same. But, like, if you think about, like, okay, let's, like, complete one task. If I want to ask, you know, an agent to just, like, fix all the bugs in my software, then my ability to, like, successfully complete that task includes a lot of, like, bug-fixing specific techniques, right? You could do test time search and then see if all of the different things that you generated actually execute as one very simplistic example. And so I think there are a lot of ways to advance in the research in very specific tasks that are much more tractable.

17:17But maybe I'm not thinking big enough. That makes sense. I think I would add one third piece to that framework you have, which is the research-driven versus product-driven. I think there's a third approach, which is infrastructure tooling-driven. And that's why you're like, I'm not going to build the agents, but I'm going to build the infrastructure that allows anybody else to build them rapidly. Now, sometimes those types of businesses or approaches work really well. And sometimes those things are solely an outgrowth of a vertical product that works really well that you then open up the infrastructure for everybody else to use.

17:46And it's very case by case dependent. It's a difference between Stripe, where it's just like we need to build payments for everybody. Everybody keeps building it over and over again. and the Facebook auth platform, which only existed because you got to hundreds of millions of users so you could open up auth as like a third-party service. And so I think as people think through that third angle of building infrastructure for others, they need to understand whether that infrastructure will be an outgrowth of an existing product area and benefit from the characteristics of the market liquidity of that product or whether it's just a piece of infrastructure everybody keeps building over and over and therefore it's a really good thing to just provide to the world.

18:20So I think it's kind of an interesting future topic. We are on a couple month bull run at this point. 2024 tech markets. What's coming? Will people be able to fundraise? Will funds be able to fundraise? Are customers purchasing? You know, I think there's going to be basically four markets next year in some sense. One market is just AI, and I think AI will continue to run in different ways. And it'll look very expensive at the time, and a handful of companies will look really cheap in hindsight, just like with every other technology wave. And I think that's separable from the rest of tech that existed prior to the AI wave.

18:56for companies that fundraised in 2021, prior to being like AI companies, a subset of them, I think if I were to sort of divvy up that pie of those companies, sort of mid to late stage private tech companies, not in AI, and what's going to happen to them next year and in 2025, I think a third of them are just going to go under, or a third of, I should say, unicorns. We'll eventually just go under, be fire sales, whatever. They won't be able to ever raise money again. A third will be at the highest valuation they'll ever be at ever in the lifetime of the company. They'll reach their terminal value.

19:30And there's examples from 2014 of companies that went through that same wave. They raised in 2014, they went public a few years later, and then they never surpassed their market cap again. And then I think lastly, there'll be a third of companies that grow past it. And so I do think there's going to be a lot of carnage next year and a lot of companies going under. And as those companies go under, three things will happen. Number one, it'll be much easier to hire people. And people are already seeing that at startups. It's easier to hire again. Second, it should have follow-on effects and ramifications for commercial real estate.

20:00And we'll see a second shoe drop there. And then third, the venture capital community will be impacted because a lot of the things that they've been using to fundraise new funds or do other things with will suddenly go to zero. Their big unicorn success will go from a multi-billion dollar or billion dollar company to basically a company that isn't worth anything. And so I think that's going to have knock-on effects to the venture ecosystem. But I think that'll take like two, three years to play out because all these things are a bit time delayed. But yeah, I think that other shoe still hasn't dropped in private tech markets.

20:34And a lot of it is just companies raised so much money in 2021, they still have lots of money. So everything still feels like it's continuing to go. But at some point, that money is going to run out. So I think it's going to be a pretty bumpy 2024 and 2025, potentially. Yeah, my advice to companies that, you know, raised a very healthy valuations during that period of time, and then are actually building businesses is to try to completely disassociate from that valuation. Because people will put themselves into all sorts of contortions to do a flat or up round to a valuation that makes no sense, right?

21:14And if you don't have the historical context of that making no sense, it's an extremely painful sort of realization to have. But if you look at, there's this one analysis of actually the very best technology companies, and the ones that endured from the internet bubble, and how long it took those companies to reach the valuations they were at before the bubble burst. And it's a decade, right? And it's like startups don't have a decade to try to get to at-par valuations. Yeah. I'm actually less worried about valuation. I think valuation is ephemeral, right? Effectively, roughly every tech company in public markets did a down round over the last year and a half, right?

21:58They all lost, or many, many companies lost 30 to 90 % of their value, right? And effectively, they just did a down round in public markets because every day you're repricing a public stock. I'm more worried about the people who burn tons of cash and they don't have a lot of revenue to show for it. And then when they're going to go out to raise more money, people say, well, you burned$50 million, you burned$100 million to generate five or$10 million of revenue. And so the issue isn't that your valuation's off, we can always reset valuation. It's the fact that you burned all this money and you don't have anything much to show for it.

22:28And that's where I think the real issues will happen because you can always reprice things and people will be forced to and, you know, it'll just happen. But I think it's the underlying business case and business model that's going to be the real issue. Yeah, I guess like the unforced error there for companies who actually have the time to make the decision is the thing you want to avoid is like not adjusting your cost profile or, you know, holding on to that valuation until it's too late. Yeah, or just deciding it's the wrong business and it's not working. And, you know, the most important precious thing for you as a founder is your time.

23:01And I think people forget that you have this golden period in your life where you don't have, hopefully, a lot of other complications in terms of sick family members or school-related issues or whatever it is. And you can take risk and you have a low-cost basis and you can do all these things. And that's the moment when you can best take risk to start a company for many people, not for all. And, you know, you're really giving up the best years of your life working on things that potentially may not work. thanks for the discussion it's a lot of fun yeah super fun thanks to everyone who sent us your questions find us on twitter at no priors pod subscribe to our youtube channel if you want to see our faces follow the show on apple podcasts spotify or wherever you listen that way you get a new episode every week and sign up for emails or find transcripts for every episode at no-priors.com

23:59Thank you.