#229 Mitesh Agrawal: Why Lambda Labs’ AI Cloud Is a Game-Changer for Developers

0:00I think it's important that people stay very optimistic about this technology. Of course, we have to have, you know, controls around a little bit controls around how people it doesn't go into the wrong side of things. I want to encourage people to to be in the know about this. Underlying it all, like, of course, we want to build a big, successful, long term business. We really care about that. It's kind of our baby to build that. But underlying of it all, we really want to be one of the ones that provide a lot of compute and ease of compute to people that are building the next step of applications for this world.

0:32And we really focus on the deep learning aspect of it. We spend a lot of time building for it. And for a lot of listeners, again, I would encourage to try Lambda Cloud if you're building any deep learning application. What does the future hold for business? Ask nine experts and get 10 answers. Bull market, bear market, rates rising or falling, inflation going up or down. Can somebody please invent a crystal ball? Until then, over 40 ,000 enterprises have future-proofed their business with NetSuite by Oracle, the number one cloud ERP, bringing accounting, financial management, inventory, HR into one fluid platform.

1:18With one unified business management suite, there's one source of truth, giving you the visibility and control you need to make quick decisions. With real-time insights and forecasting, you're peering into the future with actionable data. If I were a larger organization, this is the product I'd use. Whether your company is earning millions or even hundreds of millions, NetSuite helps you respond to immediate challenges and seize your biggest opportunities. Speaking of opportunities, download the CFO's Guide to AI and Machine Learning at netsuite.com slash ionai. That's netsuite, N-E-T-S-U-I-T-E dot com slash ionai, E-Y-E-O-N-A-I, all run together to get the CFO's Guide to AI and Machine Learning.

2:21The guide is free to you at netsuite.com slash ionai. netsuite.com slash ionai. Why don't you start by introducing yourself and how you got to Lambda into this business? I'm very happy to, and thanks for having me here today, Craig. Like, look, I think the best place to start is always with the beginning of a company and for Lambda. You know, I think the best thing was we all were engineers and in machine learning fields. And, you know, when Lambda was initially started off as a face recognition API company. So basically, we had an iPhone app that used to take your photos and convert it into a Van Gogh-style painting or a Picasso-style painting.

3:17It was called transfer-style learning back then. And we used to have our processing done on the cloud. And it used to cost a lot of money. We were never profitable doing that business. What we realized at that point of time was that actually we could purchase the hardware that we were running it on, the GPUs, and we could deploy it and we could install what we now call Lambda Stack. But at that point of time was just what is called a Debian repository, just a collection of all the libraries like CUDA, CUDNN, PyTorch, TensorFlow, languages on it and make it super easy to run. and it immediately like took out all of our kind of spend on the cloud.

3:59So we saw that there was a gap in the market where, you know, you could buy a Dell, HP, whatever hardware, but really like to make it work for machine learning, you have to install this software stack and really support it. And that's kind of how Lambda got started. And for me, I think, you know, kind of an interesting journey, you know, being in Bay Area and Stephen, who is the founder of the company and machine learning researcher himself, um you know we we were just introduced and we're talking and and he was like hey there's this opportunity we we're pivoting into infrastructure um i i was from a background of chemical engineering and and have been working in quantitative finance but was trying like starting to see early inklings of kind of at that point of time you know it was just like advanced statistics basically like just prediction prediction base and things like those was trying to understand these models and i was hey, this is a good way for me to foray into this world.

4:55I really understand the computational layer. And I joined in as a very, very early employee, number fifth at Lambda. And pretty much as the saying goes, did everything that the company needed it to be. Was it an engineer? Was it a support person? Was it an HR person? Was it an office manager? Was it a salesperson? Everyone did everything. and that's how Lambda kind of grew really from both my perspective and this external perspective is like we really branded ourselves into this infrastructure world but only very much focused on what we call deep learning applications right really like making sure that at that point of time the state-of-the-art models were convolutional neural nets the res nets all of those models really making it work for that and then over time you know look company's mission was really focused on bringing compute as quickly as possible to the machine learning community.

5:53And we realized we had an opportunity in 2020, 2021 with cloud starting to grow. But what we noticed was the hyperscalers were still, and everyone was in early innings, actually. And no one really foresaw ChatGPT, maybe other than OpenAI. And even they would probably say that they didn't foresee the enormous success of it. But we realized that the compute will be a very critical category and compute design for AI will be a very critical category. And we kind of stepped into that where it's like instead of just selling infrastructure to be deployed in an end-person environment, whether that's through a desktop or laptop at your office or through a cluster in a data center, we said, why not build?

6:37We are experts at building clusters. We're experts at the software stack about it. let's build it in our data centers and give people remote access, aka like an AWS or GCP-like service, but for deep learning. And really optimize on kind of the networking stack, the software stack, the speed at which you could do the model. And that's kind of how Lambda got the first, like, really, like, foot down into the space. And then, you know, we saw the growth curve. We got attracted to the growth curve and really then doubled down on building this kind of AI cloud and cloud for developers and researchers and enterprises, again, keeping to a theme of only focusing on deep learning applications, but really building around that.

7:19So that's been the last journey of eight to nine years for Lambda. I'm just curious, this style transfer app that you guys have, what was its name? Yeah, it was called DreamScope. So, you know, it was like, look, I mean, of course, we look at it with very like loving eyes and it was like the mid-journey before the mid-journey right you know i think mid-journey is a brilliant brilliant application that has been developed right now and and and we we we did this back in 2016 2017 uh and really it got a niche following in the arts community because you can imagine right it just it wasn't like a gender develop right you couldn't type something and it would generate it but what it would do is take an existing piece of whether it's an image or a series of images and then convert it into any piece of art that you want.

8:10And we really focused on kind of the artistic, like the noir style, Picasso, all the artistic styles that people do. And it got over a million downloads within a few months of launch. We had a package called, a paid package called$9.99 and we had a lot of paid users at one point touching well over 100 ,000 ARR, which at that point in time, I mean, now it's very small compared to what the AI companies do, but at that point in time was a new and radical kind of way of approaching the consumer marketplace. And, you know, it's a baby, but like, you know, we had to, first of all, a lot better models came out, people who developed their own models.

8:51And then second, because we focus so much on the AI learning infrastructure, we realized that focus is key to growing this market. And so we, as a company and as an expertise area, we really pivoted into building out infrastructure for deep learning engineers because we ourselves were users of those. Yeah. The reason I ask is I was fascinated by style transfer back in 2016, 2017. And I did use a few apps. I used Picasso with a K, P-I-K-A-S-O. And there was another one I can't remember. Yeah, even I'm blanking on the name. It was a very popular Russian app. Damn, I'm blanking on it. But yeah, I know too.

9:34I've used, yeah, exactly. Stryle Transfer were the first really consumer applications of deep learning. I'm actually surprised that it didn't take off or become more popular. And then there's another company that's doing something kind of similar to Lambda, PaperSpace. I got to know those guys years ago, and I can't remember what they were doing, but it's the same kind of story. they were doing something in that you guys were working on style transfer, and they saw this opportunity for GPUs, and they moved into that space. Why are you cheaper than a hyperscaler? Yeah, I think that's a fair question that you can imagine we get asked across the board, whether it's investors or the media or just users.

10:38Look, I think there is a few things that go into it. First and foremost, let me be honest. There's not like a huge magic wand here that is somehow making like, you know, Lambda has like a way of producing the cost structure or because we get GPUs that are cheap or nothing. Nothing like that. So there is a few things that go into it. One is just Lambda's philosophy. I'll be very frank with you and with everyone listening in and we have been with our investors too. We really want to grow the access to compute for all of both, not just like incoming developers and students and things like those, but across even enterprises.

11:14Before the advent of this kind of AI cloud companies, including Lambda, the GPU marketplace was a little hazy in terms of pricing. You had your website, even today, if you go to AWS and you look at AWS prices on the website, for Hopper, it's like$5 or$6 or$8. Almost no one is paying for that. And that's been the reality for a long time. But you had these different pricing strategies that the hyperscalers did. Lambda went in with a simple task of transparent pricing, straight pricing to the market that we believed allowed us to both grow and capture enough margins to continue to run the business and be successful.

11:57So that was one, it's just philosophy around pricing. Two is this, and this is one of the technical reasons. Again, not a magic wand, but if you think about hyperscurses, they have to support many, many hundreds or thousands of use cases, right, as they go. So they have microservices and supporting infrastructure for those services that they have to build. So when they deploy a GPU cluster, they don't just think about the GPUs, the CPUs around it, the memory, the storage, the networking. They have to think about a hundred more myriad different things, how to get interconnected to all the data center regions, how to have all the services.

12:35So what that results into is a slightly an additional cost structure that they want to recoup, but also a little bit time where if Lambda were to get the GPUs today and let's say Hyperscale were to get the GPUs today, we would get that deployed way faster. Not again, not because we have more people or anything like that. It's just because we don't have to think and optimize for that. We have to optimize for deep learning. So the focus there helps. And as such, if you have less time to market, so you'll spend less on operational costs, you have to don't think about the microservices and things like those.

13:08So it really allows you to price way more competitively. And the last thing I will say is this. Look, almost all the hyperscalers are public companies. They have certain expectations to meet on what their gross margins are, what their operating margin structures are, and things like those. For Lambda, I think being a private company here and really optimizing for growth allows us to take lesser margins than hyperspace. Again, as long as the business is sustainable, we go through that. So there's just multiple things that contribute to it, including, as I said, the first thing that really contributed to it was us, ourselves, as a philosophy, we wanted to provide as much compute as possible.

13:50And we were advised by fellow deep learning researchers and engineers who help us at the company and who are advisors to us saying that, look, the best way you could grow is by making sure that everyone understands and learns about deep learning and that there is an accessible way to use that compute. So it kind of makes sense from that perspective. Yeah. And then growth, as you grow, do you, as demand grows, you add GPUs or do you add GPUs and then sell the demand, sell the capacity? How does that work? And do you have any trouble getting GPUs since there's famously a shortage of GPUs globally?

14:33Yeah, two very kind of circular situations or catch-22 situations, as you mentioned, right? I think, I'll be frank, I think the last two years have been, we've been blessed in the sense that on the demand curve, the demand curve has grown so fast that no matter how much supply you're providing the market, it was above the supply curve, right? The demand curve was above supply curve. So to your point, whether, you know, how many of our GPUs we kind of ordered and deployed is a function of less of what the demand curve of the market is. So to your point, whether we saw demand and then deployed, no, it was more a function of how much capital do we have to purchase the GPUs, how much data center do we have space and power do we have to deploy these GPUs, and how much can we, to your point, how much can we get allocation of the GPUs.

15:27So it was more a function of those, you can think of like commercial factors rather than really the demand curve. The demand curve has always been there. So from Lambda's perspective, we pretty much always, if someone is doing like a reserve, what is called a reserve contract. So like committing for a year, two year, three year for compute, we're kind of always sold ahead of the curve. Like so people have come to us and say, hey, I want a thousand GPUs. I want to 5 ,000 GPUs. and you say, okay, great, give us three months and we'll get it deployed for you or things like those. So that's on the reserve side.

15:59Now where Lambda has kind of differentiated ourselves is a little bit more on the on-demand compute where to your point, we deploy compute without having an associated reserve or committed contract to it because we want to. We want to create a marketplace where their people can come to Lambda, see a GPU available and spin it up and use it the moment they want it for how much of their time they want it and then spin it down. We want to create that. So we encourage that marketplace. So we basically assign a certain percentage of the GPUs that we get for that market. Now, we have been, again, as I said, both lucky and as a business, really, you know, I guess having that kind of visibility into the pipeline that GPUs as a demand curve has always stayed of supply curve.

16:45So it has allowed us to, even when we have booked it for on demand, And those have pretty much immediately sold out. So we reach high levels of utilization within a few days to weeks of launching those GPUs. So far, it's been always ahead of the curve. In terms of securing GPUs, I think for Lambda, I think NVIDIA has created a partner-friendly ecosystem. So if you think about it, how much GPUs Microsoft would like or want, probably nvidia could park all its gpus at microsoft you know maybe i think uh but no nvidia has been very i think uh much more i guess in a way democratic and i think in the sense of like how they have uh provided gpus and and part of it is just that they follow the usual traditional supply chain principle of first in first out so in the sense that you mentioned how do we get allocations you know again not a huge no matter kind of what you see in the media it's not like a like a, oh, like, you know, somehow we go to NVIDIA and say like, hey, we are your favorite people and can you please give us GPUs?

17:52It's more around the fact that like, look, we have this demand curve. We are placing a purchase order six months, nine months ahead of time. And we are committing to this purchase. And as such, you get into the queue of where the GPUs are going and you get those allocations. And so you have to do the supply chain planning very much so to get the allocation. You know, so I think over time, you know, as Hopper, when they first came out, it was hard to get for everyone in the amount that you wanted to get, because, again, demand was about supply. But, you know, everyone got whatever they placed their queue into the, how much ever they placed their order in and queue in.

18:30And accordingly, everyone got the ratios as NVIDIA also ramped up its production system. Now it's like very, very much accessible to get a Hopper GPU in the Edge generation, Edge 100 or Edge 200. but now the same story is unfolding for the Blackwell generation you see it in the media report and they're constantly saying they're sold out and they truly are because all of us have placed you can imagine hyperscalers and us all of us have placed orders for Blackwell GPUs now almost probably like 12 months to 6 months previous to today's date for Blackwells that are going to get delivered later this year, early next year so you are in the queue to get those but let's say if you were to start a cloud company today or go out and place an order today, I mean, you're potentially waiting for like eight, nine months to get those Blackwell GPUs, such as the demand for those NVIDIA Blackwell GPUs, right?

19:24So yeah, that's kind of how the allocation curve goes. So you have to take certain risks to get the allocation and you have to constantly assess what your sales pipeline looks like, what the market demand curve looks like. But so far, it's always you kind of plan your GP deployments well ahead of time by both having visibility on the demand curve and also assuming that demand curve continues to stay above supply curve, which we really believe in that way. Yeah. A couple of questions. The workloads that you're running, are they primarily training workloads? So that's been an evolution. I think if you had asked me that question, like, let's say, 12 months ago, I'd have very strongly said yes.

20:09I think a lot of the new spend dollars in infrastructure, especially in the deep learning space, is going towards primarily training. And I include fine-tuning as part of the training bucket in this, especially since open source has really kind of allowed people to fine-tune a lot more now with the models being of very high quality. But even until very recently, I think training was the new major spend because for training, you need a huge chunk of compute all at once, all interconnected in a way and so on. And inference is an application-based usage, right? You know, if you are a consumer like company that is providing some application, if it takes off, great, you need a lot of GPUs.

20:52If you're just starting now, you can start off with a few GPUs. So until very recently, training was the majority spend, and that reflected in Lambda's own deployments and in Lambda's own usage as well. I do want to very specifically answer generically. The way Lambda Cloud is built, it can be used for both training and inference. That's a result of both what NVIDIA has done with its high-end GPUs, where they have made it extremely good both for training and inference. So if you look at Hoppers, if you look at Blackwells, they are extremely effective training and inference GPs. You really can't say that, oh, it's really great for inference, but it sucks for training.

21:31It's really not true. They're absolutely top of the line for both those applications. So first of all, that architecture and infrastructure itself helps both training and inference. And then the way Lambda designed its data center too, it's like, of course, we design more for training in the sense that we try to get as many GPs all interconnected together. but the same setup can be used for inference from that perspective. And so far, inference is not bottlenecked by some of the factors that you think they'll be bottlenecked by, such as latency, right? If you're a user in New York and your GPU is in San Francisco and you're using chat GPT or an image generation, you're okay because the networking time lag, the latency lag is hundreds of milliseconds.

22:17but the actual computation time is in thousands of milliseconds or even seconds. So that networking lag is not effectively your bottleneck. It's still the model processing time. So right now inference actually works very well. It doesn't matter whether you run it in a training type cluster. And as models are becoming bigger and bigger, they actually need that kind of interconnected GPUs. So in terms of the market use case, Yeah, training was the primary driver of revenue or GPU use cases for Lambda. I think if, you know, we constantly do service. It's very hard to always know, like have posts on the market exactly how much usage.

22:55But when we right now speak to our users, I think inference probably in terms of the new dollar and new GPU spin up, I think inference has overtaken training in terms of how many GPUs are spin up. Like we still have training customers. We still have fine tuning customers. but in terms of when we're deploying new GPUs and how they're getting allocated, it's more towards inference. When Blackwells do go live, though, I expect that ratio to, again, flip a little because the first foray into Blackwell will all be training customers, right? Because that's what they're going to try and get advantage of the Blackwell.

23:27So it kind of switches like that. But overall market, I think inference is starting to take over in terms of the dollar spent as more applications are coming. Yeah. And the reason I ask that, yeah, I think the same thing, too, because, you know, foundational models are the foundation, but applications on top of them are all about inference. And I've been talking to Andrew Feldman at Cerebris and Rodrigo Leung at SambaNova. Are you starting to buy those chips for the inference business? Or if not, why are you sticking with GPUs when these are so much better for inference or Grok for that matter?

24:20Yeah. So the first answer I can very explicitly clearly say, we currently only deploy NVIDIA GPUs, both for training and inference applications. And we have not yet forayed into deploying ASICs or alternative accelerators. inference demand is definitely starting to take off i think one of the key points there just quickly on that note is if you think about foundational models you know when the whole kind of ai we've especially around transformer started there were way more than three four dozen companies all trying to do foundational model training i think now it's getting concentrated the models are becoming massive like i mean you hear about 100k gpu clusters and so on but it's getting concentrated on how many people do it too right so more people are going towards fine tuning and then figuring out applications for it, which is also driving inference use case.

25:11Now to your point where companies like Cerebrus, Grok, Sambanova, even a few others, they're focusing on inference as a service and inference as an application, right? And all of them have their own kind of ways of niches. For example, Grok really focuses on the speed, tokens per second, throughput. So Rebrus can fit a big model, but is also competing on tokens per second throughput. So the speed, they compete on more speed than a GPU, Nvidia GPU could. For us, there's two things. One, these are still kind of up and coming accelerators and use cases that yet don't have that massive market economic drive for us to be the first adopters.

26:03Look, Lambda, I'll be frank, we are a growing company. I would say we have a limited amount of money in our balance sheet, and we want to basically put that money towards guaranteed revenue generating ability. And NVIDIA GPU today is that, right? And it's a universal card. Every single application runs on it. People optimize for NVIDIA GPUs. People build for NVIDIA GPUs. You know for a fact that every single application will work on that. I think it will be very interesting and it's really, I think even NVIDIA will accept that it's great to have competition because it keeps everyone on their toes and from that perspective.

26:44But from our perspective, really right now, it's more around both capital allocation and what we see as all-purpose applications, like where we want to be. We can't yet determine the niche of where do we want to really support that. So that's kind of what has led us to really focus on the NVIDIA GPUs in general. I think as a software stack on these companies become more robust such that we can deploy infrastructure with them, I think there will be considerations, of course, for this. But to be frank, from our viewpoint, I don't yet know things can change, but I think we'll be looking for guidance at other hyperscalers, at other players to first introduce this to market, see the market capture, and then maybe consider it.

27:30But for us, very clearly right now, it's, you know, NVIDIA GPUs are, I mean, if you look at the market capture of NVIDIA GPUs, it's well over 90 % of the total compute, right? So we will stay within that lane for now. Yeah. The other question is you were talking about inference using your cloud and the latency is all in the computation, not in the transmission. How far away are your biggest customers? or rather how... The distance between data centers. Yeah, I mean, are people globally using you guys to train or is it really a U.S.-based clientele? Yeah, I mean, look, I think it doesn't matter which cloud you're speaking to, the majority of revenue is coming from U.S.-based clientele.

28:35We do have, as it's always the case, we do have global customers. And the part of it is for training, truly, we have had customers from Europe and Asia because it doesn't matter. And vice versa, we have deployments in certain other countries that U.S. customers have used for training. For inference, though, I think majority of the user base for us is U.S.-based, first and foremost. and a lot of our inference deployments are in the United States, and it's the usual regions, whether it's kind of West Coast, East Coast near Virginia, Chicago area, Texas areas, kind of thing, and the new up-and-coming regions like Salt Lake City area and some other kind of regions where there's more power, like Atlanta and things like those.

29:27um but uh yeah generally it's it's it's united states based and and by the way for the latency ones for the smaller models it is starting to become more important right and again this is you know not kind of trying to harp on it but like going back to your point around grok and samanoas of the world they kind of for the the throughput that they they have the speeds that they have latency would matter right like especially for smaller models like an 8 billion parameter or now a 3 billion parameter model. I think that's where latency does start to matter. And I think if you probably, I think Andrew will agree with this or things like those, as AI starts to feed input into other AI applications, latency will also matter.

30:12Right now, even if you are the top reader in the world, the throughput speeds are high enough that throughput can be faster than yours or at least my reading speed. Let me just talk with myself. So it kind of doesn't matter to me when that output is like, whether it's faster by 10x or 2x, as long as it's faster than my reading speed, great. But as those outputs get inputted into other AI models, and there's a chain of AI models or agentic behaviors or actionable behaviors from me, I think it would really matter the speeds and latency. And that's, I think that's why these guys are also correct in saying that like latest, like throughput speed is important.