Modal and Scaling AI Inference with Erik Bernhardsson

0:00Modal is a serverless compute platform that's specifically focused on AI workloads. The company's goal is to enable AI teams to quickly spin up GPU-enabled containers and rapidly iterate and autoscale. It was founded by Eric Bernhardsen, who was previously at Spotify for seven years, where he built the music recommendation system and the popular Luigi workflow scheduler. In this episode, Eric joins Sean Falconer to talk about the motivation for founding his company, the market gap in ML and AI tooling, optimizing container cold start, Modal's interface design, and more. This episode is hosted by Sean Falconer.

0:38Check the show notes for more information on Sean's work and where to find him.

0:54Eric, welcome to the show. Thank you. It's great to be here. Yeah, thanks so much for being here. So I was diving a little bit into your background preparing for this. And so it seems like you spent a lot of your time working in data throughout your career, which also kind of matches my own experience. You know, you previously were at Spotify for a number of years, you were the CTO of better.com. Now you're the founder and CEO of Moldal. You know, were there certain things in these prior roles that led to identifying some sort of need for Moldal? Like, what's the story behind essentially going off and deciding to start this company?

1:25Yeah, for sure. The answer is yes. And the long story is I was at Spotify for seven years, built the music recommendation system. But as a part of building that, I also realized there's kind of a general gap in the tooling. I ended up building a vector database called Inoin that we use today, and also a workflow scheduler called Luigi that very few people use today. But generally realized like as a part of building all of that stuff at Spotify and also did a lot of other stuff, that there's very little tooling in data, AI, machine learning. There's more today, but I still never really felt like much later, like in 2020, 2021, when I started thinking about building a company, I realized there's kind of a gap in the market for like a tool I always wanted to have myself.

2:05So that was kind of the genesis of Moto. It's almost like building selfishly for what I always wanted to have, you know, throughout my years at Spotify, building a music recommendation system. And also, just to make sense, I did better. I was a CTO, so a little bit more like general role. I spent a lot of time thinking about platforms and data and stuff like that, too. Yeah, I mean, sometimes people talk about how, you know, the discipline of essentially, or like the tooling and the things that are available for data engineers lag somewhere, you know, five years maybe behind, you know, more traditional application development.

2:35Where would you say that something like ML engineering from an applied sense of being able to actually run these things in a company that may be user facing lags behind, you know, what we think about from traditional application engineering? It's definitely behind. I don't know what the number is in terms of number of years. I think like to me, it all comes down to developer productivity and how much time is like wasted on like tooling stuff versus actually delivering like business value. And the only way I found that's like, I think it's like somewhat correlated with developer productivity is like fast feedback loops.

3:07Like how fast are your feedback loops, right? And it's not like necessarily a perfect metric, but I think it sort of speaks a lot about like developer productivity. Like when you ask developers if they feel productive, a lot of it, I think it comes down to like having fast feedback loops. And when I look at other types of software engineering disciplines, like if you look at like front engineers, like I don't know if you do a lot of front of it. I kind of enjoy it because like you have like editor in one window and you have to browse in the other windows. And like today when you're writing front, it's like so crazy fast.

3:33Like you save the code and it just like automatically just like reloads the other screen and you see it right away. So you have this like, you know, sub second level feedback loops. And I think there's something similar about backend. Maybe it's a little bit different. You run unit tests or whatever. But with data AI machine learning, all of that stuff, I would almost argue we're taking a step backwards in that. As much as I love the cloud and I'm a massive proponent of cloud and it gives us tremendous power to do stuff, for whatever reason, using the cloud and suddenly you have to build Docker containers, you have to trigger things, you have to mess around.

4:07infrastructure ends up like just being this like massive friction in that loop when you're working with AI and machine learning on the engineering side. And that was like my sort of, you know, frustration. A lot of like what model came from was just, I just wanted to be fast. And I've like, maybe I have like lack of patience, but like, I want to write code and then like run it in the cloud in less than a second. So that was like my kind of starting point is like, how do I solve that problem? Yeah. To your point about front end, there is something, I don't know, like there's just immediate gratification.

4:36Like I can go in into string and I see it, you know, and I think that's probably why, you know, you see a lot of people who are starting to get into engineering probably start with front end because you have sort of that immediate feedback loop. And also even from languages, you know, adoption of languages, there's sort of a more immediacy of getting to that like aha moment and those feedback loops, perhaps using something like Python versus using like a lower level language where you might struggle a bit more to just kind and get started. And even older languages, you know, now even things like Java and C Sharp have tried to reduce sort of that barrier to entry of doing that simple like hello world, essentially equivalent application because of how easy it is in some of these other languages that have like severely simplified it and then led to a high adoption curve.

5:21For sure. And kind of as a side topic, like one of my beefs was Rust. I love Rust. It's a great language. I think one of the things that got wrong is like, it's just, you know, like they made a language that makes developers very productive in certain ways. but then you have the compiler getting in the way. It's like, if the compiler was just fast, I would actually really, really enjoy writing Rust. Now it's like, I enjoy it, but it's like, to your point about the older language, I used to write a lot of C++. And having that compilations thing take minutes every time you want to just run, it just takes the joy out of coding.

5:53So, I mean, can you break down a little bit more about, what are you essentially doing with modal? You want these super fast, essentially deployment times for ML workloads, but what's that end up looking like? Yeah, totally. Yeah, and maybe we should take a step back. We talked a little bit about like making developers productive and happy, but like what is modal, right? Like I tend to think, you know, a lot of like running data, AI, machine learning stuff today is about building code and shipping it into the cloud and scaling it up, running it on GPUs, mapping over large data sets, et cetera, right?

6:23So that's like originally what I set out to build. And as a big part of that, like, as I mentioned, I wanted the feedback loops to be very fast. we spent a couple of years focusing on that platform as kind of a core concept. Like we felt there's such a huge opportunity. This was also during the Zerp era. So we had like no pressure to make money. But two years in, you know, where we started seeing a lot of traction was with Gen AI applications. And so a lot of our use cases today that we see that, you know, as driving a lot of the growth of modal is various types of GPU inference use cases in Gen AI.

6:55So large scale, particularly like video, audio, music, like people who deploy models, often proprietary models, that need to scale up to sometimes thousands of GPUs and they don't want to handle the infrastructure or they're running like very large spiky batch jobs. So Modal is both like sort of an infrastructure provider, like in the sense that we have like a big pool of compute in the cloud, like a lot of GPUs, a lot of CPU. And then kind of the separate side of Modal is we have an easy to use Python SDK that lets you iterate very quickly. And in a way where you don't have to install anything, You don't have to configure anything.

7:29You just have to write a little bit of code. And then when you run that code, it runs in modal. And that makes it very easy to take a particular machine learning code and scale it up. And in terms of like this big cluster of GPUs, essentially your customers are sharing. Is that like a shared model? It's a multi-tenant model, right? Which is, you know, different from, I think, traditional applications. I tend to think it's the future of the cloud. There's so many benefits of us having a shared multi-tenant pool. Just to give you one example, if someone needs 100 GPUs, we can often get you that in a few seconds.

8:03Because we can pull all this variable demand, means we can do capacity planning at a very large scale. And so there's been many benefits of having this shared tenancy model. Not to mention also the fact that there's zero installation. Well, you have to do a pip install modal. But then once you do that, you can immediately start running code because we manage the infrastructure in the cloud. for people who are, you know, managing their own GPUs, do people end up typically like under utilizing the GPUs they have available? Totally. Which is another thing that modal solves beautifully is that because of the multi-tenancy, we have a model where everything is pay as you go.

8:40So it's all usage based. So we only charge for the time the GPUs are actually running, right? So there's no capacity planning with modal. You just start running modal and then we charge it per, you know, essentially GPU second or CPU second. Actually, I should say, There's many use cases where people don't use GPUs with Modala. And again, kind of going back to the multi-tenancy model, part of the reason why we can offer this is we can pool a lot of people's very bursty workloads and run an underlying shared compute pool. The other thing we also had to spend a lot of time on is doing things like very fast container cold starts.

9:09Because in order to have a usage-based pricing model, you need to be able to start containers and stop containers very quickly. Interestingly, that is a problem we have to solve in a separate context, which is like we talked about this previously, right? I wanted the ability for users to write code and immediately run it on the cloud. And that's actually the same problem, like fast container cold start. So that is another thing we've had to spend a lot of time on. We built our own file system. We built our own container runtime. We built our own scheduler. And all these things like let's just have, A, very fast feedback loops, B, a fully usage-based pricing, and C, I guess, fully managed infrastructure running in the cloud so people don't have to think about it.

9:48So can you break that down? like, you know, what's happening essentially behind the scenes. So I write, I've done pip install modal. I write some Python code, presumably like I tag it in some fashion. So I know, you know, what I want to run within the cloud. I run it, but what's essentially the magic that's happening between my machine and being able to run this on modal. Yeah. So modal is an SDK, which means like you import modal and basically like the way to think about, I think the like easiest mental model to think about modal is function as a service. It's similar to like AWS Lambda, if you're familiar.

10:17Like the idea is that you can take any function in Python and turn that into a function that runs in the cloud on invocation. So you can have, you know, all these functions in Python. And you can say this one should run on H100. This one should run on whatever, T4. Different container environments, like different, you know, even drivers, right? And you have to specify that in the code using decorators that you apply to these functions. And then you can have these functions call each other. And so you have this like function as a service programming model. And under the hood, the way that works is like we take the code, we're able to build a container image and launch that container image in the cloud in about a second, right?

10:54And more for like very large images. Like if you have an image that's like 100 gigabytes, it might take a few seconds. But we spend a lot of time on like, how do we take the code on the local computer, stick it in a container in the cloud with arbiter dependencies and launch that on a worker in the cloud that might have, you know, GPUs or many workers, maybe you need 100 GPUs that we need to spin up that container 100 times over on different workers in the cloud. Does each function that I'm specifying, let's say I have function one, function two, but I want to run them on essentially different GPUs, do those end up being separate containers that have to get deployed?

11:28Yeah. So every function ends up being a separate container. In fact, every function is auto-scaling. So if you start issuing many requests to the same function, we will just auto-scale it up to as much as is needed in order to serve all those requests. In some ways, it sounds like as a developer of these, you're sort of, it's almost like you're coding a monolith, but then the deployment is able to automatically sort of scale this out as more of like a market service architecture. Yeah, sort of, yeah. And, you know, I think that's one way to look at it. Like every function ends up being its own kind of container.

11:59I mean, you can have like a lot of code running inside a single container, but you can have like very, you know, isolated pieces of code. You know, you can have one container running, you know, Python 3.9 and another container running, Python 3.12, calling each other, just like a normal Python function call. You have this convenience of just, it just feels like local code, just functions calling functions, including handling tracebacks and exceptions and all these things. It just feels like Python, even though it's running in a distributed way in the cloud. Right. How does that function calling happen behind the scenes?

12:31Essentially, are you using some sort of GRPC call or what is essentially allowing you to call functions across these containers? Yeah. So under the hood, we use gRPC just like for internal communication and also the client library talks to the server using gRPC. There's like a few different layers, but we also like within Python, we use Cloud Pickle, just like serialize all the payload and send it between containers. But again, like that's not something that developer necessarily has to think about because it just feels like you're just calling a function in Python. So we handle the serialization, all the exception management, all of that stuff.

13:03But yeah, there's a lot of work, obviously, under the hood. Basically, it's all complex queuing theory and scheduling and doing that fast at a large scale. I mean, I think we're serving something like 100 ,000 requests a second right now across the scale of modal. So managing that, the state management and the scheduling and constantly scaling up and down, obviously, there's a lot of work to build all that infrastructure. Yeah. How's the state management and persistence in these distributed applications work? I would say modal right now focuses on compute. So there is some level of state management, You can create simple distributed.

13:37We have a primitive to create a distributed key value store and also Q. Most of what people use modal for today is things like Gen.AI. Gen.AI is actually kind of like a weird use case in many ways because it's extremely compute intensive, but it's actually kind of low IO. So if you think about stable diffusion, for instance, you send this tiny piece of text to a GPU, and then that GPU does like a trillion operations, and then it sends your little JPEG back that's like 100K or whatever. and that's like the type of applications that we do really well it's like very compute heavy stuff but not necessarily like super io intensive we're not necessarily like trying to compete with like spark in that sense which has like these operations you can wrangle like very large data sets and stuff like that so there is some state management in modal we also have a distributed file system you can set up in modal and you can attach to all the containers basically just like a local postix compliant file system which means like you can basically interact with it just like using normal file system operations.

14:33But anything you read or write is like globally distributed across all the other containers. So that's like another way you can also work with large data sets in modal. But we don't necessarily handle the sharding and partitioning and shuffling. That's not something we've built yet. When I think about something like an iPaaS for traditional applications, a lot of times people run into these challenges eventually where essentially the abstractions start to get to a place where they don't work. They want to get in there and sort of adjust things, tune it to their specification, we'll say we use a certain scale, and it relieves to this graduation problem.

15:05Is that a challenge that people building on Moldo in this world face? I think so. I mean, I think that's something I've spent a lot of time thinking about. It's like, the right abstraction sacrifices a little bit of power, but makes the remaining stuff so much easier to do. So I think any abstraction layer you add, of course, you're going to sacrifice a tiny bit of like you know capabilities but you know by doing that it turns out you can actually the remaining stuff like make it like so much easier to build i think the bad abstractions they basically like sacrifice 80 of the capabilities you know and then like turns out they're like very limiting so like how do you make modal like fairly general purpose so that we can do almost anything in modal i think that's very hard and that being said i mean i think like so far this is something I'm like obsessed with, by the way, thinking about, you know, the right abstractions and thinking about the developer experience and making sure, you know, modal is fairly general purpose.

16:02That modal is not necessarily like super frameworky. I think that's sort of like, I think I don't like about a lot of modal modern framework where like either they're like very config based or they kind of lock in a certain way. Modal is opinionated, but like I want modal to be like a bunch of Lego blocks and then you can like take those Lego blocks and build whatever you want. And I think to a large extent, it lets people do that. We've always thought about programmability. We always thought about making it possible to run any code you want. We're thinking a lot about doing non-Python stuff, for instance, as one example.

16:35So I don't know. I mean, I'm obviously biased, but I tend to think that modal is a platform that doesn't sacrifice that much capabilities. You can do almost anything in modal that you could do just using lower level cloud primitives with like 5 % of the effort. Yeah. This episode is sponsored by MailTrap, an email platform developers love. Go for fast email delivery, high inboxing rates, and live 24-7 expert support. Get 20 % off for all plans with our promo code SEDaily. What are the typical like Gen.ai applications workloads that people are running on this? Is this primarily people running their own model and they want to be able to run inference across these GPUs or are there other types of workloads that are also there?

17:23Yeah, so one user that I love, one of our customers is a company called Suno. They do AI-generated music. So they run a lot of their inference on modal, very large scale. So basically, what they run on modal is GPU-based. They have their own proprietary model and it generates music. Super cool application. I think a lot of our customers fall into that pocket. It's like there is a proprietary model. It does some very cool magic stuff on a GPU, typically in audio, video, image stuff, music. It's been kind of the domain we've seen most traction. There's other use cases too. We've recently seen a lot of traction coming from computational biotech, which I think is super exciting.

18:03So like protein folding, multiple sequence alignment, like there's all kinds of medical imaging, processing, like very large data sets of, you know, using applying computer vision to like assays. I don't know too much about the field, but like I find it incredibly exciting. And kind of in that vein, like we've seen people use model for like geospatial analysis, physics simulations, like turbulence stuff. And then there's a lot of people who just like the developer productivity, and they don't necessarily run big things, but they have a little web scraper running in modal or a little web server.

18:36There's a lot of that stuff. So modal is, the goal was always to build a fairly general purpose platform. We found initial core product market fit and the Gen AI. That's been the main use case for us. But there's just so many other things that people always surprise me. I was just talking to someone the other day who was running a chess engine on modal. I don't know why, but it was really cool. I mean, some of those examples you gave are pretty high compute examples, which I think make a ton of sense. Are there certain workloads that don't make sense? I guess you mentioned things that maybe require a high I.O.

19:06are probably maybe not the right fit. Yeah, high I.O. I think we're like pushing into that. I think it's going to be a big focus for 2025 to also handle. Similarly, I would say like very low latency things is something we're like very excited about. Right now, there's like an overhead in the system of like 100, 200 milliseconds of every function call. a little bit less if you run it in the same region. But that's enough for a lot of use cases, especially in Gen AI, like stable diffusion. No one really cares if it takes 200 milliseconds because the inference in itself takes a couple of seconds.

19:34So that's pretty negligible overhead. But it's not enough for something like real-time streaming of audio or real-time video. So that's another thing we definitely want to push into over the next year is how do we get the overhead of the system down to 10 milliseconds, 5 milliseconds, whatever. In terms of getting to a place where you're able to run these for things like real-time, you know, video, audio streaming? Like, what do you see as the main sort of technical hurdles that you have to solve in order to reach that kind of performance? Today, it's mostly about geolocation or basically like we run a distance.

20:05Exactly. Like the speed of light is pretty high, you know, and as you may know, like, you know, it doesn't take that many 100 milliseconds to like send something to Australia and back to the US, like, you know, 200 milliseconds or whatever. But, you know, it is a challenge when you're, you know, doing real-time stuff. So a big challenge for us is like, while we have a distributed data plane, like all our workers run in many different regions, many different cloud providers, our control plane right now is not distributed. So one of the things we want to do in 2025 is decentralizing the control plane so that we can use, you know, smarter ways, basically like kind of route to, you know, multiple edge.

20:39I feel like the word edge is overused, but you're running a model in many different regions across the world, like also the control plane so that we can route things and execute it faster. That's a big re-architecture. So in this multi-tenant architecture that you have today, you have your control plane, you have your data planes, your data planes are distributed across these regions. Is primarily the sort of reuse of resources happening in the control plane? Yeah, the control plane makes all the decisions, right? And it sort of has the global state of the world, which is another thing to some extent.

21:10When you have a very large worker fleet and many different scheduler running in different regions, I think you also kind of need to change the truth of the system to be owned by the workers themselves, because they always know the latest. Because, again, speed of light is not always as fast as we wish. So there's a lot of that state management. Where do you make the decisions? Who has the authoritative view of which worker is running which containers? And how do you propagate that information? I mean, these are hard technical challenges, but luckily we like those at Modal. Was the plan from the very beginning always to build this out as this like multi-tenant structure?

21:46It is challenging sometimes because I think people are just not entirely comfortable with that model yet. But like, I don't know, I've been coding for 30 years. And like, I remember when the cloud came in 2007, it was something like that. Like, you know, AWS launched or EC2, I think something like that, 2006 maybe. And my first thought was like, that's insane. Why would I put my code on someone else's computer? And then like, you know, a few years later I was doing it. I was like, this is kind of nice. I like this. Yeah. So like, you know, it took a few years. I mean, it's still taking a few years.

22:16Like many people still run on Prime, right? But like, I think there's been like, obviously, you know, people are seeing that the cloud makes sense for a lot of stuff, even though it's like arguably like a shared resource. If you look at, you know, a company like Snowflake, they were like, hey, we're going to build a cloud native database. And at that time, which maybe doesn't seem that crazy now, because basically everybody's doing that. But at that time, it was like, what do you mean you're going to take this thing that we run in, I don't know, on-prem Oracle today and move it to the cloud? Like, we're not going to do that.