#186 Ronen Dar: Maximizing GPU Utilization for AI with Run:ai

0:00Enterprises are securing access to GPUs and they are buying and reserving those GPUs. So there is this shift in how cloud resources are being consumed from on-demand fashion, just reserving blocks of GPUs. Research institutes are also doing that. Gen AI startups are also doing that. OpenAI, the most successful AI company right now, raised$10 billion two years ago. $10 billion mostly goes to GPUs. Passwords are the bane of my existence, and I'm sure they're the bane of yours. How do you make a password that's strong enough so no one will guess it and impossible to forget? Think of a password. Now add a number and a special character.

0:45No, not that one, a different one. And make it eight, no, 12 characters. Great. Now remember it forever and do it a hundred more times. Don't reuse it and make it so everybody in your company can do the same without ever needing to reset them. Sounds impossible. Unless you have one password. That's the number one, P-A-S-S-W-O-R-D, 1Password. 1Password combines industry-leading security with award-winning design to bring private, secure, and user-friendly password management to everyone. 1Password makes strong security easy for your people and gives you the visibility you need to take action when you need to.

1:31Companies lose hours every day just from employees forgetting and resetting passwords. A single data breach can cost millions of dollars. 1Password secures every sign-in to save you time and money. 1Password lets you securely switch between iPhone, Android, Mac, and PC with convenient features like autofill for quick sign-ins. All you have to remember is one strong account password that protects everything else. your logins, your credit cards, secure notes, or the office Wi-Fi password. Protect your global workforce with simple security, easy secret sharing, and actionable insight reports. I use 1Password myself, and I encourage my family members to use it also.

2:19I don't know how I would get along without it. 1Password's award-winning password manager is trusted by millions of users and over 100 ,000 businesses from IBM to Slack. It beat out 40 other options to become Wirecutter's top pick for password managers. Plus, regular third-party audits and the industry's largest bug bounty keep 1Password at the forefront of security. Right now, my listeners get a two-week free trial at 1Password.com slash IonAI. That's E-Y-E-O-N-A-I all run together. That's two free weeks at one, number one, onepassword.com slash IonAI. Don't let security slow your business down.

3:07Go to onepassword.com slash IonAI to get a two-week free trial. Hi, I'm Craig Smith, and this is IonAI. In this episode, we dive back into the world of AI infrastructure with Ronan Dar, CTO and co-founder of RunAI. Ronan talks about the challenges of efficient GPU utilization and inference in the midst of the ongoing GPU shortage. He explains how RunAI's innovative software platform optimizes GPU clusters, empowers machine learning teams, and shapes the future of AI deployment and scalability. I hope you find the conversation as useful as I did. Happy to be here, first of all, so thank you for inviting me.

3:57I'm Ronan, I'm the CTO and co-founder of RunAI. and RunAI is an AI infrastructure company. We started six years ago and we're an Israeli-born company. And so all of our engineering and product teams are here in Israel. But all the go-to-market teams, most of them are distributed globally, right, remote companies. So as time goes by, the company becomes half Israeli-based and have remote and distributed globally. So that's fun to see how the company evolves. Before running, I've been mostly in academia. So many years in academia, I did my bachelor's degree, my master's, my PhD in Tel Aviv University in information theory.

4:49So I'm an engineer, electrical engineer. So a lot of years in academia. In Israel, I did my postdoc afterwards. In the States, I've been there for a few years. I really enjoy the academia and doing research. That's fun. And in parallel, I've been also in the industry. So I worked for chip companies, and there are quite a few of them, quite a lot in Israel. So I worked for Intel and other chip companies here in Israel. So I've been an algorithm engineer in those companies. And so I gained a lot of experience around how chips are being designed, how chips are being manufactured and all of that, those things.

5:29And Run.AI is a software company. So, right, of course, experience also in software. So I guess academia and hardware companies, software companies, that's my background. And now in Run.AI. And so tell us about Run.AI. I mean, in our email exchange or my exchange with Ali, we were talking about the GPU shortage and ways to address that. But what is Run.ai at its core? Okay, so Run.ai is an AI infrastructure company, as I said. When Omri and I started, it was six years ago. So Omri is my co-founder, right? He's the CEO. And we met in the academia, both of us. We had the same supervisor. We worked quite a lot in the academia.

6:22It was really fun back then. Still very fun to work with Omri. He's an amazing guy. And we started back then, and both of us knew that we want to start a company. We want to be in the entrepreneur in some days. And the first decision that we got is that we want to be a team. We want to start a company together. The second decision was around AI. We want to be in the AI space. back in 2017, 2018, we saw that there is this new technology, AI. Back then, it wasn't that clear that AI is going to revolutionize the world. A lot of people spoke about it, but people also spoke about AI winter and whether AI will succeed and bring value to the world or not.

7:10And I think we saw that there is a big revolution coming in, and we wanted to be part of that. We wanted to contribute our part. and we wanted to lead our space right there when it comes to infrastructure so both of us as i said we were very close to hardware we know a lot of on hardware and we saw that compute is going to be a critical component in a app and a lot of gpus are needed to train AI models. And we saw the trend that started around 2012, 2013 of more and more compute, more and more GPUs being used to train state-of-the-art models and new innovation coming from the fact that having a huge model is being trained on huge amounts of data using huge amounts of compute.

8:07So compute is there together with data and the algorithms and compute is critical. And we saw that GPUs are relatively new in the data center. So GPUs were known in the gaming industry. They were very successful in that industry, but in the mainstream data center, they were quite new. And the entire software stack that runs clouds and data centers was built for CPUs and traditional applications, traditional microservices, and not for AI workloads, which are really compute intensive. They consume a lot of compute. They may run on distributed servers, distributed computing, and they run on GPUs, which is a new beast in the data center.

8:57GPUs, if you compare it to CPUs, they are much more powerful. They are much more expensive. They are much scarce also. So there are a lot of challenges around how to schedule workloads on GPUs, how to manage them, how to utilize them. And we saw that the software stack is not the right fit when it comes to GPUs and AI. And so we took a chance of rebuilding that stack. And we're still using cloud-native technologies like Kubernetes and containers. and we brought a lot of technology when it comes to just scheduling and orchestrating and fractionalizing GPUs. So we built this engine that runs within GPUs, within GPU clusters, that provides much more advanced scheduling capabilities and ability to fractionalize GPUs and allow more workloads to share the same GPU.

9:54and essentially we brought that technology that allows machine learning teams to get much more out of their GPUs and to get much more utilization out of their GPUs, much more availability after their GPUs. And we're seeing from our customers like 2x, 3x more GPU utilization, more GPU availability after they start to work with our software and they start to use it correctly. And you're getting huge, huge benefits on that aspect. GPU utilization and GPU availability. And we've built this platform on top of that engine, which simplifies for machine learning teams the way they train models and deploy models in production.

10:43So we built tools, and we've integrated with tools in the ecosystem, with open source tools out there. And we built these abstraction layers that allows machine learning teams to much easier way, much simpler way to train their models and take their models to production, deploy them and do inference. And so we've built all of that. We're working with a lot of great companies, great machine learning teams in enterprises and in startups. And we power the most advanced research labs today in the AI space. and it was fun to see that growth of R &EI and we're always thinking about the next step and continue growing it always becomes harder and harder to continue to grow but it was amazing six years and we're looking in 2024 I think it's going to be an amazing year because there are so many GPUs out there we started to speak about the GPU shortage So a lot of GPUs are going to be required in the market right now.

11:54And we're expecting some amazing growth this year. Yeah. And so I've been talking, as I said, to people about this issue from various angles. I just had Cerebris Systems on Andy Hawk, and I've had Andrew Feldman on one of the founders previously. And then I've talked to people about different strategies for chunking prompts and different ways to get through the rate limits that large model, foundation model companies put on because of the GPU shortage. you guys are focused on increasing utilization of each GPU. That's kind of the core value proposition, isn't it? Exactly. So the core value is around more performance from GPU clusters.

13:05Clusters, yeah. Think of AI, OpenAI, they have a lot of GPUs clustered in some way. And researchers in OpenAI use those clusters, those GPUs to train their motors. and OpenAI also running chat GPT on those GPU clusters. So the way all of those workloads are running on GPU clusters and sharing those GPUs and being scheduled on these GPUs and how they utilize those GPUs is what our software brings to the table, a much better way to use and utilize those GPU scheduler. So use clusters, yeah. And is your customer base cloud providers? I mean, does this sit between the cloud and the customer, and it's employed by the cloud providers to increase utilization of their GPUs, or is it employed by the customers on their end in managing their clusters in the cloud?

14:15Okay, amazing, amazing question. So we have, well, our software is being divided into two. So one software, the entire engine runs on GPU clusters, on GPU infrastructure that is owned by the customer. A customer can run their GPUs on premises or in the cloud, on Amazon or on Google or on Azure. We support all of them. We have customers all over. And then there is a control plane of us, which runs either as a SaaS component, as a software, as a service on our machine, and connects to the different GPU clusters that are out there and controls them and runs workloads on those GPU clusters. Or that control plane can run on the customer environment as well, as a self-hosted way or as an air-gapped environment.

15:14So we have quite a few defense companies, which are defense organizations who are our customers. And so we support that as well, the most secure, no internet environment as well. Yeah. One of the things that struck me in talking to Cerebris, their thesis is that you etch a silicon wafer and then you cut it up in the little chips. and then you when it comes time to to train a foundation model or some heavy compute exercise you stitch all these little chips together again so that they can work in parallel and their vision is you know why do you cut them up in the first place if you're going to have to run them and clusters uh and my understanding that that the the the the trick to a working uh with gpus and clusters is is the cabling the hardware cabling as much as it is the software can can you talk about that that's something i don't quite understand how can you address this purely from the software end?

16:35I mean, isn't there also a sort of proximity in cabling and a hardware side to it all, a physical aspect? Right. So on the infrastructure level, you have machines, racks, you might say, with GPU servers within them. and that can be GPUs and can be any other accelerator. There are other accelerators that are not NVIDIA GPUs. And those accelerators are essentially connected with communication cables and with usually very fast communication cables. and the idea behind it is that when you're training models and usually the models are so big when it comes to the state of the art models the models are so big that they run on a lot of gpus a lot of accelerators that are not just physically within one server or within one rack right so So GPT-4, for example, according to us, was trained on more than 20 ,000 GPUs.

18:01So usually in one machine, one GPU machine, you have eight GPUs connected very fast interconnects. And then you connect multiple servers, multiple machines like that. And also with very high interconnects, very high speed interconnects. and all of those gpus need to process together in a synchronized way a lot of data and essentially train a model on that data now models became so big in the last decade that you need to train GPT-4 more than 25 ,000 GPUs. 10 years ago, when the big breakthrough in deep learning happened around 2012, 2013, AlexNet was the model that triggered that big, high, big revolution in deep learning.

19:05AlexNet was trained using two GPUs. That's it. So GPT-4 was trained on, according to Moos, on more than 20 ,000 GPUs. So much more GPUs in 10 years. Now, GPUs also became much better. And also researchers started to train those models with much more data for longer time. So research has showed that the increase in compute power required to train state-of-the-art models grew 100 million times in 10 years. 100 million times. That's mind-blowing. And so we saw that trend, and let's see what the next decade will bring, right? If we'll see more models, bigger models that consume more data, that are trained on more compute power, more GPUs.

20:04And I believe that that trend will continue. I heard, I was listening to a panel on which a deep mind person was speaking, and she was saying that a sort of master's level computer science researcher can train, can on their own work with GPUs to train a model up to about 70 billion parameters. And she was saying that after anything above 70 billion, it becomes a really challenging engineering problem to get clusters to work together efficiently and that you end up with very low utilization. So you have to employ a lot of tricks as far as she went to get the utilization up and to get all of the GPUs working together to train above 70 billion parameters.

21:20And I assume that's where run AI comes in. I mean, maybe at a university, they do it in their lab, they do it on their own. But is that right? Is that what Run.ai essentially is doing? You're like scheduling an orchestration software that optimizes the parallelization between GPUs. And can you talk about how you do that? I mean, presumably there's some AI involved in doing that. How is that done? Yeah, so we're doing also that.

22:06As you say, to fine tune a model with 70 billion parameters, taking an open source model like LAMA2 and fine tune it with available data, You might need a few machines, GPU machines to do that, like four GPU machines. So one engineer or one researcher can handle that. But scaling up beyond that and training maybe bigger models or training from scratch using much more data, then that needs much more careful engineering and fine-tuning and configurations and making sure that GPU servers don't fail and things like that. So a lot of optimizations at that level. We do also that. We have customers who are training models on hundreds of GPUs, if not thousands of GPUs.

23:06So we have that. The thing that is unique to run AI is that we give a software platform that allows a team or a few teams to do the entire scaling regime in terms of training models. So you might have a researcher just opening a notebook, Jupyter notebook on a fraction of a GPU. You might have a few of them doing that. You might have some researchers in parallel training, fine-tuning LAMA 2 with 70 billion parameters on a few machines. And you might have more advanced users training on hundreds, if not thousands, of GPUs just running one job. And you might have also inference workloads running and serving actual requests coming from actual users.

24:00So all of those workloads are running on the same clusters, on the same platform, right? And all of them need to be scheduled and orchestrated and managed and utilize the GPUs efficiently. So that's where we shine OneAI. We allow all of that regime to happen on just one cluster or maybe multiple clusters and allow all of them to get access to those GPUs and utilize those GPUs in the most efficient way and in the most simple way. I'm curious how you do that. I mean, what are the algorithms involved and is there dynamic switching between GPUs within the cluster? And also, does RunAI, does a user access the cluster through RunAI?

24:57And so RunAI is the interface for the user, or is it kind of a bolt-on where the user is still working directly in the cloud with the cluster, but there's some side optimization going on? Okay, great question. So we're bringing a few algorithms, you might say, into the engine that runs those clusters. We're bringing scheduling algorithms. That's one thing. So a lot of scheduling decisions on how to dynamically allocate the GPUs to the different workloads and making sure that all the GPUs are being utilized and no GPU is sitting idle and making sure that all of that switching, that dynamic switching between workloads is happening in a fair way between all the users according to business goals, according to priorities of the organization.

26:05And making sure that always researchers can submit a lot of work, just fire up a lot of workloads to our system, to the cluster, and all of those workloads will be queued and be allocated and run on GPUs in a systematic, automatic way. So I think as a researcher, they can just launch so many jobs. Those jobs can be of any scale and everything will run in the most optimized way. The second thing that we're bringing is that software layer that sits on top of GPUs. And we do what we call an API level virtualization. We sit at the CUDA level and we intercept CUDA calls and we manage the access to the GPUs.

26:53So with that layer, we can allow multiple workloads to get access to the same GPU without colliding and without memories being used by the two workloads in a colliding way. So we prevent those workloads to collide. we bring this isolation level and each workload sees like a virtualized smaller gpu and so that is very significant when it comes to inference not all of the models when they run in production are very big and they require you know a whole gpu or a lot of gpus any times you have small models that needs just a fraction of the gpu so we bring that a CUDA level interception layer that controls the access to the GPUs and bring more efficiency in that sense.

27:45And when it comes to launching jobs, so we got a decision early on to build a product which is open and have a platform where users can launch workloads and deploy models using run AI tools, but also use any other tool that can integrate with a cloud-native platform. Our platform runs on top of Kubernetes. So any open source or third-party tool that runs on Kubernetes can integrate with our AI. And so we have a lot of customers who are actually not running with our tools but utilizing third-party tools or just launching jobs with a command line interface and things like that. So that was a decision that we got very early to integrate with the ecosystem because the AI ecosystem is moving so fast.

28:46Tools are being built every day and the advancement is so fast and we see different machine learning teams using a variety of tools. So supporting that and supporting the advancement as time goes by was crucial as we saw it. I think now with LLMs, we see it back again, right? It was all about deep learning a few years ago. And then JITGPT came and everything now is about LLMs and so many new tools are being built right now. And so many new tools are being used by machine learning teams. So staying open, that's a decision I'm glad we took early on. And then on the inference side, you know, the foundation model providers have currently have rate limits.

29:44I mean, they keep changing, but you can only put so many requests through per minute or so many. you can only call on the APIs so many times a day or various limits like that. Does this help a user of a foundation model increase their rate limits so that there is more throughput? or does that require the foundation model provider to be running AI or some optimization software? I mean, if I'm a big company, I want to put a product that calls GPT-4 into production, but I can't get either. there's a latency issue because I can't get my inferences through and back quickly enough. Does RunAI help solve that problem?

30:52As you said, latency and throughput are big challenges when it comes to inference, and it impacts the cost of AI, and the cost of AI right now is very high. and we see companies like OpenAI deploying rate limits and trying to make sure that the compute and the models are being used in the most efficient way. So I think there is a lot of work going on in the ecosystem, in the industry right now on reducing the cost of those LLMs and in different aspects, right? You have an amazing advancement happening at the serving layer, just on how LLMs are actually running on GPUs and how the architecture of the model is being run and using the compute power of the GPU, using the memory of the GPU.

31:52So a lot of optimization at that level are happening and amazing new things are coming out. there is also a lot of optimizations that can be done at the cluster orchestration level, at the compute management layer, as we do. We're not powering OpenAI data center. If we would, then they would get a lot more efficiency coming from all the better scheduling, better management. I'm sure they're working on that. I'm sure they're working hard on their orchestration layer. And that will help to reduce the cost. So when it comes to running AI, so we help companies who are deploying their own models or third-party models on their infrastructure in their cloud environments, in their virtual cloud environment.

32:50And we help them to host those LLMs in the most efficient way, right? Yeah.

32:59There seems, I mean, you were talking about building an abstraction layer or abstracting this problem into an orchestration layer, however you want to say it. I spoke with a company called Lightning AI yesterday or the day before, And they're building something called Lightning Studio that is an interface that pulls together all these different tools and services into one environment. And so you can very easily sort of click through different. Are you integrated into Lightning Studios, or are you another sort of a competing layer? So, okay, so Lightning AI, a great company. We're integrated with PyTorch Lightning.

33:58So the builders of Lightning AI are the creators of PyTorch Lightning, which is a great tool to launch PyTorch jobs. So we're integrated with that. So we have a lot of, of course, PyTorch users, and all of them are using different tools from PyTorch Lightning to maybe using Ray and running PyTorch on top of Ray. So there are several frameworks out there, open source frameworks, which we integrate with all of them. I mean, I get pitched a lot as a journalist, and I see these all-in-one platforms coming through. I don't have time to look at each one, but I guess what I'm asking is Run.ai is one of those tools that will be integrated into whichever of these higher level abstractions sort of wins the market?

34:58Or is Run.ai itself going to become one of those higher level abstractions that runs various tools? Passwords are the bane of my existence, and I'm sure they're the bane of yours. How do you make a password that's strong enough so no one will guess it and impossible to forget? Think of a password. Now add a number and a special character. No, not that one, a different one. And make it eight, no, 12 characters. Great. Now remember it forever and do it a hundred more times. Don't reuse it and make it so everybody in your company can do the same without ever needing to reset them. Sounds impossible.

35:40Unless you have 1Password. That's the number one, P-A-S-S-W-O-R-D, 1Password. 1Password combines industry-leading security with award-winning design to bring private, secure, and user-friendly password management to everyone. 1Password makes strong security, easy for your people, and gives you the visibility you need to take action when you need to. Companies lose hours every day just from employees forgetting and resetting passwords. A single data breach can cost millions of dollars. 1Password secures every sign-in to save you time and money. 1Password lets you securely switch between iPhone, Android, Mac, and PC with convenient features like autofill for quick sign-ins.

36:32All you have to remember is one strong account password that protects everything else. Your logins, your credit cards, secure notes, or the office Wi-Fi password. Protect your global workforce with simple security, easy secret sharing, and actionable insight reports. I use 1Password myself, and I encourage my family members to use it also. I don't know how I would get along without it. 1Password's award-winning password manager is trusted by millions of users and over 100 ,000 businesses from IBM to Slack. It beat out 40 other options to become Wirecutter's top pick for password managers. Plus, regular third-party audits and the industry's largest bug bounty, keep 1Password at the forefront of security.