A developer's toolkit for SOTA AI

0:06Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at Fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

0:43Welcome to another edition of the Practical AI Podcast. My name is Chris Benson. I'm your co-host today. Normally, we would have Daniel Whitenack joining us, but Daniel has just gotten off a plane. He flew halfway around the world, and we decided to give him a break from today. I would be, he was more lucid than I would be under the same situation. Today, I wanted to dive right in. we have a super cool topic. It is not dissimilar from some of the other general things we've been talking about, but I have two guests today. I'd like to introduce Varun, who is the CEO and co-founder of Codium, and Anjul, who is the lead of their enterprise and partnership.

1:24Welcome to the show, guys. Thanks for having us. Thanks for having us, Chris. Hey, you're welcome. I'm really interested in learning more about Codium. When Daniel lined you guys up. He's like, Chris, you got it. He sent me this thing saying, you got to look at this. This is really cool and everything. And I'm like, get him on the show. He's like, I'm already doing that. So, so really glad to have you guys on. And he's going to be bumming that he missed the conversation because he was pretty excited about it. And so I guess I wanted to, before we even dive into Codium and the problems it's trying to solve and such, if you guys could each just tell me a little bit about how you found yourself arriving at this moment, kind of a little bit about your background, how you got into AI, and how this became the thing.

2:09Varun, if you want to kick off and then Anshul afterwards. So maybe I can get started. It actually starts in 2017. I started working at this company called Neuro that does autonomous goods delivery. So it's an AV company. There I sort of worked on large-scale offline deep learning workloads. So as you can imagine, an autonomous vehicle company needs to run large-scale simulation. They need to basically be able to test their ML models at scale before they can actually deploy them on a car. And sort of in 2021, I left Neuro and started ExaFunction, which is the company that is building out this product, Codium.

2:46And ExaFunction started out building GPU virtualization software. So you can imagine for these large scale deep learning applications, one big problem is GPUs are scarce, they're expensive, and also hard to program. And sort of what ExaFunction started building was solutions and software to make it so that applications that ran on GPUs were more effectively using the GPU hardware. And we realized that our software with ExaFunction was best applicable to generative AI tech and started building out Codium around a year ago. Very cool. And before I dive in, because I have several questions for you, but I want to give Anshul a chance to introduce himself here.

3:26Go ahead, Anshul. Surprisingly, my story is actually quite similar. I was also working at Neuro. So Varun and I used to work together back in the day. I was not actually working on, you know, the ML infrastructure side of things. That was something that, you know, Varun had hands-on on. But, you know, I decided to kind of also join the team at ExoFunction. And I think, yeah, as Varun mentioned about, you know, a year ago, I think we noticed there was like, I think three things kind of happened at the same time that we noticed that led us to Codium, right? I think the first one is that, you know, we're engineers.

3:57All of us here are engineers. and we had all tried the GitHub co-pilots and all these cool AI tools for code in their beta. And we're like, wow, this is absolutely going to be the future of software development. But at the same time, it's still scratching the surface of potentially everything that we do as engineers. So that's, I think, number one, I think that we realized. Then number two was talking to a lot of our friends at these bigger companies or anything like that. A lot of them were just saying like, oh yeah, it's cool. I've tried it for my personal project, but I can't use it at work, right?

4:29My work's not allowing me to use that. So it was like the second thing we heard. And the third thing was exactly what Varun alluded to. We were building like ML infrastructure at scale for really large workloads. Like when this entire generative AI wave started coming, we're like, wow, we're actually kind of sitting on the perfect infrastructure for this. So I think all of those three things kind of combined together for us to be like, do you know what? Let's build out an application ourselves and build an application that we as engineers are customers ourselves, right? And that ended up becoming Kodium.

4:59As you were getting into doing GPU software, what was in general some of the challenges that you were seeing? You know, with NVIDIA has their various software supporting things like that. Clearly, you saw that there was a need for something beyond that. Can you talk a little bit about just the layout that you saw in the environment before you got to all the generative stuff and the fact that you had infrastructure? What positioned you for that? And what was the thing that you decided that you needed to address? Maybe I can take it a step back of why these GPU workloads are just a little bit annoying compared to CPU workloads.

5:32One of the really sort of unique things about GPUs is that unlike CPUs, it's kind of tricky to virtualize. Like one common thing that we have with CPUs is you can put a bunch of containers on a single VM, and then you can kind of make use of the CPU compute like effectively. Right. You can basically dump 10 applications onto a CPU and it's perfectly fine. For GPU, it's a little bit more messy because the GPU doesn't have a ton of memory. So you can't just load up infinitely many models on there. Like let's imagine you have a GPU with 16 gigs of memory and each of these models takes like 10 gigs.

6:06You can't really even put two applications on there. So then that already becomes a big issue. And that's sort of what a lot of these large deep learning workloads were struggling with. So when I was at Neuro, one big problem we had was we had around like tens of models, but we had these workloads that needed hundreds of GPUs, some of them even thousands of GPUs. And we struggled to basically make it so that we were even able to use the hardware properly. And then, you know, you could imagine the complexity then stacks with now we're in a state where companies have trouble even getting access to 10 GPUs because of NVIDIA sort of scarcity issues.

6:42And then also the cost of a GPU is like not like a CPU. It's like significantly more expensive. Like the cost of a single H100 chip is well over 30 grand. So these aren't like very cheap chips. So there's like a big need at the time to figure out how do we leverage the hardware properly. And sort of that's what we had to build software for. And just to clarify for me, was that why you were still at Neuro? Or was that after you started ExaFunction? Yeah. So while I was at Neuro, we sort of worked through, or I sort of led a team that sort of built software that kind of fixed these problems. But ExaFunction was focused on generically, how do we make sure deep learning-based applications could best leverage GPUs?

7:21That's sort of what we started out building, actually. And then Codium came out from that, actually. Gotcha. Tell me a little bit about, as you have been right in the middle of this progression, just to frame it for a second, if you look at the last couple of years in particular, and the pace of change has been so much. And so you were right there starting at Neuro and then creating ExaFunction, seeing some of the challenges, could you talk a little bit about how the industry was evolving and changing as you were seeing it so that we can get a sense of kind of how you moved toward Codium, you know, to give a little bit of the history instead of just starting from where that is.

8:01Can you talk a little bit about, you know, the itches that you were scratching and why it led that direction? What did this AI industry look like to you? Yeah. So when we started, like, you can just imagine everything was a lot more smaller scale, right the hyperscalers or the cloud providers just didn't have nearly as much gpus like if you ask them like what fraction of cloud spend is gpu spend it's probably like very small single digit percentage points maybe even less than that at the time so this is like a very small workload for them when we sort of started both me and anshul started in a row in like 2018 but then over time this grew a ton like we could see it from the training workloads these were no longer like even single node training workloads like back in the day a single gpu node that had maybe like eight V100s or something was like considered a lot of compute.

8:46And suddenly now we were able to witness the fact that this was slowly becoming eight A100 nodes. And then more than eight of these nodes were necessary then even to train these models. And similarly, to prove out that these models were capable, like in an actual production setting, you needed to run offline testing at massive scales, like on the order of like 5 ,000 to 10 ,000 T4s scales, which is like kind of incredible in terms of raw flops. So we were able to see this hockey stick happen in front of us. And then that's sort of what made us want to start ExaFunction in the first place. We realized that there were going to be large deep learning workloads.

9:22One interesting fact is for us, like for just the ExaFunction GPU virtualization software that we ended up selling to enterprises, we ended up managing over 10 ,000 GPUs on GCP in a single GCP region. So we ended up managing more than 20%. And we realized that, that, hey, this was only going to keep growing. Like when we talked to the cloud providers, they were only going to keep growing the number of GPUs. And we realized, I guess the interesting thing was in the future, generative AI was going to be potentially the largest GPU workload though. That was the big thing we realized was GPT-3 came out, which was, I guess, in 2021 now.

9:57Gotcha. So, but you had already, at that point, were you already in Dexa function? Had it already started at that point? Yeah, it had already started and we were sort of selling GPU virtualization software to large autonomous vehicle and robotics companies. Gotcha. And so basically, if I'm understanding correctly, the whole generative tsunami just kind of landed on you when you were already sitting in that space doing GPU virtualization already. So you just managed to land right in front of the wave, it sounds like. Yeah. So we started working on Codium like maybe four or five months ago before ChadGPT.

10:31It was interesting just because we realized that an application like GitHub Copilot was going to be one of the largest GPU workloads, period. Like, I don't know if you've probably tried the product out. It's like every time you do a key press, you're going out to the cloud and doing trillions of computations, right? So it's like a massive workload. And we had like, as Anshul said, the perfect infrastructure to basically run this at enormous scale. Not to mention we were in love with the product from day one. Like we were early users of the product the moment it came out in 2021. Very cool. And so as generative is starting to take off kind of with chat GPT hitting the world and really changing things quite rapidly, you know, I think people are still shocked at how fast things have moved.

11:12You would start Codium already. What kind of synergy were you starting to see there in terms of knowing that you have one of presumably many, many GPTs coming and other similar generative models. You had just gotten into Codium. Can you talk a little bit about what that was and what were you putting together in your minds to recognize the opportunity that it was? Yeah. So I think one of the great things about the entire chat GPT wave is that everyone was using it. This is a thing where literally every individual is using AI. And so it helped us in general, right? You know, like a big wave raises all ships kind of thing.

11:51It really helped us, we weren't really going out and now telling people like, hey, a tool like Codium can help productivity because that was kind of just now assumed by everybody. Like, oh, yeah, if I do any kind of, you know, knowledge work, then there's potential for AI to help. Right. And I think so from that sense, when the star like, you know, chat GPT wave really came about, that overall kind of just like helped us in terms of convincing people to even try the product. The other thing that we we recognize is that we were positioning ourselves very specifically from the beginning, when it comes to code.

12:22Code is actually a very interesting modality. It's not like your standard chat GPT where you have a long context that a user puts in and then it produces context coming out. Code is interesting in the sense that, as we mentioned, it's an autocomplete that's like a passive AI rather than like an AI that you're actually instructing the model to do something. It's happening every keystroke. So it has to be a relatively smaller model. You have these hundreds of billions of parameter models being used. It has to be relatively low latency. And then code itself is interesting. If you have a cursor in the middle of a code block, the context both before and after your cursor really matters.

13:01It's not just what comes before. So there's all these interesting situational constraints about code that you put all these things together and we realize that, OK, all these chat GPT waves and conversational AIs are happening. That's great. but we're still not going to be like, you know, rolled over by that because we're kind of focusing on a very specific application and modality of LLMs that was pretty unique in many ways.

13:44Could you take a moment as we're diving into Codium Ingenerative AI and its unique capabilities there and just differentiate a little bit about for those, you know, so many people have tried Copilot. And so it's kind of inevitable that you're going to get that comparison to some degree. Can you talk a little bit about what Copilot's not doing for generative AI or how you're approaching it that allows you to show people this as a better way forward from your perspective? I mean, we have tons of respect for the Copilot team. I'm just going to start with that. As Bruin said, we were all early users of it.

14:20Definitely not putting you into conflict with them. That just is a starting point for people. Absolutely. Yeah, I think the way we kind of view this, and I kind of alluded to this earlier, is that writing brand new code with autocomplete is really just one small task that we do as engineers. We refactor code. We ask for help. We write documentation. We do PR reviews. And so kind of our general approach has always been, let's try to build an AI toolkit rather than an AI autocomplete tool. Got it. So we can get more into this, into the weeds here, but like autocomplete is just one of our functionalities that we provide, right?

14:55We provide like an in IDE chat. So I think like chat GPT, except integrated with the IDE, natural language search over your code base using like embeddings and vector stores in the background. So like we're really trying to expand, like how can we address like the entire software development lifecycle? So I think that's probably the most obvious difference with a tool like Copilot from like an individual developer point of view. But then the other thing which really kind of builds off of all the infrastructure that Vern was mentioning earlier is that we were already deploying, you know, ML infrastructure in our previous customers' private clouds.

15:27Like we already had all this expertise of how can we take actual ML infra, deploy it for a customer in a way that, you know, they can fully trust the solution because, you know, we're not getting any of their data. And so another really big differentiator for us was like, okay, I think this might actually be a tool that enterprises can use confidently and safely because we have the infrastructure to do the deployment in a manner that they would be open to using. So I think that was like the other differentiator when it came specifically to enterprises, but we can dive more into that later. No, that sounds good.

15:57I want you to connect one more thing for me. Going from being able to deploy the infrastructure and helping your customers in that way to Codium as a tool, what's the leap there that got you from one to the other? How did you get from infra-focused to Codium-focused? Oh, yeah. I think we had to do a full 180 when we started. We went from a full infra-service company to let's create a product for consumers. It was a full 180 in terms of product. A pivot. Yeah, full, in some degree, a pivot, because we knew that, you know, eventually, okay, we'll deploy to customers VPCs. That sounds great. But, like, if we're going to ship something to a customer, we had to be, like, super confident that it was a product that would work well, right?

16:37Because we're getting no feedback from their developers. And so we actually first focused for the first, like, six or so months of Codium just building out, like, an individual tier, right? Any developer can go try it. We can see how they like it, right? like try our new capabilities, get feedback from an actual community, do all these like community building things that we hadn't really done as like, you know, infra as a service company. But that was like a really huge focus for us. And, you know, we've grown our actual Codium individual plan to like over 100 ,000, you know, active developers using us for like, you know, many hours a day because you code for that long if you're a developer.

17:11You know, that's like plenty of feedback to us, right? Plenty of people actually using the tool telling us like, yeah, this is good. This isn't good. Like, oh, you tried pushing a new model? That's worse. Like all those things we actually learned so that we can get a product that's good. So that was like the, I guess, the intermediate period, right? Really learning from actual developers what is a good product and what is not. And I think that's like that's always going to be a key kind of part of our development cycle. You're coming into this with this rich knowledge and infrastructure for customers.

17:40That's a huge area of expertise. It's an area of expertise that even though you're moving forward into the kind of the Codium era, if you will, in my words, that is a skill set and level of expertise that very few organizations have deeply that you would have had there. How did that inform you in terms of Codium and differentiation against whether it be Copilot or other tools that are out there or just, you know, developers, you know, throwing things into chat GPT? What did that background give you that gave you that differentiation in the marketplace? Yeah. So I think when we started, the thing we started with is like, no one cares if we have better infrastructure once you're a product.

18:20Like if we have better infrastructure, that's great. But if that makes a product that's the same, no one should care. They just assume that you should. Yeah. So what we started with is we set a very high bar for ourselves. Codium is an entirely free product. So like for the individual user, it's something that they can install and use immediately for free. There are unlimited. There's like no limits at all. So like when it comes to autocomplete, you can use it as much as you want. And this is, by the way, They forced us to do things where infrastructure is as efficient as possible. Just to give you a sense of the numbers we're talking about here, we process over 10 billion tokens of code a day.

18:55That might sound like a large number. That's like over a billion lines of code a day that we process for our own developers. We're forced to do this entirely for free. And then on top of that, we probably have one of the world's largest chat applications also because it's in IDE as well. And all of this put together has allowed us to build a very, very scalable piece of infrastructure such that we're the largest users of our own product. We're the largest user of our own product. We learn the most from our users. And we can then take those learnings and deploy in a very cost-effective, very efficient and optimized way to our own enterprise users.

19:28It's one of those things where we force ourselves to learn a lot from an individual plan and then take all those learnings and actually bring them over to the enterprise. And a lot of the learnings we were only able to make because we place like very, I would say like annoying infrastructure constraints in ourselves by saying, hey, you guys got to do this entirely for free, basically. And we're committed to building, Codium is going to be a free product forever. Actually, the individual plan will always be free. And it's one of those things where our users are just always like, how are these guys even doing it?

19:56Like, what are they even doing to make this happen? And most of our users, by the way, are users that have churned off of Copilot. We have spent very little, if not anything on marketing. So it's just one of those things where our users are like, how do we make this free? We take the approach of, we think some of the best products in the world are free, like products at Google, right? They're entirely free. Google doesn't tell you all the time that they have the best infrastructure, but they do have the best infrastructure. It just so happens to be the case that that shows itself off in the best product.

20:21And we could talk a little bit more about how we take our sort of focus on infrastructure and make a much better enterprise product as well. But like, that's the way we sort of look at it. It's like, how do we deliver materially better experiences with our infrastructure? And our users shouldn't care that we actually did that. You've brought it up. You got to go there now, man. Go ahead and dive right into it. I guess like one of the interesting things, like just going to how we run one of the world's largest LLM applications, what that sort of focus forced us to do is give it a single piece of compute, like let's say a single node or a single box of GPUs, we can host the most number of users on there.

20:53So like, let's say a large company comes to us, they can be confident that whether they're on-prem or they're in VPC, we can give them a solution where the cost of the hardware is not going to dominate the cost of the software itself. Because right now there's kind of this misunderstanding that the GPUs are really expensive, which is true, they are. But the trade-off is they have a lot of compute. Modern GPUs like A100s can do 300 teraflops of compute, which is some ungodly number. That's a crazy number compared to what a modern CPU can do. And we can leverage that the best. And we've sort of been forced to do that.

21:28If we didn't do that properly, we'd have outages with our service all the time. Because of that, enterprises trust us to be like the best solution to run in their own tenant in an air-gapped way, which is fantastic because that's like the way that we can build the most trust and deploy these pieces of technology to them the most effectively because they don't want to ship their code outside of the company. Anshul can talk a little bit more about how we leverage things like fine-tuning as well. It's like a purely infrastructure problem that's very unique to us versus like any other company as well.

21:56Anshul, do you want to sort of take that? I mean, yes. I think, you know, as Viren said, there's a lot of things that we do from like the individual infrastructure point of view so that we can do crazy things like make it all free for all of our individual users. But once we actually self-host, there's actually a lot of things that you can do that just any other tool can't do without being self-hosted. And one of the ones that everyone just mentioned is personalization. If you're fully hosted in a company's tenant, you can use all of their knowledge bases to create a substantially better product.

22:27I think the way we generally think about it is that you have a generic model that's good. It's learned from trillions of tokens of code in the public corpus. But if you think about any individual company, they have themselves hundreds of millions of tokens of code that has never seen the light of day. And that's actually the code that's the most relevant for them if they want to write any new code. Think of all the internal syntax, semantics, utility functions, libraries, DSLs, whatever it might be. And a model like a Copilot or a Codium, by the nature of it having to be low latency, can only take about 150 or so lines of code as context.

23:03So this is not like one of those chat GPTs or GPT-4s where you're putting in files and files of context. It's really small where you can put in. And so there's really no way for a single inference to have full context of your code base without actually fine-tuning the base model that we ship to them on all of their local code. And so we've actually done a bunch of studies on how this actually massively reduces hallucinations and all these other things that you always hear coming up with LLMs. But things like this, things like providing more in-depth analytics, all these things actually come up by being self-hosted.