#221 Varun Mohan: The AI Tool That’s Changing How Code is Written (Codeium)

0:00When you use a product like ChatGPT, it's largely speaking ephemeral. It only knows what you've chatted with it, not about the hundreds of millions of lines of software that exists within the company or in your code base. And sort of we've come up with a product that's able to kind of, with a human in the loop, execute many steps autonomously to come up with a final work output. And maybe a crazy stat, the company actually has over 50 % of the company non-technical because we have a go-to-market team as well. And everyone in the go-to-market team has used the product to build successful applications, like entire applications, which is crazy to think about.

0:28Okay, so why don't you start by introducing yourself, where you went to school and how you started ExaFunction and pivoted to Codium, and then we'll talk about what Codium is doing today. Hey, I'm Varun. I'm originally from the Bay Area, right now in the Bay Area as well. I went to MIT, got my bachelor's and master's there in computer science and computer systems. And actually after college, worked at an autonomous vehicle company based out of Mountain View called Nuro. Worked there, was one of the first 12 engineers at the company. The company grew pretty big. I became a tech lead and manager at the company, managing large deep learning infrastructure.

1:10and after that actually left the company in 2021 to start ExaFunction, which is actually the precursor to Codium. And we were actually a GPU virtualization product. So we made it so that computations could run entirely on CPUs that actually had a GPU component. And transparently, we'd offload all the GPU computations to a remote machine, right? And for us, we were able to manage workloads upwards of 10 ,000 GPUs. And we even built GPU compiler technology. In the middle of 2022, we realized that transformers were going to become very popular and powerful with the advent of these generative models.

1:49And we felt that generative models would usher in a brand new age of computing, similar to the internet. And when the internet is going to get started, we wanted to be a Google or an Amazon. And we believed by becoming a vertically integrated application company, we could go out and do that. And that's why we built out Codium. And we started training our own models and running in them ourselves. And in the process, we actually recruit over 800 ,000 developers using the product. We actually process over 150 billion tokens of code. We're one of the top five to 10 large generative AI apps in the world right now.

2:21And some of the worlds are just enterprises currently use Codium. So JP Morgan Chase just inducted us in the Hall of Innovation. Over a thousand enterprises currently use the product. And maybe the most exciting thing that happened very recently is the product is largely speaking available in the place developers write software, which is called the IDE. And we work in every IDE and every place that developers store their code, regardless of if they store them on GitHub, if they store it on AWS, or any of these other large companies' products. We work everywhere, which is a big differentiator. And we work in a way that is self-hosted.

2:55So the company doesn't need to share their code outside of the company if necessary as well, which is security is obviously a big concern for IP as deep as code. But very recently, actually this week, we launched a product called Windsurf, which is our own IDE, right? Our own way of writing software. And this is the first agentic IDE that has come out. And it actually enables developers to not just write code and auto-complete their software, but actually complete tasks end-to-end and continue from where they left off. So very different from some of the current agents out there where you just give it a task and it completes it, we are actually building the product.

3:30So the mind of the human and the AI kind of meld together, right? So if the human goes out and make some changes, the AI can just kind of continue from where the human left off. And the agent is also able to deeply reason about large code bases, right? Which is actually a big deal, right? When you use a product like ChatGPT, it's largely speaking ephemeral. It only knows what you've chatted with it, not about the hundreds of millions of lines of software that exists within the company, right? Or in your code base. and sort of we've come up with a product that's able to kind of, with a human in the loop, execute many steps autonomously, right, to come up with a final work output.

4:02And maybe a crazy stat that's kind of interesting is the company actually has over 50 % of the company non-technical because we have a go-to-market team as well. And everyone in the go-to-market team has used the product to build successful applications now, like entire applications, which is crazy to think about. And the interface is what? Is a conversational AI or a graphical interface? So it's very familiar. It's very familiar with VS Code. It's actually backwards compatible with VS Code. So everything you can do on VS Code, you can kind of do there. But the difference is sort of you can give high level instructions to the system and the agent-like system that lives within WinService called Cascade.

4:41And Cascade will actually go out and edit files, many files for you. It'll actually reason about your code base and potentially even run code for you, right? One of the unique aspects of code that's very different than marketing copy or legal review is it can actually be executed, right? Which is why code can be very easily tested. And because of that, it can actually be self sort of introspective, right? Because you can actually run code, see if there's an error and kind of go and fix it itself. And it's able to complete very complex tasks because of that, much more than what just a generic product like JGBT.

5:11First of all, Is this built on your own LLM or is it built on one of the proprietary? I mean, are you hitting GPT-40 or O1 with an IEPI caller? How do you, what's the foundation? So it's a mix of both, right? Like for us, we have many different models that run in the product. We have models that run on every keystroke that are not only auto-completing code, but what we like to call super-completing code. And what that means is as you're typing code, it's dynamically refactoring the rest of the code. You can actually tab to accept an entire refactor of your function, right? And then also at the same time, though, there's a system in Cascade that it does high-level planning.

5:53And then also on top of the high-level plan, there's from the high-level plan actually doing retrieval of your code base. And then actually going and applying the high-level plan to the code base. Those are proprietary models. But actually for us right now, we've actually settled on for the highest level planning model, we're actually using products like OpenAI or Cloud to kind of do that. And down the line, we're happy to also use products like Lama or whatever as they continue to get better. But it's kind of a mix of everything, right? It's almost like for different kinds of domains, we want to use the best products.

6:21And we do feel like right now OpenAI and Anthropic have potentially the best sort of like planning models out there. And I saw, I mean, this is evolving so quickly. OpenAI has just announced a further extension. I'll cut it out because I can't remember what it was, but it was related to coding. Um, the, uh, and how do you compete with co-pilot or code whisper, which are, are focused on the same task or market? Yeah. So I think, I think the way we, we compete with these companies is, um, I would like to say like the product is more beloved. Like if you look on BS code and just like the rate of adoption, it is, it is, it is pretty remarkable right now.

7:04We have, you know, a five-star review on the, on the marketplaces, uh, where developers use the product. So I think it's delivering more value for the end user because it's more aware of the code base. We're not locked into any ecosystem, so we're not locked into the Microsoft ecosystem, nor the Amazon ecosystem, which is important because customers actually have multiple clouds that they operate on top of. They use many different source code management tools. The only source code management tool isn't just GitHub. There's GitLab, Bitbucket, Perforce, Garrett, Mercurial, CVS. I can keep going down and down on all the source code management tools that kind of exist inside the enterprise.

7:35And then finally speaking, I think because we are vertically integrated, we can actually provide experiences that are more personalized, right? We can more deeply understand their code base. We can run it in a way that runs entirely within the firewall of the customer. And all of those put together makes it so that we're able to offer a differentially sort of better and structurally better offering that's not just like tied to the ecosystem, right? That these big incumbents already have. Yeah. And so you're writing code from scratch or is this for existing code bases that you want to, as you said, refactor or do something else with?

8:11It's both. I think like probably the most valuable application is existing code bases, right? People in large companies aren't like, you know, building these apps from zero to one. The product can very easily write apps zero to one. But if you look at where most of the value is, like, you know, if you look at probably the, you know, the most valuable companies out there that use a lot, like, let's take a JP Morgan Chase right now. JP Morgan Chase isn't giving you a new app to use every week, right? They're kind of taking their existing app and adding more functionality and making it more robust, right?

8:38And that's a very hard problem because the amount of software that already lives within these companies could tend to the billions of lines of code, right? Maybe tens of billions, right? So, you know, you put that all together. And actually, the hard problem is, how do you actually take existing applications and not only understand them more deeply and onboard developers onto them, but actually modify them and extend them in ways that are most valuable to the company? Yeah, that's something I've always wondered. I mean, if you have an enterprise with a code base, billions of lines of code, how do you understand it?

9:12And how do you work on it? And does something like Codium, can it ingest or review a code base? And if so, what's the limit? And how much time does that take? Yeah, I mean, this is like a deep problem that we spend a lot of time on. So, I mean, look, I want to say it's a very hard problem, but we've also solved very hard problems like technology problems in society. Like, let's even take a product like Google, for instance, like think about the amount of code that exists inside a company and compare the amount of text that exists on the Internet. And Google is indexing the entire thing and able to do good retrieval on it.

9:51Right. So there are like hard problems that have been solved. and I'm going to say like there are other companies that have solved it in crazy scale. I think what makes code a particularly more challenging problem is everything is interdependent, right? A code base is actually, when you ask it a question, it's not, you know, a little bit different than maybe Google where every web page is kind of, you know, its own thing. You know, maybe it's hierarchical of like the website has some metadata attached to it and the web page has something. Code base is extremely complex, right? Like you have some directories that are related to some other directory that's like many, many layers different internally.

10:25So I think it's a hard problem. And we spend a lot of time finding out ways like how do we cluster code, right? How do we actually use static analysis to make it so that we more deeply understand it? And how can we do retrieval very quickly? Use properties like embeddings, use properties like keyword search, use properties like LLMs to be able to find like semantic meaning in the software. And yeah, I don't think there's like a very clean answer of like there's one approach that kind of works. kind of similar to like how, you know, if you were to take a product like Google and ask like, how does retrieval work?

10:53Yeah, there's an original idea called page rank that says it signifies what the rank of a page is given a match. But you know, if all you did was do page rank, I doubt you'd be able to recreate Google. You know what I'm saying? When I started working on my own, and I didn't have the structure of an organization behind me, I knew that I needed to incorporate, file taxes, and all of that stuff. I looked into getting a lawyer, but it was prohibitively expensive. At that point, I really wasn't making much money. So I turned to LegalZoom to get the corporation officially registered and legally compliant.

11:32This episode's sponsor is LegalZoom, a trusted leader when it comes to setting up a business right. I'm a true believer because I still use LegalZoom. So business owners, have you set up your company properly? The pain-free way is LegalZoom. It's the most trusted way to set up your business and stay out of legal trouble. So launch, run, and protect your business to make it official today at LegalZoom.com and use the promotional code IONAI to get 10 % off any LegalZoom business formation product, excluding subscriptions and renewals until the end of the year. Get everything you need from setup to success at LegalZoom.com.

12:20That's L-E-G-A-L-Z-O-O-M.com and use the promo code IONAI. That's E-Y-E-O-N-A-I. LegalZoom.com with the promo code IONAI. LegalZoom provides access to independent attorneys and self-service tools, but it's not a law firm and does not provide legal advice, except where authorized through its subsidiary law firm, LZ Legal Services, LLC. Yeah, sure. But when someone adopts Codium and they have a massive code base, what are the steps? is it um do you add uh codium as uh as like a plug-in to your to your repository so you can add podium as a plug-in to your ide right um and and by the way it's also like recording statistics about your company uh which are like helpful of like how are different teams getting value from the product right like what are the most valuable parts of the product like how much of your code that's actually getting written is generated by codium um but then also now like you can install Windsurf, which is our own custom ID that I think is like the best ID experience out there in the market.

13:36So we don't want to be, I guess like our goal as a company is not to be dogmatic. I think like developers should choose, right? Like it's kind of like how developers don't just do one thing across like the entire enterprise. And we will provide like a host of suggestions that we feel very confident in and we'll put the stamp on it and the company should decide afterwards. Um, but, but so just, just walk me through how, how someone would use Codium if they have an existing code base. Yeah. The way these applications work is you would open them in the, in the IDE, right? Um, you would open the piece of code on the IDE.

14:07And then after that, you would be able to, uh, do a host of factors. Like if you already know kind of like what you're doing as you're typing, it would actually auto-complete and suggest like changes within the code passively. Uh, right. This is what we call the super complete experience internally. You can either like do that or highlight a piece of code, use this feature called command and just suggest like a high level refactor and it'll rip through the code base in hundreds of lines a second and kind of like give you the final answer of like, here's what the change should be, which is awesome for refactoring.

14:32or at the very high end, if you have like a very, very high level idea of what you want to do, you can actually use something like Cascade, which will go out and actually look at the entire code base, find out sort of where the changes need to get made, make the changes, right? And then maybe even run your unit test for you to say like, yeah, I feel confident about those changes. And you can even ask it to go and commit the code and push it to the final repository. And Cascade is part of the codeium, I'm sorry if I missed that. It's part of the Windsurf IDE. Yeah. Yeah. And it writes the unit tests as well?

15:06It can also write the unit tests for you. Yeah, I can write it. And then after that, I say unit two. And then run it. Yeah. I mean, is, you know, these are probabilistic models and they, even with all the safeguards and engineering around them, they can still make mistakes. So how do you, and if you're working on a massive code base and you're doing something like Cascade, you're going back into the code base and changing different elements. how do you guard against screwing it up, making a mistake and then not knowing where that mistake was made and suddenly you've got a bug in the code? Yeah, I think the way we like to do it is like, look, the AI is almost treated like a person that can make mistakes.

15:59This is a big part of the product. Their UI is also important. We give people easy ways to go out and actually review the changes that are actually ultimately made it. and after reviewing it, we want to see if people feel that it makes sense, right? Like you have to review the software and like the same kinds of bugs that humans make, AI is also going to make them. And I don't think this gets away from it. In fact, like think about it in general, like even with and without agents, like people were copying Stack Overflow code and stuff into their code base. Like that didn't eliminate the need to review software.

16:28So I think reviewing software is still incredibly important. I think AI can actually be very helpful in the review process. It can kind of give you explanations on like what this code does so that you can actually like kind of find out like, hey, what are the pieces that are probably most important from like most important to least important for me to read? Right. And it doesn't mean you don't read the least important thing. It just means that if you want to get a general sense of where the most bugs are, probably start with the part with the highest likelihood of bugs. Right. So it makes the process of reviewing software much more efficient.

16:55And but no, this doesn't eliminate the need to review software. In fact, it probably increases the need to review software. And that review, as you said, is AI aided, but you still need people that can look at chunks of code and read it and ensure that it's accurate. That's right. Yeah. The review process is not going away. And how does someone use Codium to write applications from scratch or models? Same process. You can actually just, you know, empty directory, just say like, I want to build app X and it'll go out and systematically build it. It might ask you some questions about what maybe some more requirements it has.

17:35And, and, you know, you can kind of see where it goes. Yeah. And that's with natural language that you're, you're. That's with natural language. That's right. It's just amazing how this is progressive. Go out and download it and try it out. Like, I think you'll be like excited about what you could actually build in within five minutes so this is a wildly successful people are using it all over the world uh where do you go from here because you you said that there's a certain amount of autonomy um but um you know agents are the hot new things society of agents or networks of agents uh are are are you looking at that at uh uh introducing agents so that you users can direct them and they'll go do the work uh without a coder actually having to type the code um i think i think the place where this stuff is moving is um you know when i think about the process of software development it's like there are many stages right there's like uh writing software reviewing software designing software, testing software, deploying software, debugging software, right?

18:44There's a lot of different pieces. I think what's going to fundamentally happen is it's going to be like chopping wood from every single one of these places with time. Like very clearly just writing software fast doesn't suddenly like make the time to review software faster unless you like actually make a concerted effort there. And you're going to be chopping wood from every single one of these places. And as you sort of chop more and more wood, what I think that will enable developers to do is spend more time doing the things they actually want to do. I don't think like wasting time deploying software that is actually, you know, kind of like, like a pretty deterministic thing should be something that developers spend a vast majority of their time doing, right?

19:17I would love to get them back to actually designing software, solving the pain of and solving the problems that they actually want to solve, right? Like, what's the purpose of software? The purpose of software is to fundamentally solve a problem. It's not to write software, right? So I think, I think, I think it's going to fundamentally give people a lot of time back. And I think more people can then participate in the software creation process than was possible before. That's what interests me because I'm not a coder, but I'm very interested in software. And I've talked to a lot of people about various no-code platforms or code completion.

19:52I've talked to Amazon about or AWS about Code Whisperer. So your selling point is mostly ease of use. Is that right? I don't think I would say that. I think maybe the right way to look at it is, in fact, I think this is enabling people to do things that they could never do in the past. I'll give you a couple of examples here. With a system that now deeply understands a piece of software, let's just take one example here. People that are new to a team or new to the company can now onboard onto a very complex system a lot quicker, right? Because they can actually ask deep questions about the code base, right?

20:25That's one. Maybe a more important one that I think is important is a lot of languages are actually languages that are not very common, right? Like let's take a company that's writing in a language like COBOL that was mostly written in the 70s, but it's largely speaking needs to be maintained because some applications still do run on mainframes, right? I think Genetive AI makes it so that it's easier, right? It's easier to introspect these applications and update them. So I think maybe the right mentality is not actually this is easier. I think the whole point is actually to make things that were not possible in the past now possible.

20:53Because easier is one thing, you can throw more people at that, but smarter and more capable actually means that the business could do things that it never could have dreamt it could do in the past. That's interesting, the comment about COBOL. How many languages can Codium handle? Yeah, so I think the crazy thing is it can actually handle sort of the entire gamut of sort of languages in the software development lifecycle. It can handle like every language out there. Is that because it was trained to do that or it can just on first glance recognize the patterns in the code? I think it was because we do actually oversample on some languages, right?

21:38Like some languages like Verilog, RTL that are more uncommon. But I think it's also because the base model understands a lot about different languages. And then after that, we're able to use forms of context awareness that you read different parts of the code base to then provide better suggestions. And what are some of the biggest enterprises using code? I mean, you don't have to name them. But yeah, so very recently, we just kind of like won the JPMorgan Chase Hall of Innovation Award. So a lot of like, you know, I think like we've been great vendors to one of the largest software sort of software companies in the world.

22:14Right. They actually work with the most number of software companies out there. And and I think I think sort of some of the largest companies that you can think of are currently using Codium. To build this, you started with GPU optimization. did you just have excess development muscle and say, yeah? No. I think we decided that this was more important and more likely to actually succeed and be a durable company. And because of that, we pivoted almost overnight. Wow. And you had to build the system overnight. Yeah, we needed, it took time and we didn't, there was a period of time when we had, we needed to work on something when we had nothing and we prodded along.

23:04How does Codium relate to things like Devon, which got so much press when they released their first marketing video? Yeah, I think the, I think, I think it's hard for me to know what that product actually does, given it's not accessible. I think like maybe one of the things is we're generally accessible so you can use the product now. That's like that's that's one nice differentiator. Right. I think. But but but beyond that, I think I can't really tell. But in terms of of that, the being able to ask the the system. I think for what it's worth, if if they are capable of doing what they what they said they were, which is replacing a developer, which is effectively what they said, right?

23:49software engineer, it's like, I guess that's way more powerful, but also at the same time, like I've not heard anything about them doing that in the last six months.