#287 Sahil Bansal: Why Developers Are Switching to CodeRabbit's AI Code Reviews

0:00At the end of the day, the LLMs are only as good as how you prompt them. and the prompt enrichment comes from the context that you bundle into it. While we don't train our AI agents on any code base, we do extensive evals of the LLMs. At the end of the day, the LLMs are trained on different code bases. Whenever a new model comes out and we have a very close working partnership with both OpenAI and Anthropic, so we get early access to those LLMs before they are released to the world. So we are able to test out, okay, this new model, we have early access to it, not let's test our AI agents with the new model against some sample code bases.

0:40And then we see the review feedback quality and we tune our LLM prompting appropriately. In business, they say you can have better, cheaper, or faster. But you only get to pick two. What if you could have all three at the same time? That's exactly what Cohare, Thomson Reuters, and Specialized Bikes have. since they upgraded to the next generation of the cloud, Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment, and spend less than you would with other clouds.

1:29How is it faster? OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50 % less for compute, 70 % less for storage, and 80 % less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is a cloud built for AI and all your biggest workloads. Right now, with zero commitment, try OCI for free. Head to oracle.com slash IonAI. IonAI, all run together, E-Y-E-O-N-A-I. That's oracle.com slash IonAI. It's great to meet you. I've been listening to a bunch of the Eye on AI podcasts and love that you've been bringing in guests from so many different backgrounds.

2:37My name is Sahil Bansal. I have a master's in electrical and computer engineering from Purdue University. And these days I work at CodeRabbit mostly on our enterprise go-to-market strategy. occasionally my friends who are not in the space they ask me what exactly does that mean or what do you do and sometimes I joke and say that my core skill is translating engineering speak to sales speak or to customer speak right you the way you know the the way the engineers spend their time and the what the customers really need to hear those are sometimes two completely different languages in. So I helped to bridge that gap a little bit.

3:21But in the past, you know, I've had my career in multiple domains of the tech stack everywhere from silicon to SaaS. Worked at applied materials on their silicon semiconductor devices for a couple of years and then in the cloud infraspace for like about seven years, most notably at a company called Nutanix. But then just the layer of the startup world and trying out new ideas and you know just the excitement of being in the middle of trying something new that brought me into a couple of different startups and most recently I joined CodeRapid about coming up to about a year ago now. Can you talk about you know there's a lot of talk about Vibe coding and and how it's helping developers code faster but at the same time it's adding a burden to code review because there's so much code that's being written by AI.

4:25Can you talk about how CodeRabbit helps in that problem? Yeah, absolutely. Would love to just share with your listeners like the background of our co-founders and where they came from. So CodeRabbit was founded maybe July 2023, about two years ago now. And our co-founders were engineering leaders in their own rights. And while the term wipe coding hadn't come around until much later, but the use of AI for code generation had already taken off by that time. And what they were realizing as they were trying to get their teams to implement the early versions of the AI coding assistance in the IED was that a lot of these new AI tools were helping code generation become faster.

5:12And obviously in the last two years, that industry has just evolved so much. We've gone from AI basically being a quote unquote smart autocomplete to now agents running completely in the background independently. yet potentially even new exploration happening with multiple agents spinning up at the same time and trying to get complicated tasks done in parallel. But what our co-founders at CodeRabbit realized a couple of years ago was as engineering leaders, they were measured on the amount of stuff that gets shipped out. And while obviously you can't really ship more stuff unless you create more stuff, but just creating more stuff wasn't resulting in more stuff getting shipped out.

6:03And they decided to, you know, focus on what the real bottleneck to shipping code was. And that is, you can't really rely on unreviewed code. You don't want to put any code out there, whether that's written by human being or AI agents or whoever. So code reviews were really the bottleneck even two years ago, which is, you know, what, 20 years at the way the industry is moving. And so that's where they decided to focus their efforts on. And we built CodeRabbit, which is an AI code review platform. So it's completely analogous or orthogonal to your cursors and windsurf and GitHub Copilot, where we don't go into the code generation space.

6:55But once the code is written, developers or somebody is still going to have to review that code. And so that's where our USB comes in, is to really help the developers not be blocked in terms of their release velocity and to just focus our capabilities on the code review side. And you talked about wipe coding, right? So with wipe coding, we've seen the need for code reviews has gone up even more. manual code being reviewed obviously being written obviously still needed to be reviewed but with wipe coding we've heard from our customers that the developers feel like they've now sort of like moved away from the best parts of their job the parts that they enjoyed the most was writing the code but now they're getting stuck in just reviewing the code written by the ai agents and that's where we've sort of like built a tool that's tried to help them and we've had some great feedback and great traction in the market where people are now able to fasten their code reviews we're not eliminating the human from the code review process at all but we are automating the stuff that used to take them more time quite often that you know extra time that goes into the code review processes simply because of factors that might be out of your control, like office politics in a way, right?

8:28Like you might raise your pull requests and then you have to wait for somebody to come in and take a look. And you might be in a distributed team globally, or, you know, the people that you tag as reviewers, they may not necessarily like you or whatever kind of interpersonal relationships you might have with them. And quite often these pull requests would just keep sitting there for days, if not longer. But that's the part that we can automate as soon as the pull request is raised, is do that first pass of the human review and leave the comments just as a human reviewer would, but then still let the humans come in and review the comments and decide whether they want to accept those comments or not.

9:11So that's essentially what we've seen with the rise of wipe coding more so recently, but just with the rise of AI coding agents in general, it sort of created a second order effect, where the new bottleneck or, you know, even before the AI coding agents, the release bottleneck was always for code reviews, but that bottleneck just becomes worsened with more and more code being generated, but developers really being bogged down with code reviews. So that's sort of like the process that we've tried to automate and try to find the bugs. that the developers can then just come in and review and accept the comments and make the changes and so on.

9:55Code Rabbit, it writes comments. It doesn't change any code. Not without a human being taking any explicit action. Can we make the code changes based on the review feedback? We can, but never without a developer explicitly saying, And, okay, CodeRabbit, I agree with your feedback. Go ahead and you have my approval. Go ahead and make the decision. I mean, in that sense, it's a little bit like the original co-pilot that acts as a pairs programmer that is offering you snippets of code as suggestions and you decide to accept or not. But why does that have to happen after the code is written? Why couldn't it happen while the code is being written?

10:59Yeah, yeah, absolutely. Let's take AI out of the equation. Let's pretend we are seven years, we found that time machine, and now we're back in 2018, pre-AI or something. Nobody cares about a word called pandemic or anything. And developers are mostly writing code manually. I would ask the question, at that time, why do we have code reviews? Because the developers are never really self-certifying. Okay, I wrote the code. I also reviewed my code. And my review passed the checks. And there are no bugs in the code. And it is going to function properly. we didn't push out the code that way. It's the same with the AI coding agents.

11:43That whole need for writing the code doesn't change just because there is now an LLM that is involved in the process of writing the code. Really, it's just that the way you review the code, we come in with a different lens. We are not biased because we didn't write the code, The LLM or the coding agent that is writing the code isn't really going to say, here, I generated the code. Now let me review my code. Oh, I found bugs in my own code. You need a different pair of eyes that is unbiased by how the code was written. And that is not simply an LLM wrapper. If it's just making any API call, then yeah, exactly what you said.

12:32You're making the API calls during the code generation. you could just do that for review. But really the review process, if you think about how do humans do reviews, let's say if I'm tagged in a pull request and I generally don't have a whole lot of idea about the code changes, I'm going to come in and review file by file. Okay, what are the changes that have happened? I will try to understand the dependencies of those files. What are the downstream dependencies? I might go to the past pull request where those files were previously included. I might look into the code base, how the various functions are interacting with each other.

13:12I might, you know, run some sort of like static rules analysis. But in the complex dynamic code base nature of today, most of the bugs are just coming because of the system dependencies, right? How is the code change going to impact other aspects of the code base. So you need some sort of a mechanism, whether that's a human reviewing the code, or whether that's an AI agent reviewing the code, that doesn't come in with that biased nature of the entity that wrote the code. That's why we believe that needs to be a completely separate market of AI agents that can help to review the code, even if that code is written by other AI agents yeah and and does code rabbit review the entire code base or or can you give it a window like of how much the code base you want it to review or does it only review the fresh freshly written code you know um in in college or in school whenever i would see an option that said all of the above i would sometimes glance over the question and just tick all of the above because I knew that would always be the right answer.

14:28So the answer to your current question is all of the above. So we work both in the IDE as well as in your Git platform, whether that's GitHub, GitLab, Bitbucket or whatever. When we are reviewing in the IDE, at that time, it's reviewing the individual commits. It's up to the developer as they're generating the code, they can just pink CodeRabbit and and say, okay, go ahead and review these 100 lines of code or 50 lines of code or whatever. But at that time, you're really relying on your entire team to have that review, right? Big companies might have hundreds of developers, thousands of developers.

15:13It becomes really hard to enforce a governance layer if you're just saying everyone, hey, in your IDs, guys, go ahead and make sure that you've reviewed the code. And again, thinking of the pre-AI world, we always mandated reviews in the pull request because that's where all the commits are coming together. That's a single choke point. That's where none of your commits, none of your code changes are going to go unreviewed if that's where you enforce the code reviews. And that's really where a lot of the cross-file dependencies are easier to understand And when the developers are building code in their isolated environment, a lot of those issues because of downstream dependencies just won't show up even if you did the reviews in the ID.

16:01So we have a dual approach to code reviews. Our reviews are completely free in the ID. We believe that should just be an easy way for you to review your code changes locally. But the reviews and the pull requests is what is mandatory. That's where you can enforce the governance layer, and that's where you can catch most of the bugs. But doing the reviews in the ID can be a quick, easy way to catch some simple bugs, and then the thorough review happens in the pull request. That's where you are reviewing your entire code changes that have happened, and in the ID can be more bite-sized reviews. And how often does it happen that, well, two things.

16:46what's the engine behind CodeRabbit? Is it a proprietary LLM? Is it a RAG database, a vector database full of properly written code or common coding mistakes? I mean, where is the decision coming from? And the other thing is every coder, I mean, although code is deterministic, there's also many ways to write a function.

17:24So how do you know that CodeRapid is not just suggesting a different way to write a function that actually a unit test completes on? And, you know, so that it's a preference thing, not an error. Yeah, absolutely. Please allow me to break that question down into two parts. The first part is, like, really, how does CodeRabbit work? So we work with both OpenAI and Enthropic, most of their latest models.

18:02But in order to get the best feedback from the LLM, So the LLM is doing the actual review feedback. But the USP is really in how you prompt the LLM. The other day I saw Karpathy, Andrej Karpathy, talk about context engineering. And that really struck a chord with me. At the end of the day, the LLMs are only as good as how you prompt them. And the prompt enrichment comes from the context that you bundle into it. If I have a function in my code where I've incorporated some changes, what has happened in the past? If I just throw that code change at the LLM, it's not really going to understand all of those dependencies that can only come from the additional context.

18:55If a developer has attached a certain Jira ticket to the pull request, what was the actual request in the Jira ticket? Did that actually get met by the pull request? If there were 20 things asked in the Jira ticket and the pull request says, oh, this pull request closes that Jira ticket, does that actually add up? or if there are, you know, if we're using the latest Go language version, maybe there's a security patch. The LLM has, has the LLM actually been updated up to that latest version? Does it have the latest version? Or if it's not, can you get gathered from the internet? So there's a lot of context engineering that we do in our LLM prompt.

19:42And when we send the prompts to the LLM, it's more or less like a 50-50 ratio of the code and the context that we have generated that explains the code. And then there is also a lot of goodness in how you package the prompt. You will have a complex code dependency graph. That's something we generate in real time every time we start a review. And then we have to build that graph for the LLM from bottoms up. You can't start middle out, right? You really have to explain, okay, these are the base functions, and here are the other functions that they're calling, and then build it out from there. And then you package all that context in a certain order.

20:34And then you prompt the LLMs. And then I mentioned the Jira tickets. So we index all the Jira and linear tickets. We index your repositories and create a clone of your repositories. And the clone is what we run the reviews on. But we also look at your past pull requests. Maybe you're changing a certain functionality right now. And it could be dependent on some other functionality changes that have happened in the past. And then we package about 40 linters and static tools out of the box. Most customers might have three linters, four or five linters that are configured with them. But we spent the time to provide 40 linter configurations out of the box.

21:24That means you'll catch more bugs. If you have more rules that are checking along with the dynamic context enrichment, that's the way to catch bugs that would be missed otherwise if you're just doing a simple LLM API call. So that's the we take a lot of time up to like five minutes or so in all of that packaging of this context then we feed the LLM we explain to the LLM okay here is what is happening now that you have the context now give me feedback on this code and then the LLMs will return some review comments. And then we also run verification to weed out the hallucination. We don't simply pass on every comment from the LLM because there could be hallucinations.

22:10It could maybe even not hallucination might provide a feedback that was already implemented by the developer. So we verify the feedback that's coming from the LLM. Does it actually add value to the code? So we run our own internal verification scripts in an isolated sandbox environment and drop off the low-value feedback. So all of that context enrichment and post-review verification before you post the comment back to the developer, that's really the USP of how we find bugs that you wouldn't find otherwise if you haven't taken the time to build that context engineering. Right. And in production, have you done tests of code bases that run in production and then have been reviewed by CodeRabbit and improved according to CodeRabbit's review and then run the improved version in production to see what the difference in production is?

23:15I mean, that's certainly there are bugs, but a lot of bugs don't really impact production outcomes. So, yeah, is there some way to measure the effectiveness of CodeRabbit? Yes. So while we don't train our AI agents on any code base, we do extensive evals of the LLMs. At the end of the day, the LLMs are trained on different code bases. But whenever a new model comes out and we have a very close working partnership with both OpenAI and Anthropic, so we get early access to those LLMs before they're released to the world. So we are able to test out, okay, this new model, we have early access to it.

24:05not let's test our AI agents with the new model against some sample code bases. And then we see the review feedback quality and we tune our LLM prompting appropriately. But in terms of whether or not to post a certain feedback, because maybe it's really not a bug and the customer doesn't care about it, that power we give back to the user. They can tell CodeRabbit, great, I see the feedback that you posted me. I don't care about this. Don't give me this feedback in future. So our bot, in a way, it's basically implemented as a GitHub app and similarly on the other platforms. And you can chat with it just like you would with any other human reviewer.

24:58And we learn from that feedback. and then the next time we won't give you feedback of a certain nature. Now, chat can be very quick and easy and very intuitive way to provide feedback on reviews. Sometimes you want the review to be more deterministic, like always do XYZ for a certain file. Maybe in my directory named application, new application 2025, always validate against the latest JavaScript style guide. So that's also something that you can provide as instructions, path-based instructions in this directory for files that match a certain path, always run these rules or do not run these rules.

25:46Those can be custom and our customers make extensive use of those kind of rules. And then we include those rules and we'll tune our feedback accordingly. But these days, a lot of those rules are already now being written in the AI coding agent themselves. And if you've already written the rules for the code generation, it makes sense to also expose those to the code reviewer. So if you have something like your cursor rules, markdown file setup, cloud code, GitHub Copilot, you name it, We integrate with virtually all of the commonly used AI coding agent tools. We'll automatically import those rules and tweak our code reviews based on those.

26:29So that it's really up to you how you want the review feedback to come out to you. Yeah, and I guess generally code review falls to a junior employee, a junior coder, or at least a mid-level coder. and the ideas that, you know, with the context and with looking at the entire code base, that CodeRabbit will actually do a better job than giving it to a mid-level coder who doesn't want to do the job in the first place. Is a really interesting phrase that one of our customers' key value systems said during our discussions with them. CodeRabbit has helped raise the floor of the knowledge of their team because their junior developers now felt more comfortable asking the AI agent questions about the code, right?

27:33Like if there is an ongoing review, you can simply ask us questions. Wait, I don't understand why are you telling me to rewrite this function? I thought this would work perfectly. Why? Tell me more. Explain me your thought. And so you can do sort of like a Q &A. about your code base even. And CodeRabbit will explain you, okay, here is what I see in your code base and here is how this will work and this is why I'm recommending you to refactor your code. And quite often the junior developers may feel intimidated reaching out to a senior developer or somebody who's been with the company for a long time.

28:05And so it was really interesting to hear from this customer that, you know, the junior developers now feel like they were more empowered. But in terms of reviews, The review quality is actually what I would say is that a more of a mid to even senior level engineer. You know, and with all these AI agents, there is no longer any blocker on any languages. We support our architecture is completely language agnostic. There have been customers, I believe, Sales Rabbit, Groupon was another one, where portions of their code bases were written in legacy languages. is C-Hash. Now, there's various reasons why that code base can come into your company.

28:53In the case of SalesRapid, it came through an acquisition, but they didn't have anybody who knew C-Hash. So that's where now you can go in and ask the AI agent to review the code that you might not even have the expertise in-house to review. So we really see the review capabilities at the mid to senior engineer level, But we also see that the entry-level junior engineers, they feel more empowered by being able to use AI and not having to try to people-please or anything and not having to rely on any kind of interpersonal relationships to get their reviews unblocked. And the idea is that with the context you feed the LLM and the guardrails against hallucinations, this is a focused tool that will give better performance than just garden variety or even like Claude Sonnet 3.7.

29:58because you've done all this sort of prep work on the code before you run the review. Exactly. That prep work is really, you know, where you can really elevate the review quality. Otherwise, you're just relying on the non-deterministic nature of the LLM. You really have to feed the LLM information with the proper context in bite-sized chunks. and the right amount of context. If you give it too much context, it will get overwhelmed and it will try to treat different portions of your code base with relatively the same importance. If you give it too little context, it might not catch some of those complex bugs that come in because of the interdependencies.

30:47So really that tweaking is what the engineering that we've built in in-house, how to give it the right amount of context, what are the sources from where we should gather the context, and those sources include custom rules that you might have set up, more than 40 linters for the static analysis, doing a real-time web search in case the LLM might be out of date, building that file dependency graph. And a lot of this context building happens in real time every time a review is triggered. then we gently prompt the LLM with the right context, get the review feedback, validate the review. Is this actually going to address the comments or is the review not worth it?

31:33And then we post the review feedback. And that's actually really, if you think about it, that's how a senior engineer would have done the review anyway. So we're really mimicking that human way of thinking, but automating it so that the review starts immediately and even catching the edge cases. Sometimes people might forget to update their documentation or to update their unit tests because of new edge cases that will get introduced. And just by the very nature of our human way of thinking, it's not always easy to capture those edge cases, but the AI can do that. So we can capture those edge cases and prompt you, you need to update your unit test to catch these edge cases, or you made a certain change, but it's not documented.

32:23And we can do all of that for you. Can you talk about some of the customers that have used, I mean, you mentioned one, but about some real world success stories with CodeRabbit? Yeah, absolutely. We have more than 6 ,000 paying customers. We use CodeRabbit today for their PR reviews and more than, I believe, 70 ,000, 80 ,000 open source projects. So a lot of our initial traction came in from the open source community. If you think about it, that's really where the pain of the reviews is even worse. You might have an open source maintainer or a few maintainers, but there's loads of contributors.

33:06and now those maintainers have the thankless job of reviewing all of the contributors' code. So they really like it a lot when they have the power of AI to come in and automate their code reviews for them. And a lot of times, you know, they're working in their own day jobs and so then they use the CodeRabbit for their open source project, bring it into their day jobs. That's really how the company got a lot of product-led growth in its early days. But lately we've also been seeing a lot of expansion into the enterprise space, you know, where you have big teams, complex code bases. No individual person really has the grasp of the full picture, so to speak.