Sourcegraph and the Frontier of AI in Software Engineering with Beyang Liu

0:00Sourcegraph is a powerful code search and intelligence tool that helps developers navigate and understand large codebases efficiently. It provides advanced search functionality across multiple repositories, making it easier to find references, functions, and dependencies. Additionally, Sourcegraph integrates with various development workflows to streamline code reviews and collaboration across teams. Biang Liu is the CTO and co-founder at Sourcegraph, where he has worked for the past 12 years. In this episode, he joins the show with Sean Falconer to talk about the frontier of leveraging AI and software engineering.

0:37This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

0:56Bion, welcome to the show. Great to be here. Thanks for having me, Sean. Yeah, absolutely. So you've been working on Sourcegraph now for over a decade. I guess like how have things evolved and what are some of the big pivotal moments in the company's evolution? Yeah, definitely. So, I mean, as with many things, it's like some things stay the same. Some things change drastically. Obviously, the impact of AI has, I think, introduced in everyone's minds the potential of changing the way that software is built. And there's certainly a lot more bot or computer generated code now, a lot more use of large language models to write all sorts of functionality across many different verticals and industries building software.

1:37But there's still some of the fundamental problems that still exist, which are kind of the same as the ones that we were originally started to tackle. So the fundamental problem of software development, not really having economies of scale. So like in contrast to every other industry, every other industry, you build a bigger factory, get more efficient, the stuff gets cheaper to produce as you scale. Software engineering is basically the opposite, right? Like as you grow, things get less efficient. So you've got mythical man month. Yeah, exactly. The mythical man month still applies today, even in the age of LLMs and AI that we live in.

2:14And, you know, this harkens back to the founding story of Sourcecraft, which is really born out of my co-founder Quinn and my work inside very large enterprise code bases. We were both engineers at Palantir. He was before deployed. I was just a regular engineer, but we're both going into the field a lot and working with a lot of Palantir's like Fortune 500 customers. I shouldn't say a lot. We worked with like the first two that Palantir landed. We were kind of on this like SWAT team of sorts that was trying to open up a new line of business for Palantir. And that was our introduction to very large, messy codebases.

2:52I mean, Palantir's codebase itself at that point was probably like five, six years old. So, you know, it wasn't as bad as the codebases that we were being deployed into, but it was still enough to have, you know, a bit of cruft, a non-trivial amount of tech debt. Was that their sort of like desktop version at the time, like basically running some sort of Java suite? Palantir's code base? Yeah, yeah. Yeah, yeah, yeah. So, well, actually, it was kind of a combination of things. So, you know, originally, the project I started on was like a Java thick client, because in those days, you have to remember, like this is before React, before kind of like the modern web development stack, Backbone JS and CoffeeScript were new at the time.

3:35So like that was the hot new thing. and so to get the richness of interactivity they needed for a lot of the like data analysis workflows that they're trying to enable java swing was was the only option when they got started but then by the time we started we got put on quickly on on this like upstart team and so we were using web technologies on that team so you know front-end javascript and and that sort of thing but we had to integrate in the context of these very large like banking code bases because those were the customers they were trying to go after and that was just a huge pain it was just like you couldn't get anything done.

4:07It took like weeks, sometimes even months just to like, ask around and acquire enough understanding of how the code fit together to even, you know, begin to think about like what you should be building or how that fit into the broader picture. And that was our first introduction that kind of gave us this insight into the scope of the problem, because like we were being deployed, Palantir was brought in to solve like very high priority problems. We were talking about like, you know, things that were costing banks like millions of per day that were on some level existential if they weren't solved within a finite amount of time.

4:41This is all kind of like hangover from the 2008 financial crisis and cleaning that up. And so, yeah, it was kind of crazy to us that these institutions had all the money in the world to throw at solving this problem, could not solve the problem. And I just think that speaks to the fundamental problem that faces any software engineering at scale, which is software engineering really doesn't scale. That's still the problem that we're here to tackle. So we've kind of tackled it for a lot of customers at some level. And I think being able to integrate LLMs and AI into our tech stack, I think, makes it possible to actually sort of solve the mythical man month problem in the next decade or so, which is really exciting.

5:19Can you elaborate a little bit on that? Like, how do you see AI helping solve that problem? Because presumably, I think the problem with scaling engineering is you end up with this large scale dependency graph, essentially. Yeah. With tight coupling, essentially, between people. Communications break down, everything just grinds to halt. Yeah. Could you potentially recreate the digital version of that if you were using AI as well? Because you're going to end up with a lot of these sort of digital dependencies, essentially. Oh, totally. And I think where the attention is today is all about AI-generated code, which is great.

5:52I mean, it's truly mind-blowing. that you can create a whole simple web application basically in one shot with any of the big frontier models today. You can do a lot with them inside existing code bases too, in terms of updating and editing the code with the right context. But I think the bottleneck, people still don't fully grok that the bottleneck in large-scale software engineering is not on the writing code side, It's on the understanding and reading code side. And so to your point, it's like right now, everyone's just like taking code out of the LM and basically pasting into their code. That maybe, you know, increases by an order of magnitude the volume of code you're able to generate in a given amount of time.

6:36But, you know, does that actually move the needle? Like each line of code in, you know, the engineering discipline, we tend to view as more a liability rather than an asset. Like the functionality you get from a line of code is an asset to your users, an asset to the business. But the fact that you had to write a line of code in order to do that is kind of like a liability. It's like a column in the tech debt stack. And so I think there's a very real world where like the amount of code in the world just like balloons. And there's all this like slop, which wasn't even written. There was never a human brain that conceived of it and understood how it worked, at least with like human written code.

7:08Like somebody at some point understood how it worked. With the AI generated code, like that might not ever be the case. And then it might even exacerbate this problem of like, hey, the existing code is a tangled spaghetti mess. to the point where, you know, the AI can't work in it. And I as a human can also not work in it. And so, yeah, I think that remains the challenge ahead of us. And that's one of the challenges that we want to solve. And that is sort of this like fundamental challenge of the mythical man month. So for those who haven't read the book, it's kind of like a classic about the nature of software development.

7:39It's core thesis. It was written in the seventies, basically, you know, someone looked at the progression of software projects inside these like mainframe companies back in the day and noted the fact that like there was this weird phenomenon where when they would add people to a software project in order to speed things up, it would actually end up slowing things down because it introduced just more communication overhead and a loss of kind of like coherence or unifying vision in the code base. You know, a lot of like abstractions that would clash with each other, things, pieces that wouldn't fit nicely together.

8:11And so the conclusion there was, you know, like software development, it's more like birthing a baby rather than like churning out widgets in a factory. And I think, I don't know if it's from this book or somewhere else, but it's like, you know, you can't have nine women make a baby in one month, right? Like there's no way to speed that up because there's all these like serial dependencies. And the task is fundamentally like, it requires like one person to shepherd it along. And the analogy applies to software. It applied to software development back then. It applied to software development, you know, in 2019 before ChatGPT, and it still applies today.

8:43But I think the, the possibility, the potentiality that we see and are excited about is that with the benefit of LMs, you actually have a lever around human intelligence to be able to go to like an architect or like a senior engineer and say like, we will give you this tool that you can wield to enforce the type of coherence and standards and consistency across the code base that you need to keep things clean as you add more contributors and you scale your engineering org. I don't think there's a way around the scaling because the fact of the matter is, if your software becomes successful, you're going to have more users get on it.

9:19You're going to have more customers. Those customers are going to have feature requests that tailor the software to their needs. And so you kind of have to spin up more people to build all those features. The challenge is how do you manage the complexity and maintain that coherence of vision as you add more contributors into the project? In that case, this idea, maybe you can leverage AI to do a lot of the code generation, but then you sort of have, you know, this like architect that's acting like the moderator or the orchestrator. They understand how the pieces go together. They understand what the standards are.

9:49And they're there to do essentially the understanding piece of this where AI is maybe not ready to help us with today. Is that the idea? Yeah, that's exactly right. In an ideal world, you would have like one very smart individual, you know, reviewing all the code that goes in your code base today, ensuring that it fits architecturally into the vision that they had in mind. Now, people don't do that because if you try to do that, you literally could not fit the amount of reviews you'd have to do. That person would basically have to spend all their time just doing that. And even then, they would only cover a fraction of the code that's being committed into the code base.

10:23On the generation side, I think that AI for coding, I think, is one of the areas where AI is finding a lot of success today. And And in a lot of ways, that seems surprising in some sense. But I also think that it also makes a lot of sense in a number of ways as well, because there's a lot of examples, there's a lot of code that's been digitized, that's available for training. And the other thing is that one of the hard problems, I think, when working with large language models is it's hard to kind of benchmark and eval them versus if you're building like a purpose-built model, you know what the inputs and outputs relatively should be.

10:56So I can essentially create a set where I can evaluate the model. I make changes, I can reevaluate the model. It's hard to do that with a general purpose model. But if I'm doing code generation, it constrains the universe and I can actually build like a relatively good benchmark and eval set to test against. I'm curious, you know, from your perspective, like, do you think that is one of the, you know, primary reasons why AI encoding is finding such success? Or is there also other factors that at play? I definitely think that that is one of the primary reasons why it's finding such success. I think like valid code is much more constrained, a much more constrained domain than valid language, because you have this kind of like Oracle that tells you first, you know, does it compile?

11:37If it doesn't compile, you have something that like tells you exactly the error that's preventing it from compiling. And then if it does compile, you also have these things called unit tests that validate the correctness of the function. And so that I mean, you can use that in a variety of ways, like you can use that inference time to verify and validate the output that you get to make sure that it's correct. You can also use it at training time. You have a way to generate a high quality set of synthetic training data by basically taking like the output of an existing model, running it, and then checking the errors that it produces when it attempts to generate code that conforms to the user query or prompt.

12:14And so you put those two and two together and you have a vector for making models much more precise and useful at inference time and also a way to improve the models at the training step itself, which I think a lot of the frontier labs have been doing. And if anything, it's almost like the fact that this exists for code, it's great for the coding domain, but it also probably generalizes like logical reasoning capability outside the coding domain. I remember when JetGPT was first released, there was a lot of speculation that a lot of the emergent capabilities came when they started training the model, not just on natural language, but also on code.

12:52Because there's certain logical referential patterns in code that map to logical reasoning that then can also generalize to logical reasoning patterns in natural language as well, which is really interesting. So that kind of helps us with the generation part of this. So what is Sourcegraph doing to kind of help with this understanding problem? Like if I'm using AI or even, you know, I have engineers, humans that are generating all this code, how do I actually, as that thing grows over time, maintain an understanding of it? Yeah. So there's kind of two ways where we differentiate in the understanding domain and make our product, you know, just like much more powerful and impactful on like big, messy production code bases.

13:37One is at the kind of like context retrieval layer. And then the other one is at the kind of like validation and verification layer. So the way to think about this is if you're trying to use AI to generate code, you know, first you want to give it an example so that it gets within the rough ballpark of what is right or what is reasonable in your code base. Like without the benefit of contextual snippets, it's basically analogous to asking a human to write code for your code base, but not actually letting them look at any of the other code that exists in your code base. So like, what would a human do in that case, they would just go to Stack Overflow and like, write code that, you know, fits your description, but uses just like the common open source frameworks that are well represented on GitHub or Stack Overflow.

14:21And that just doesn't fly within a lot of like messy production code bases, because, you know, the longer code bases existed, the more kind of divergent and special snowflakey it becomes. So the first thing we do is we use our code search engine, which is our first product, our original product, which is kind of like a Google for code and does much more than search. It also does like code navigation and it has access to a couple more like structured databases about code. These like ontologies of code and technical context. We bring those relevant snippets into the context window and that helps steer the LLM to generating something that is much more within this distribution of what is acceptable in your organization.

14:59So what we find is that there's a lot higher acceptance rates for the code that we generate inside the organization. It doesn't have to be like reworked or rewritten as much because it has the benefit of that context. And then on the second side of things is like the validation verification layer. We've been partnering with a lot of our enterprise customers to figure out how to like automate a lot of the review that is done inside their organization. So, you know, code review is this process that's, you know, kind of like half box checking exercise, but half like, you know, actually there's, there's some like useful things that we want to catch.

15:33And I think the way that humans do it, oftentimes it becomes like 98 % box checking and like maybe 2 % actually catching stuff because the box checking part just takes up all the time and everybody wants to get over with. So they can get back to like actually building stuff, which is more fun. And also something you get more credit for, like no one, no one gets a promotion for doing a great, you know, code review. Right. But machines can do it much more thoroughly into much higher quality than we can. So one of the things that we kind of got pulled into is we found all these customers of ours were basically hitting our APIs and building their own internal code review bots against them.

16:08And so we're like, wait a minute, there seems to be some commonality across these things. And so we started building a code review agent that's now in early access. I just gave a talk about this at the AI Engineer Summit in New York with one of our partners, booking.com. And the interesting thing here is I think it actually is, it opens up like a new paradigm for modifying or constraining what code goes into your code base. Our kind of like catchphrase way of describing this is like, it's almost like a declarative style of coding where you declare these rules and invariants that you want to hold in different parts of your code base.

16:42These are things that like your architect or your senior engineer could specify. Like I wrote this class this way. It should only ever invoke this other component in this other way. Or if you ever see this pattern code, this is an anti-pattern, rewrite it to be more like this. Change all your map functions to for loops or maybe vice versa, depending on what your overall philosophy is. Now you can define these rules in one place and just have them automatically enforced. You don't have to go and manually review every piece of code looking for these sets of patterns and anti-patterns. What's the representation of those rules?

17:14Are you using ontologies as the underlying representation? The representation right now is mostly just like natural language, plus a way of specifying like which set of files in the code base this rule should apply to. So there's kind of like a selector for like, hey, this is the subdirectory or this is the logical project where this rule should hold. And then the rules themselves, I think right now they're mostly natural language. I think in the next phase of this, there'll probably be like, some will be natural language and some will be natural language descriptions, but implemented by the invocation of like a precise tool.

17:46Like, you know, this particular pattern, the AST should never exist. I've described it in natural language, but like, you know, translate this to a, you know, tree sitter query or something like that. And that's how you can like test and verify. In the context retrieval or the code search, you mentioned that you're using ontologies there. So is that ontology like a base model representation of the concepts? Or are you starting with the base and then making something that's like customer domain specific as well? We think of it as like a knowledge graph that is built around the structure of the code.

18:19So like this basic skeleton of the knowledge graph is really the reference graph in the code. So you have definitions and you have references. And that's all code is at the end of the day. It's a bunch of definitions and then they're referenced in other places. and this is the knowledge graph that you're kind of like implicitly walking when you as a human go in and try to understand how something works right like you maybe like you do a couple searches to get within the rough ballpark but then after that it's like go to definition go to definition find references you know hover over the symbol to see what the docs are and so like that knowledge graph is very useful for humans it's also very useful for surfacing potential relevant context for lms as well.

18:58And then around that kind of like core code graph, there's other things that also come into play. So there's like a long tail of technical knowledge that is stored in things like issue trackers, corporate chat, production logs. And we've built a way into our system to integrate these other pieces of knowledge. So we had this thing called open context, which is basically like a simple protocol for saying like, hey, there's this knowledge source, you know, maybe it's your issue tracker, and here's how you query it, here's how you select the specific item is very strong parallels to MCP now, but kind of like predates MCP by year.

19:32We built an open context to MCP integration. I think we might like merge the two in our platform moving forward. But this was something that we had very early because we recognized the strong need to pull in relevant context from all these different other sources that were ultimately connected back to the code graph in some way. Like you have an issue tracker that pertains to a certain area of the code base. You have a production log that has a stack trace and all those things in the stack trace map to lines of code at some specific revision. So constructing that knowledge graph just allows you to have this map of how do I pull in all the relevant pieces of context when a user asks a question or wants to generate some code.

20:13Yeah. I think having developed something similar to MCP before MCP became public makes a ton of sense because if you're doing any of this stuff with relatively complicated systems, you're just going to end up with your AI having to pull in data from all kinds of different places. Like you can't have essentially a gravitational model for data where you're just dumping it all into a wake and then surfacing it from there. You have to essentially tap it directly from the source in order for it to be the latest information and not have to deal with people like re-architecting the data. You're not going to be storing your source code in your lake anyway.

20:47It's not going to make sense somewhere else essentially. So you have to liberate the data from all those different places. Yeah, totally. And it helps to have like a structured knowledge source, right? Like these two things are complementary, right? Like LLMs have this kind of like basic form of reasoning and the ability to integrate context. But what they don't excel at is structuring things down to like a very precise, you know, spec or description. And you saw this with like, you know, especially like the Gen 1 LLMs where like they had trouble emitting valid JSON. on. Now that after billions of dollars of training, they can finally do that.

21:23But it's still a very inefficient way to acquire that structure. It's far better to just put it into a symbolic representation and have the LLM walk that and then use that as a context fetching mechanism to complement its core memory and reasoning abilities. In terms of the commit review or PR review agent, can you walk me through what does that workflow? So I submit a commit of PR, this agent monitoring my PRs, then what happens essentially? So there's kind of like two steps to this. There's like the rule definition process, and then there's the rule enforcement stage. So the rule definition process just actually happens in these files, these like source graph rules files.

22:02And you can put these in a particular directory or subdirectory or repository, you know, wherever they're defined, you can define like a pattern of like what files these rules apply to. And so that's something that you define in your code base and you define once. And once you define them, we're actually building multiple hooks into the software development lifecycle where they are enforced. So one of the things we're doing is we're actually bringing these rules into context in the editor. So when you're generating code with AI in your editor, using our editor extension, the rules are part of the background context of the LLM.

22:36And so the LLM can take into account the rules when generating code. Now, that's not 100 % guarantee that they'll actually follow the rules. Oftentimes, they ignore them, but it's better than nothing. It saves you some time. It's kind of like shifting left this process of like getting feedback and constraining the code that you're creating to be valid or acceptable in your code base. Now, the second place where they're enforced is at code review time. So you push up your patch for review. And then after that, you tag in our code review agent. And then it goes and takes a look at the rules that apply to each hunk that's been changed and then post comments to that hunk, noting problems that break the rules or suggestions on how to fix the code so that it conforms to the rules.

23:19Is my understanding there, basically every source file has some set of rules that represent its essentially file-related context. And then you can use that as a way to essentially evaluate the commit against the lines that changed and what the rules are for that specific line, that specific context. It's not every file. It's more, I would say, at the subdirectory level. Okay. Because it might get kind of tedious to have to define. You probably will have a lot of duplicative rules if you had to define it per file. But you can define it, think of it as roughly analogous to an owner's file or a code owner's file where you're not necessarily defining that per file, but you can say like, oh, this subdirectory belongs to this project.