Building an Immune System for AI Generated Software with Animesh Koratana - #746

29:24And if you predict correctly, right, you can actually kind of backpropagate through the entire trace, through all of the different actions and choices that the agent took iteratively in order to actually get to a particular prediction and make sure that we basically, you know, reward or, you know, disincentivize that particular trajectory. Is your ground truth, your kind of verifiable output coming from actually executing the code or from some knowledge of past code behavior? Like when we're actually running a simulation, right, We don't have a way to actually verify. I'm more talking about training and RFT and giving that signal to the model as you're training it.

30:13Yeah, so that is actually a combination of verifiable outputs. So we actually have code bases where we found specific types of issues and then back propagate against that, run a simulation, and make sure that we're actually reaching the same verifiable output. And then, you know, there's actually ways to evaluate against kind of probabilistic outputs as well. And so you can actually kind of guess, for example, we went across a lot of different open GitHub repositories. And we said, OK, well, what are open bug tickets there? And can we go and verify that those bug tickets are still correctly open?

30:49Meaning you have, that's really interesting. You have a ticket that was filed sometime in the past. look at the current state of the code and try to predict whether that ticket is open or closed. Yeah, and so you can use things like this as like heuristics to basically get, you know, 80, 90 % of the time, yeah, it's still correctly open. And so it ends up giving you like a directionally correct signal for being able to tune these models. I would think for a lot of tickets, like, I don't know, you look at enough like GitHub threads and like there's a lot of discussion like about just like providing basic information about the ticket or about the you know just uh kind of prerequisites to reproducibility and uh maybe this goes back to where we started the conversation around debugging but it seems like a lot of value would be created in just like you've got this janky ticket with no detail like can somebody you go like, you made the connection, run that up and, you know, run that or, and what, what is that?

31:54Is it like, you've got the ticket, like simulate it or, um, infer information about, you know, the ticket from like, you know, you might have structured fields about the OS and the person mentioned the iPhone. And so you can make some inferences there. Like what's the process there? That's a, that's a great question. I, I, you, you, you made the jump for me. So the the way this helps debugging is debugging is really about hypothesis testing and the best engineers in the world are the ones that are really good at taking morsels of context and then basically collapsing the space that they have to test to two or three really high value things and that's basically what we're trying to do here right where we can actually take these simulations and use them as a way to actually run a hypothesis test.

32:45That's essentially what a simulation is. And then you could start taking context from, you know, whatever's in the ticket or whatever is in your telemetry system and use that to heavily inform which hypotheses you should actually test to begin with. And so if you have a rigorous enough understanding of the code base, if you have a rigorous enough understanding of, you know, what the customer actually was kind of talking about, right? What were they complaining about? You could use that as a way to go and automatically come up with a few hypotheses, simulate them and say, hey, this is the most likely reason why this particular problem happened.

33:21And so the simulation actually becomes a really powerful tool, not only for software verification, but also for debugging. And it creates this really interesting trade-off, which is how important is the bug actually for you? because you could actually turn a dial and say, I want you to go test a thousand different scenarios, a thousand different hypotheses, and come back and tell me, hey, is this, you know, like exactly why did this happen? Now, obviously that costs money. It costs a lot of inference, right? That's an expensive thing. But you are almost guaranteed at any sufficient scale to say if I turn enough inference, if I throw enough compute at this problem, then I could figure out why this thing broke.

34:03I just need to test enough. scenarios. So enough monkeys at typewriters and you're going to get the right answer. No, but it's just a really interesting trade-off. Like that never existed before, right? Like you can never actually scale inference or reasoning to the point where you can actually use that to say, I will always get you to the right answer if you give me enough time. You know, maybe the monkeys in typewriters is, you know, is, I don't know if it's apt here or not at, but like continuing with that analogy is, you know, it's not having one of those, you know, millions of or thousands of outputs be Shakespeare.

34:46It's being able to figure out which one is Shakespeare in a scenario like this, because you said that you're not like running the actual code. So these things are coming back with, oh, I think this is, well, and this is maybe a uh, proceeding question, your agents are going out and like simulating, you know, something, what, what's the output? Is it English? Like I have looked at this ticket and I think it's this, or is it like fixed code or is it better tickets? Is it, uh, something else? it's incrementally adoptable. And so the output of a simulation is, yes, it's English, right? It says like a scenario is defined as an initial state, a few steps and an expectation.

35:36The output of a simulation is, did the simulation result in the expectation or not? So there's a pass fail result and then there's an explanation as to why it passed or an explanation as to why it failed. Now you could take that simulation result and say, go back to player zero and say, hey, could you actually go fix this problem for me? right oh i forgot i didn't realize that i broke this thing right could you just go fix it for me right and there's a nice little loop right that actually you know forms there um in terms of kind of ticket triage right i think there's something similar so a ticket comes in you know we test a few hypotheses we come back with a few recommendations someone often can come back and say could you go write up a better ticket for me right i want to go pass it into cursor or cloud or whatever your tool of choice is.

36:22Or you could just say, hey, Player Zero, could you actually go and make this fix for me as well? Right. And so there's many different kind of ways that people actually use this. But at the end of the day, they're basically kind of flexing that muscle of, you know, Player Zero has a really deep understanding of your code base. And as we said, right, code base and its relationship to reality. And they're basically exercising that relationship, exercising that understanding to be able to create specificity to the problems and then verify software as it's being produced. Yeah, I'm still wanting to kind of sharpen the distinction between this and an agentic coding type of tool.

37:01Yes, like at a presentation level, like a ticket and, you know, copy pasting an error message into cursor, like are kind of different things, but they're kind of the same thing. like something's broken. There's some, you know, a string of characters describing a broken thing. And I'm telling a system that I don't like this, go fix it. In fact, like, I don't even need to tell it, I don't like it. I could pass an error message and it knows it's an error message and it'll go find the issue and fix it. And so you're saying like one distinction is, you know, let's call it mechanistic. Like, you know, all these tools, like, you know, we think they're LLM tools, but they're really RAG retrieval tools, but we don't call them RAG because like that's for words and this is code.

37:52But it's all like how we construct this context. And you're saying like you do better things to construct the context with graphs and stuff like that, which, okay, great. but in terms of like

38:06I'm trying to A get to the question but also get to the answer like there's a distinction here that I feel like we're trying to articulate so let me I'll pass the ball yeah let me try to create like a good mental model right like what we're really focused on at player zero is what happens to your code after it's left the agentic developer right after it's left the IDE. And, you know, as we said earlier, right, like that is software verification. So how do things break? What things might break? And then after that, ultimately fixing it. And you're right. I mean, there's a little bit of overlap in some of these workflows.

38:47But I think like from a technical perspective, I think there's like two or three things that really drive the difference. One is, yes, like we take a lot of pride in our ability to understand really large and complex code bases. And I think the muscles required to do that are different than the ones required to be, you know, dropped into your IDE, right? That is a little bit more of, okay, well, you know, which files are open? What is the exact context that's, you know, of the specific repo that I've cloned right now? The muscles even around retrieval, right, are different when you're talking about, you know, hundreds of repos or, you know, dozens of different services, kind of all interacting with one another.

39:31So that's one, right? So just like the way that we understand and train across these really large repositories. The second, I think, comes down to how we capture institutional knowledge. An analogy that I really like to understand this is a lot of what we talk about internally is building an immune system. And this is something that I get really excited about because, you know, what is an immune system to us as humans, right? It is a barrier between our biology and reality. And it's constantly learning, right? It's constantly learning from every, you know, interaction that we have with reality.

40:12Anytime we're exposed to a new, you know, pathogen, a new virus, bacteria, whatever, it's constantly learning. It's remembering those things. And then after that, it's becoming a part of the way that we live every single day. so that way we as humans can basically go outside you know live freely you know play outside do whatever we want right um and and it's this thing that we can very easily take for granted but it's this kind of long-term memory um that stays with us our entire life just to make sure that we are resilient uh to the pace of things that we're basically throwing at our biology and you know we started this conversation talking a lot about the pace of development has increased and we can't think about software as a a machine anymore we have to think about it as an organism right it's evolving and it's evolving in ways that we don't fully understand and it's interfacing with the reality in ways that we don't fully understand and a lot of what we're trying to do with these scenarios and code simulations and stuff like that is build that immune system layer for your software and so it's constantly being exposed to reality, you know, through its connection to the ticketing system, through its connection to the telemetry, through its interaction with just, you know, the commits that are happening.

41:28It's basically building this long-term memory and accruing that over time. And that requires a discipline that actually spans beyond the code base as well, right? And that's really what I'm trying to get to here, right? That's kind of a secondary and important differentiator from, you know, what you would really want out of cursor or a cloud code or something like that. And then the third thing is, you know, our ability to actually run verification without compiling. And that's the code simulation piece of this, right? That's a really unique capability. It's a really hard thing to do, to actually be able to say, hey, throw a billion lines of code at me and I'll predict how, you know, if this invoice comes in, in this country, in this particular, you know, configuration, exactly what would happen in the system and would it behave correctly or not, right?

42:13And it's built on those first two differentiators. and adds on to it as well, you know, with some of our own custom models, with some of our own, you know, evaluation, with some of our own research that we have to throw at it. The underlying machinery is all LLMs and, you know, that ilk, as opposed to, like, it's kind of calling to mind, like, automated verification and, like, you know, kind of NASA-style, like, going through the code and, like, really understanding what every line is doing and using that to predict outputs given inputs, but we're not talking about anything like that. Yeah, like it's a combination of different LLMs.

42:55There's a couple of different reinforcement style or reinforcement learning style models that we've trained as well that are actually both LLM and not LLM based. So there's a few kind of just graph models that we've trained as well that can just go and traverse different paths through the graph. And then, yeah, there's like a couple of different retrieval and embedding models that we've got to tune. So it's a combination of many different things, but the capability, I think, here is what's really important. On the models side of things, I'm assuming you're using pre-chain models and fine-tuning them.

43:28You mentioned RFT reinforcement, fine-tuning. Can you talk a little bit about the base models that you selected and like how you think about that space? Yeah, that's a quickly evolving arena as well. I think just yesterday some cool things happened. So, you know, I think there's probably two or three, like, important models that we rely on. So in terms of retrieval and stuff like that, we use a lot of open source sorts of things. So there's, like, a tuned version of Llama. I think we're actually evaluating the GPT-OSS stuff that came out yesterday. So that'll be really interesting. seems to perform quite well, given some of our internal training sets.

44:17So yeah, there's a lot of stuff around retrieval that we've trained. In the code tracing, and a lot of the ontology built on top of that graph, a lot of that comes from slightly more effective reasoning models. this tends to be a combination of some of Anthropics models they call it some of the older Sonnet models plus like a variant of DeepSeq that we host that's able to kind of reason through a bunch of these different traces and explore a few different hypotheses and then after that we found that Sonnet you know Anthropics flagship model tends to be really good at kind of being an orchestrator between a bunch of different models to be able to kind of like manage.

45:12Yeah, kind of like a router tends to be really good for that as well. And so all of these different things all come together at the end of the day. I would say that middle layer tends to be where a lot of the heavy lifting happens. So that's the thing that's, you know, tracing through a code or, you know, trying to retrieve all of the right context to basically predict the next step. These types of models tend to be, in terms of raw tokens, the maximum. They tend to be the ones that are most heavily leveraged. And so those are the ones that we also spend a lot of time tuning, bringing in-house, managing the inference for.

45:49And so when you have a new customer or a new system with a new or old customer, are you doing a lot of pre-compute on their code base or is it all like, you know, query response and you're kind of doing everything real time? Yeah, there's always a balance here. Yeah, there's always a balance here. We do have some pre-compute that needs to happen, especially at the types of code bases that we deal with. Like creating embeddings and that kind of stuff? Yeah, so it's embeddings, it's building that graph. Building the graph. And, you know, we also do some sort of training. to basically say that there's some sort of online learning that happens as well to say like hey every time a pr is merged in right make a prediction for yeah what might break and once the thing has actually gone in we then go and say well what actually broke and did our prediction match reality and then basically go and tune our models further right and we do this on a tenant by tenant basis right so we're not using one customer to go inform someone else but this is constantly happening for all of our customers using like laura adapters per customer or something like that yeah so laura laura was a great way to start um we've actually found that like you know rft over kind of longer time batches um tends to just work a little bit better uh laura has some uh some limitations especially when you start getting to like reasoning models and stuff like that um but yeah i mean there's there's a ton of pre-compute that happens um usually most customers are live in a couple of hours after they've actually imported their code bases and we're off to the races.

47:38But sometimes it takes a little bit longer, sometimes a little bit shorter, really kind of dependent on the size of the code base. And the number of H100s you thought? Yeah.

47:51You said it, not me.

47:57Yeah, no, exactly. Exactly. When you started a new customer, like, maybe this is taking a step back from Player Zero and what you're doing. I'm curious, like, what you're seeing, you know, in terms of the landscape and the way the customers are thinking about, you know, whatever they call this phase that we're in, identity coding, AI-sisticating, vibe coding, whatever. Like, what are you seeing? How, you know, far down that path they are? Like, I'm assuming, like, the bias will be towards larger code bases. I'm kind of envisioning a bias towards, like, more mature companies or more traditional companies.

48:44For Player Zero? For Player Zero, yeah. Yeah, I mean, we have a pretty diverse set of customers. I would say that the problem that we solve around supporting the code base tends to be something that happens when you have a code base that customers are actually using. And so, you know, like we're not really working with super early stage startups for sure. But usually the problems of scale and complexity. Do you see both like your like digital natives, your metas and your ADTs or visas or is it more the latter than the former? Oh, yeah, that's it's a pretty healthy mix between both of these. I mean, like one on the former side.

49:32So think a little bit more tech forward is Zora, for example. Right. And like really, really sophisticated engineering team distributed across the world. Zora, like the pricing SaaS or subscription SaaS or something like that? Yeah, exactly. And yeah, they really care about the verification side. And I think this is really interesting, actually, because you talk to different companies and they articulate the same problem a little bit differently. But some of the, I would say, more tech forward companies tend to care a lot about the seat belt. So they tend to care a lot about, well, how do we move faster and not break things preventatively?

50:16I think some of the more maintenance mode companies who basically have just a large amount of accrued software over time tend to think about it a little bit more as, Now I have a really expensive set of software and TCO, right? Cost of ownership is really high. How do I lower that burden in a world where the velocity is increased, right? Four or five X, right? And so like some people are thinking about it as a like support burden and other people are thinking about it as a, you know, how do I really just kind of help accelerate development by scaling verification at the same pace as I'm scaling the speed of code output.

51:03And so this is the... Maybe this is just another way of asking you to... Maybe this will only result in a restatement of what you just said. But I'm wondering if you're able to infer from what you're seeing if all of these tools will... allow companies to like, you know, close the same amount of bugs with less resource or, you know, I guess, you know, tell me this, tell me there's like a utopia of software quality right around the corner because of all this stuff we have coming online. Yeah. You know, I actually, I actually believe this really, really fundamentally. Like, I don't think that using AI to reduce resourcing is necessarily the right way to think about it.

52:08And I think what's happening, right, with we're seeing this with agentic development. And I think we're going to see the same thing with software quality and support is what we've done is basically leveled up the judgment that the humans can actually apply into the workflow. Right. In development, the developers are no longer thinking about, should I name this variable this or that? Or how should I implement this particular function? Right. They are thinking about what to implement. And I think as these models converge and they get better, the judgment that developers are able to exercise basically levels up one degree.

52:52That doesn't mean that the world needs less software. The world is hungry for probably a thousand times the amount of software that it's able to produce today. And I think thinking about this in terms of the asymmetry that we were talking about at the beginning, as we start tackling some of these fundamental challenges in the development side of the house one by one the burden on quality both in QA and support across the rest of the SDLC is going to exponentially increase and I think for the amount of software that we're going to be able to produce there aren't enough humans in the world to be able to manage that process the way that we are today.

53:39And so I think thinking about it as a, oh, I could, you know, bring in Claude and fire 50 % of my developers, right? And just to kind of say it bluntly, right? Like that's like a narrative that's being told or saying this in the other way, like, oh, I can bring in player zero and, you know, let go of a bunch of my QA or support staff is the wrong way to think about it Because I think all of these things are going supersonic. And what we need to do is basically think about our workforce and say, how do we help them exercise higher judgment in the workflow to create more leverage, right? To create more leverage in how do they verify, how do they test, can they test more in the same amount of time, and use what they understand about the business to be able to guide these systems to get more work done for them.

54:32Yeah. Yeah.

54:38It's like a false premise, I think, to say that the world has enough software. And like, yeah, demand is going to keep increasing as supply increases as well. It's an interesting dynamic. Are the kind of next gaps already obvious to you? Meaning you got this SDLC that's like evolving rapidly because of the infusion of AI. you know, you tackled code gen, you know, you saw maintenance quality or some sections of that as an opportunity, but like, there's still like this long workflow, or maybe not, maybe it's just those three, four things. Like, how do you, are there kind of, you know, is what you're doing exposing other things that are broken that need either you or someone else to fix?

55:30You know, one of the things that I realized as we were building Player Zero was that quality is not an exercise, like a point in time exercise. It's a process. And it's how do we learn from the mistakes of the past to be better in the future? And I think a lot of what we're doing with this, you know, analogy of an immune system and, you know, scenarios and code simulation and like all these different things coming together is basically trying to create this this process right to operationalize this process and make it a part of how we build software every single day and i think as the ai native development era matures a little bit i think the uh uh The question that I don't have an answer to is, what does cohesion in that development workflow actually look like?

56:32You know, how do these different things have to work together in order to orchestrate and operate software at the next level? And I think there's many different bottlenecks here today. I think you actually mentioned one of them, for example, which is... If we think about developers as prompt engineers and are basically asking Cursor to go iterate on some software that they want to build, and they have zero understanding of the actual code, does the language still have to be Python? Or can it be Java? Or could it be some other obscure language that only computers understand that is easier for language models to write?

57:19I don't know. And I think as we start seeing different bottlenecks in the development process actually shift and this entire broader SDLC mature a little bit, I think you're going to start seeing these second or third order sorts of questions being asked to say, how do we optimize the entire SDLC end to end? So that way, these workflows are a little bit more cohesive. another I think like simpler version of this is just you know how does player zero interact with cursor or how does it interact with cloud code and stuff like that so like workflows are something that we're really focused on internally to make sure that this is as expressive and easy to adopt as possible but I think it does beg the macro question of yeah adoption is going to be really important.

58:11It has to be very accessible in the SDLC, not only for its current moment, but for what the workflows look like, you know, two years from now. I think another way to ask the question that I asked was, you know, if you, you know, didn't start player zero or you did start player zero, but you're some other person and you're looking to start a company to like work on this SDLC, like, what would you start, you know, now? So there's actually one other thing that I think about often, which is security. It's not too dissimilar from quality, but it requires a different muscle to be able to understand, in the same way that we understand the relationship between code and reality from a quality and kind of verification standpoint, I think there's an important muscle to be built about if we're producing this much code this quickly, how do we make sure that there's no holes in it?

59:11And I think this matters, especially for larger companies, but I think it's a universal problem. I think that's one of the obvious ones that comes up. I mean, you mentioned one thing that is on my mind, and it was kind of the question that I asked about what you're seeing in your customers. We have all these tools, and It's not, you know, because it's all moving so quickly, it's not fully clear how they're all going to fit together. Like, you know, take our conversation. Like, you're kind of like, well, you can ask ClogCode to do it or you can just ask Player Zero to show it. Like, there's a lot of overlap.

59:53There's a lot of like, you know, shifting sands. There's, you know, we're in this like land grab phase where, oh, this CLI seems to be catching on. let's all create CLIs like a big question that I you know find myself asking a lot is like what are all the pieces going to be like once you know once the sand start shifting or like we start creating stop creating new pieces like what are the fundamental pieces that will continue to exist and how do they interact with one another I think there's still a lot of questions there I think the best analogy, I mean, like software creation is like a non-deterministic, imperfect process, right?

1:00:41There's many different ways to produce the same software. There's many different ways to write the same code. There's many different ways to verify it. And so we have these gates that we've built deliberately to say, here's how we produce software at scale for our customers and do it well. and i think the the human process is some indication for what are the important centers of excellence that a good sdlc has right and so production is one i think verification is another evaluation is another and then support is another and so production is okay well how do we generate more code faster and i think there's going to be something like you know a cursor or Cloud Code or something like that, right?

1:01:26That really kind of sticks their flag in the ground for that. I think verification is really where we want to play. And that's really where we take in all the stuff that Cloud Code and Cursor are producing and say, is this actually correct, right? Verify correctness of the code. I think evaluation is, okay, well, I think there's like observability and companies like that where I think you say, well what's actually happening out there in production right now and you know is there a spike in 500 errors and you know is something breaking and basically kind of the incident response and i think there's there's a couple of cool companies in that category um and then i think in terms of support i think that is basically kind of closing the loop and bringing it back into development of saying well what broke in our verification and how do we make sure it doesn't

1:02:22kind of puzzle that we're working in. Yeah. So those are kind of like the high level, like broad strokes, like what I see is the important areas of investment in the SDLC. I think we call it different things. We call it developers and support and QA. I think in the future, it'll be a couple of different agents doing the legwork and like a few human orchestrators above it. But whatever we call it, I think those are important things to be done. um and high quality software some of that does kind of strike me as like fossilized conway's law like you know we've got these organizational groups dev and qa and you know you know what have you and like we built these workflows that like uh well fossilized like these organizational boundaries and you know part of what's interesting is the point you made earlier like if you know we're able to like re-imagine the whole stack, you know, in light of a kind of a native AI approach.

1:03:28What does that look like? And to what degree does it differ from what we're looking at now? Yeah, I mean, I think it's going to change a lot. Hopefully we get to have some hand in making that happen. What, meaning and not just like the AIs?

1:03:50we as people or we as player zero both actually

1:03:59awesome yeah awesome well adam ashley it was great uh catching out with you learning a little bit about what you're up to and uh looking forward to keeping an eye on how you guys are playing in the sdlc absolutely sam thank you so much for having me on this is this is a real pleasure i had a lot of fun talking about this yeah my pleasure thanks so much Thank you so much.