This AI Agent Builds Better Code Than Most Developers (Factory AI)

0:00Eno Reyes:These aren't humans, they're a different type of tool that humans can use. We have customers that have quite literally run sessions that go well under the 80 million tokens. Two or three years ago, the term agents had really negative connotations because the only agents that were really out there were extremely unreliable. No one had seen this sort of fully autonomous interaction form factor yet. It's a fully self-contained binary that runs on any operating system, on any device, in any environment, with any interface that you can imagine.

0:39Welcome humans to the Neuron Podcast. I'm Corey Knowles, here with our purveyor of words and slinger of cat puns, Grant Harvey. How are you, Grant? Doing well, doing well. How are you, Corey? I'm doing good, man. Doing good. I'm excited because today we're digging into the rise of autonomous software agents. Factory AI is one of the fastest growing companies in this space, building droids that can take tickets, modify real code bases, and work inside existing dev workflows. But before we get started, please take a quick moment to like and subscribe so you don't miss out on videos just like this one.

1:12And joining us today is Factory AI co-founder and CTO, Ino Reyes, to talk about how this works, what breaks, and what it's like to scale a company at this speed even. Eno, welcome to The Neuron. How are you?

1:26Eno Reyes:Thanks for having me. Excited to be here and excited to chat more about the droids. The droids. The droids. Were you inspired by Star Wars with the name droids? My lawyers say that that is not something that I can comment on. Just kidding. No. Actually, the name's origin comes from the fact that two or three years ago, the term agent had really negative connotations because the only agents that were really out there were extremely unreliable they would break at every corner and when we wanted to sell to enterprises if they heard the word agent they would associate it with basically like a research experiment so we came up with a name that we felt would also prevent people from thinking this is like a human that's going to replace us like a lot of people name their ai tools with a human name and we said, look, these aren't humans.

2:17Eno Reyes:They're a different type of tool that humans can use. What problem with agentic coding were you trying to solve when starting Factory? Like what felt broken about how software teams worked at the time that you started it? When we started Factory, my co-founder was doing his PhD at Berkeley and I was at a company called Hugging Face that worked on LLMs and open source tooling for the related. And actually, when I was there, it was pre-LLM. So there were BERT transformer models and tons of different variants of that, diffusion models, all this other stuff. And one thing that was pretty consistent was for the organizations that wanted code models.

2:59Eno Reyes:All of them were looking to fine-tune a model for their code base and maybe get autocomplete. when you when we talked around to a bunch of companies we saw that everyone was very fixated on coding which obviously is extremely important but the biggest bottleneck oftentimes in most software organizations is not can you write the lines of code to solve the problem but it is the entire end-to-end process and the bottlenecks and the the the blockers that occur along the way right? Gathering context, understanding decisions, planning changes, stitching all this together. And once you get the code, you know, that's almost like the easiest part of the process.

3:42Eno Reyes:So we wanted to make sure that AI systems that we built for software development could help people with that entire end-to-end process, which required us to basically do research on agentic systems that didn't exist at that point in time and figure out what can we do to make them general and very capable across these different tasks. So was the idea like immediately that we're going to do fully autonomous agents or was it, were there stepping stones along the way you were hoping for? How did that vision kind of evolve if it did along the way? Yeah, I mean, it's kind of funny and there's actually a quick answer.

4:22Eno Reyes:It basically has been autonomous the entire way. I think this might be one of the bigger differentiators between us and a lot of the other players in the space. We had no point wanted to build autocomplete or wanted to build an IDE. From the beginning, it was droids, fully autonomous systems. You know, our mission when we first started and to this day is to bring autonomy to software engineering. And so we built out the first droid prototype. It was a Slack bot that you could chat with and it would work on its own and come back to you. And this is in 2023. and so basically you know that no one had seen this sort of fully autonomous interaction form factor yet yeah but along the way there were a ton of lessons I think we were way too early with that vision of software development and interaction both because the models weren't necessarily strong enough to do the breadth of tasks you wanted to do and required a bunch of harnessing but also because people weren't used to that interaction pattern it was surprising to us after like six to seven months, we got to this point where customers would say to us, you know, it's actually like working pretty consistently, but we're not really sure how to teach people how to use this interaction pattern.

5:38Eno Reyes:And that made us realize that we were, we needed to meet people a little bit closer to where they are. And so then it evolved a couple times to the current form factor that it has today. So how did your background and hugging face shape the way that you approach the autonomy question, I guess. Yeah, totally. And I think that it actually goes back a bit further. My first sort of foray into deep learning and LLMs was in a research lab in undergrad that focused on computational models of cognition. And basically, the thought process there was, you know, you want to be able to look at different ways that fMRI, EEG, brain activity interacts and see if you can model algorithmically or with deep learning how those interactions might play out.

6:28Eno Reyes:Not to say we know exactly what's happening in the brain, but just to say, you know, we can actually imitate some input output algorithm that clearly the brain is also executing. And so there's some insight there, even if it's not the same thing. And I found that to be like a really interesting idea is like, look, maybe it doesn't matter what's going on inside as long as you can model it. And, and so from there going to Microsoft and seeing like a very large organization, software development life cycle, experimenting with ML strategies and techniques, and then hugging face where most of these, you know, ML models were hosted.

7:06Eno Reyes:I think that there was sort of this, this clear side of like what software development looked like at different scales, how people were thinking about different deep learning and ML strategies. Transformers had, you know, very, very recently become apparently the strongest method for most deep learning techniques. And it really was seeing how people were very fixated on the singular use cases of transformers. Like we want to build a classifier or we want to build a, you know, entity recognition system that when LLM started to become strong, seeing how general they were and how they started to eat all the different use cases, it became clear that there was a different approach towards software development that wasn't going to involve piecemeal things like, oh, now we have autocomplete.

7:58Eno Reyes:Now we have a tool for classifying whether the code has a bug in it or not. And so all of that bubbled when, you know, basically the LLM APIs came out in a way that was accessible to a software developer, where you could just call OpenAI's API. You could just call, you know, Cohere was actually like a popular API at the time. And so I think that there was a couple of folks in the open source community, including myself, who basically just said, what if we put this in a while loop? Like, what if we keep seeing what the LLM can do? What if we add structured parsing? And so I think those ideas were all bubbling up right around 2023, early 2023.

8:42Eno Reyes:and seeing it actually all come together. Like when I met my co-founder, it was actually a seven day period from the time we met to, you know, basically, you know, he dropped out of his PhD. I quit my job and we started factory. That's how fast you could build something that was different. What was that conversation? Yeah, like how did you guys have that conviction in seven days where you're like, let's do this? I think that there's so much to it. You know, he likes to call it intellectual love at first sight. I think I agree with that. We knew each other in undergrad very lightly, you know, Matan and I, and I think that the thing that was nice is because of undergrad, you know, you sort of knew, we knew each other, we had social circle overlap, but we weren't actually very good friends.

9:29Eno Reyes:We like barely, barely had chatted. And I think that it came at a really serendipitous moment where he was basically, you know, had basically kind of stopped doing his work as a physics PhD, was transitioning towards AI and very interested in AI research. I had built out a bunch of these prototypes for, you know, while loop LLMs. I thought it was going to start a company with this other person and around like finance AI. And interestingly enough, the, the turns out when you're trying to build a finance AI agent, you really need to build like a python ai agent and so having a system that can write python and execute it in a loop very quickly makes you realize well maybe there's some adjustments that can be made to make this just general software development and so when we when we joined together and we spent that week basically hacking and putting it together to a to a point that made it more general it became clear that he was very convicted that i was very convicted and then a healthy sprinkle of, you know, we're both young and, you know, you only have a couple shots on goal when you're in the early days.

10:36Eno Reyes:I think that's actually in retrospect completely untrue. I think you have infinite shots on goal. But, you know, at the time it felt like, oh, we have to do it, do or die. When you say droids, what essentially are they? I know we're talking about agents, but what are they really doing under the hood and how autonomous are they today? You know, I think that there, this is probably the most interesting question is like, what, what actually is a droid? Uh, you know, and for us, there is a lot of work that we put into, uh, because it's changed a lot, right? When it first started, it was this like server side, you know, effectively while loop on a backend server.

11:13Eno Reyes:Uh, then it was, uh, you know, a full stack application that you had to interact with either in the browser, um, or you could connect it to your computer. And now I think that one of the biggest architectural insights that we've introduced is, you know, the Droid should really be the most simple possible program that you can imagine. It's a fully self-contained binary that runs on any operating system, on any device, in any environment, with any interface that you can imagine. So the actual Droid itself is just this very simple program that orchestrates and manages LLM calls and that can be run anywhere.

11:57Eno Reyes:And what that enables is really this extremely wide range of use cases where when you take a Droid and you put it inside of a terminal interface, it becomes this like really snappy, fast terminal agent. When you take a Droid, that simple binary, and you wrap it in an IDE extension, it becomes an IDE pair programmer. When you take a Droid and you put it in your CICD pipeline, like your GitHub actions, your Azure DevOps pipeline, it becomes an automated code review or an automated security review. When you take it and you put it in a cron job, it becomes an automation that can do almost anything you could do on a computer.

12:37Eno Reyes:And so the architecture is built for generalizability so that this individual agent can now take control over the file system and the environment it operates in and really do anything that it can with computer control. So it's this really powerful concept that is very inspired by the Unix philosophy of simple, modular, composable programs. And I thought you had, you have four kind of main ones that you were kind of marketing at first. I think you touched on some of those, like a security one, or like ones that are customized for different use cases. Is that accurate? Or is it, has it expanded since then.

13:10Eno Reyes:Yeah. So the actual droid itself is actually, you know, is totally general. Now, what developers can do, though, is there's tons of customizations that they can introduce. You can introduce skills, which give it the ability to basically pull in context and work on any workflow or bucket of information that you want to pull in. You can do hooks. You can do custom slash commands, all these different customizations that you can bring in, sub agents, et cetera. And so what people can do is if they want a code review droid, then they might place droid inside of GitHub Actions and say to it, you should be code review agent.

13:52Eno Reyes:If you want a security droid, you can place it in your, you know, whatever system that you use for either pipelines or security review, and it becomes a security review agent. If you want to expand that and make it customized to your organization, you can introduce skills and custom sub agents and all these other things that basically allow it to conform to your organization standards. I've heard that Droid is like kind of almost like an engineering team that runs like in the background, like you can use it in that way. How do the agents then understand large, messy, real world code bases versus like, say, like another agentic tool and how they?

14:31Eno Reyes:I think that this is one of the most interesting questions because there are several things that go into being able to explore and understand a very large code base, especially for hard problems. I think that the three angles I would say is there's the environment. That matters a lot. Basically, what information does the droid have access to in order to understand? Two, there's the droid's inherent ability to explore and search. and three is how it handles long-term goal-directed behavior. That third is arguably the most important piece of research that, you know, as factories in the agent research lab and that's basically the most important piece of research is how do you do long-term goal-directed behavior?

15:12Eno Reyes:But I think on that first point of like the environment, one thing that I think makes droids really effective in larger codebases, enterprise environments is that it makes it very easy for you to understand is your environment actually agent ready? In other words, it will tell you, you know, is Droid operating in its best environment possible? We have tools like agent readiness that help you understand is the code, does the code base have agents.md files? Is it documented? Is there linter feedback that the Droid can use to verify its own work? And it will proactively seek those verifiers so that if it wants to explore a code base and see, hey, is this code base over here?

15:53Eno Reyes:sorry, is this like section of the code base doing this function? It will use, you know, grep and glob and bash and all the tools available on the computer, but it will also look for validation and verification. Can it, is there some sort of a mono repo structure that it can use? Are there tools built into the repo that help it search for information? And so from an environment perspective, the droids are very tuned for that from you know, a, from the perspective of, you the agent basically be able to run for that period of time. I think that there's a lot of work that we've done on things like context compaction or compression.

16:33Eno Reyes:There's a lot of work that we've done on things like prompt caching so that it's fast, right? If your agent takes two hours to search the code base, that's obviously not good. So keeping it really fast and keeping it effective over a long period of time, that's another major area that we focus on. And I think that for our users, they find that the more you invest in making your code base agent ready, the more tools like Droid work effectively. So you have some model flexibility built into here too, right? And how do you choose the right model for a specific agent with the way these work? Are you able to adjust them on the fly or are you able to uh you know i think of that whole idea of of like you know having a basic and a power model kind of paired together where for certain parts of tasks it's using this but maybe it recognizes when it's in over its head and needs needs some some extra goju's yeah totally and model being model agnostic is is actually very core to the the droid as a product experience, we believe that there is a lot of value in being able to use basically the frontier models the moment they come out across all of the different providers.

17:51Eno Reyes:We've been in a lockstep battle across, you know, five plus different foundation model providers for the last three years. And our view is that, you know, as a developer, you deserve choice. You should be able to pick any of these and have an extremely high quality experience. And so the Droid has a bunch of different customization options. One, it built in, we have the ability to route to the most capable models at any given moment. So you know, one click when you're a factory plan subscriber, you can just pick from Opus, Sonnet, GPT 5.2 codex, GPT 5.2. Now, there's also the option of bringing your own model and bringing your own keys.

18:32Eno Reyes:So you can choose any model. And so people can use really any inference endpoint that supports the three major inference endpoint standards. Right. Yeah. You can even use on your own device. Right. VLM or Olama. So so you can have a fully local coding agent experience with Droid if you have a powerful enough computer. Yeah. Yeah. Now you also can customize beyond just, you know, I'm using this one model in this one session and you can do things like I would like to plan. And we have something called spec mode, which is basically specification. And you can plan or specify with one model, but then execute with another.

19:14Eno Reyes:And so what that lets you do is it lets you say, I want to use an expensive or powerful model to do X task. And then a cheap model to actually execute because most of the efforts in the planning. And I think people take advantage of this most when they might have one model for code review. They might have one model for security review. Their daily driver is a third model. And then when they really need to pull out the juice, they'll use like the most expensive model. Right. So that flexibility is super important because most model providers basically give you one, maybe two options at most. You know, you're a research company and, you know, the agent is going in and it's deciding or it's figuring out, is this environment good for me?

19:52My question is, how does it know to do that? Is that like a system prompt? Is that something you trained into the harness? Like how, how does it know to, to look for those things? And I guess the second thing is, would you ever create your own models because you are doing research? Like, is that on your roadmap? Are you interested in that?

20:10Eno Reyes:On the, so I'll, I'll speak on the latter first, uh, in, in like, will we build our own models? Our view is that we're definitely not opposed to building models. I think that the challenge is you want to build, you basically want to fine tune or build models as basically as late as possible before it becomes important to, right? Right. That's sort of that's sort of our view. And what I mean by that is, you know, if if you had fine tuned GLM two or some model that that's earlier, you know, it gets blown away by the model providers at the next level. Right. And so you have three weeks away usually.

20:46Eno Reyes:Exactly. Exactly. You have, you probably have three weeks before there's a better model out and you have a month to two months before it's completely forgotten about. Right. So if your bet is we're going to beat the people with$50 billion data centers, I think that you have to have a very clear strategy. And, and, and, and I think that there are reasons to do this by the way, costs, right. If you know that you can execute on a specific task at the max that a human cares about and you just need to bring down cost, that's a great reason to fine tune a model. Right. Um, yeah. But at the end of the day, uh, I think that we've already seen this a bunch of, you know, software development organizations, uh, or, you know, software development, AI tooling, uh, they've all trained their own models and, you know, within two weeks to Gemini flash came out and it's like better, uh, or within three weeks later, 4.5 opus is now like not just better, but it's like two orders of magnitude better.

21:40Eno Reyes:Uh, and so I think that it's going to be a really challenging battle to fine tune or build your own models in the near future. But we're definitely not ruling it out. On the earlier point about being an agent research lab and sort of like how we tune the agent's behaviors and in particular for seeking verification or validation. There's a lot that you can do at the harness level to enable this, right? Some of it might be like context engineering. So adjusting not just like the base system prompt, but reminders that you can insert automatically or, you know, pre-built environmental reactions to tool calls.

22:17Eno Reyes:You know, the third time you write or edit a file without calling a linter, maybe you insert a reminder that say, hey, by the way, you should just check up to make sure this work is actually, you know, valid. You can seek these out in the background. And whenever you edit a file, provide like language server feedback, right? That helps the model understand, oh, I might be missing these things, right? So there's tons of things you can do within the harness that make the model more aware without needing to actually waste LLM calls. Then from a behavioral perspective, there's also a lot that goes into enabling the tooling to get easy answers from important questions.

22:55Eno Reyes:So like we have in our product, an API for agent readiness, which basically and a dashboard. So there's 150 plus signals that a code base could have, ranging from is a linter present to, you know, are tests available? Is there documentation across the code base? And this applies to mono repos, where it will segment out each project, or single code bases. So when you make it really easy for an agent to seek that information, like one API iCall to get the answer uh you you make it easier uh in general for the system to rely on that information versus if it needs to find that out from scratch every time you're going to waste your users tokens trying to like get the answer yeah i've i've lived that actually like working with an agent like in in my terminal i've i've been constantly reminding it like remember you should check all the scripts to make sure that you you know know all the dependencies of everything like I'm doing that.

23:51So like, yeah, use factory because it'll save you those tokens.

23:56Eno Reyes:Totally. And hooks are also great for that. Like, I think there's a pretty popular one right now called Ralph Wiggums, which is an odd name. And I always feel silly saying that it's from the Simpsons, but it's basically a hook that you run at the end when the agent loop stops and it sends a prompt back to the agent that basically says, you know, here's the step-by-step plan of what you were supposed to do, did you actually validate and complete the work that you're supposed to do? If not, restart, right? And so it's effectively a while loop in a while loop. I honestly think that the existence of plugins like that means the agent's harnesses aren't taking advantage of what they could be.

24:36Eno Reyes:But regardless, people find this out and then they build and they customize. And I think that's one of the coolest parts about building a really simple and modular tool is the community goes out and they find all these interesting ways to use the tool and they get a lot of value out of it. So let's talk about the blog that you just published called Evaluating Compression. Really, really awesome blog and research that you did. The hardest part about working on large code bases is context, right? So why is maintaining context across long agent sessions so difficult? And how did you attempt to solve this?

25:10Eno Reyes:Yeah, this is a really tricky problem because I think that, you know, we've actually published like a couple of different posts about just how hard this problem is ranging from, you know, there is tons of information and there's obviously context windows in all models, right? You have, let's say, 1 million tokens of context or 2 million tokens of context. There is cost questions involved, right? So most model providers actually increase the price that they'll charge you for beyond a certain level of tokens, right? And then there's like speed and quality. the classic like lost in the middle evaluation where people talked about as you increase the context that's utilized 1 million tokens of available context and that they'll let you send the API call does not mean that the LLM will actually be able to reason through all that information so if you think about it like what information is even in an LLM call to an agent you have the task description that the user gave you or like the user's messages you have tools which need to be available to both the agent and the system prompt, as well as the tools that they then call throughout and their responses.

26:25Eno Reyes:So one tool call to Bash that says, get me all syslogs might be 200 ,000 tokens, right? You have the developer persona. So all the information about their environment, their role, you know, is there, are we in a Git repo? Of course, code and all of the files that it's reading, markdown, code, Maybe it browsing, it browses the web and retrieves information from the web. You have historical context that might have occurred previously. And you have like all of the other sort of artifacts that might pull in from system reminders. So you basically, the LLM is seeing so much information about the task at hand in order to solve it.

27:06Eno Reyes:And so over the course of a very long conversation, you know, it builds up. And so in order to prevent this from crossing over a certain threshold, you have to figure out a way to compact or compress this information over time. And so what we did was we initially had a naive solution that just compresses it and summarizes it and says, keep going. But you quickly realize that that does not work. Um, and so then we started to try to figure out, you know, what are the individual dimensions that matter for a compression of a situation? Like, you, you know, it's going to lose some information. So what matters to preserve?

27:47Eno Reyes:How can we design a system that not only doesn't just summarize, but sort of provides a very high quality and active block of information that the agent can then resume its task with little to no issues. And so, you know, you can look at all these things like the accuracy, if whether or not the actual compressed artifact is accurate, you can look at its context awareness, like what information is or isn't included about what the what is currently happening, what has previously happened, you can look at the artifact trail, are the files and the logs and the critical, like, you know, singular information pieces present in the summarization.