Factory’s Matan Grinberg and Eno Reyes Unleash the Droids on Software Development

0:00I would have thought this would be different 13 months later, but this is still very much the case where Agent is synonymous with unreliable, stochastic, demoware, vaporware, and I think something very important for us is we want to build these systems that aren't just like cool examples of what is to come, but rather valuable today and not just valuable for like a hacker on a side project, but valuable to enterprise engineers today

0:43Hi and welcome to Training Data. We have with us today, Matan Grimberg and Eno Race, Founders of Factory. Factory is building autonomous software engineering agents or droids that can automate everything from the drudgery of maintaining your documentation to actually writing code for you. In doing so, they are building the ultimate compound lever. Last week, Factory also announced some impressive results on the key AI coding benchmark, Sweet Bench, being state of the art by a wide margin. Stay tuned at the end of the episode for more context on how they built it. We're here with Patan Grinberg and Eno Reyes, Founders of Factory.

1:20Gentlemen, thank you for joining us. Thank you so much for having us. Yeah, thanks for having us. Let's start with a little bit of personal background. And Matan maybe we'll start with you and then go to Eno. So Matan, one thing that I believe you and I share in common is an affinity for a well executed cold call. I know at least two cold calls that have had some bearing on your life. One we start with the one that you did as an undergrad at Princeton to somebody who my partner Sean McGuire tells me is quite a famous physicist. Can we start with that cold call? Yeah, absolutely. So while I was at Princeton, I was studying string theory.

2:00And the most famous string theorist happened to be working at the Institute for Advanced Study, which is an academic institution right next to Princeton University, but not technically affiliated with it. Part of the allure of going to the IAS is that you don't have to take on graduate students, much less undergrads. That said, there's a professor there, Juan Maldesana, who is by far the kind of leader of the string theory movement, and being a young, ambitious, undergrad, I decided might as well see if I could snag him as an advisor. So with some advice from some graduate student site, send him an email, ask if we could meet.

2:46And the thing about Juan is the way he works with people. You know, he'll take a meeting with anyone basically and we'll spend about two hours at the chalkboard with you. And in this two -hour chalkboard session, he'll subtly drop problem that you basically have 24 hours to solve, get back to him with the solution, and then you'll officially, you know, be a student of his. Luckily, I was warned about this, about this right of passage, so, you know, I was paying close attention to that any hints he was dropping. You have down the controls. Indeed, yes, yes. So found the problem, ended up spending, you know, basically the entire night working on it, and you know, luckily ended up having him as an advisor.

3:30We were able to then publish a paper together, which was very exciting. So, yeah. Typical undergrad experience. Yes, yes, exactly. So there's a second cold cold that I wanna ask you about, But before we get to that, why don't we go to, you know, you similarly went to Princeton. You have a CS degree from there. You spend some time as a machine learning engineer at Hugging Face, which is where we first intersected, spend some time at Microsoft. But like a lot of great founders, your story before then started with some humble beginnings. Could you say a word about the stuff that doesn't appear on LinkedIn that has helped to shape who you are today?

4:06Yeah, absolutely. And I think that, you know, my family on my dad's side came from Mexico in the late 60s to San Francisco. And my grandparents were both working for a bit, but when my dad was born, they started a Mexican restaurant in Los Althos. And that was in the 70s. They moved it to hate and coal in the 80s. were a very kind of like San Francisco immigrant story. They actually ended up leaving to Georgia where I grew up, but really I think it's the drive that they had to give my dad a successful life in America. And it was my dad and my mom that drove that same kind of mentality into me growing up.

4:51And I think it's really cool because this story is one that I think a lot of Americans share and something that makes it really exciting to be back in San Francisco and building something to potentially make the world a better place for everyone. Very cool. That is the dream. Natana want to get back to that other cold call because I think it leads directly into the the thunder or the forming of factory. So our partner, Sean McGuire, who I mentioned earlier, who I believe shares a similar academic background to your own received an email from you a year or so ago that led to a walk and very shortly thereafter factory was formed.

5:33So I'm curious, what caused you to call Sean McGuire? And this is less of a Sean McGuire question because we know plenty about Sean McGuire. This is more of like a, you're doing, you're on a very good path, you're doing really good research, you're on a on track to get a PhD in physics, and something inspired you to go in a different direction. And I'm curious, what was it that inspired you that led to that cold call and maybe tell us a quick story about what happened shortly thereafter. Yeah, absolutely. So, you know, I was like he said, I was doing my PhD at Berkeley. About a year in though I realized that I was only doing theoretical physics or instinct theory because it was hard and not because I actually loved it, which is obviously a bad reason, a bad reason to do anything.

6:17And, you know, I would have had such tunnel vision on this path that, you know, when I came to this realization, it was kind of earth shattering, and looked at the paths ahead of me, and there were basically three options that seemed realistic. And so it was either going into quantitative finance, going into big tech, or going into startups. And by this time, I'd already kind of switched my research at Berkeley from being purely physics to an ML and physics, and then slowly more ML, and then mostly AI. So it was kind of quickly, quickly cascading there. At the time, I think I saw a video of Sean speaking, I think over Zoom to some founders at Stanford or something.

6:58And I recognized his name from String Theory research because I had read his papers way back in the day. And it was particularly shocking to me because I'm not sure how much time you've spent with String Theoryists, but normally they're quite introverted. Not, you know, not, not, not the most social. Yeah, and so Sean is, you know, this very different example. And so to me, I kind of like, look, I looked at his background and it was just, it was just shocking to see someone who was like so deep and like a bonafide string theorist, then go in and like, you know, start his own companies, invest in some of the best companies, join Sequoia and be, you know, a partner there.

7:36And to me, it was just like, oh my God, this seems like someone who is of my kind of background of my nurturing, I guess. And so sent him an email and I was just like, Hey, you know, we both were string theorists. I don't want to do string theory anymore. I'm thinking about AI. We'd love to get your advice. Like you mentioned, that then turned into a walk. It actually was supposed to be a 30 minute walk. We ended up going from the Sequoia offices in Menlo Park all the way to Stanford and then back. And so it ended up been three hours. He missed a lot of his meetings that day. So it was it was pretty amusing and basically at the conclusion of the walk he So the one thing was for sure he was saying you must drop out of your PhD There's way too many exciting things to do and he kind of left me with the advice of You should either join Twitter right now because this is just after Elon took over And he was saying it's you know only the most badass people are going to join Twitter right now to you should join a company of mine as just like a general glue is what he said.

8:40This was foundry, by the way. Yeah. Or three, if there's some ideas that you've been thinking about, you should start a company. And I was like, you know, very grateful of all the time that he spent and we kind of left off there. Beautifully in parallel, Eno and I had just reconnected at a chain hackathon. And he was in Atlanta the weekend prior and he basically got back the next day. So that next day, you know, and I got coffee and I think we got coffee at noon and then basically every hour since then until now You know, and I've been working together talking constantly About co -generation and what became factory I guess originally there's no each other an undergrad We had like the maximal overlap of mutual friends without ever having had a what for one conversation Yeah, it's pretty funny.

9:35We were in eating clubs at the time opposite from each other and we had just so many mutual friends and It really wasn't until I moved to the Bay Area that we had a face -to -face combo and we It was a very fruitful conversation for sure. It was intellectual love at first sight. You could say Absolutely I love that and it's so serendipitous with the Lengchain connection How did you guys decide on, I'm curious, I mean, you're both brilliant. And I think for a lot of founders starting out in AI right now, a lot of them find it hard to resist the sirens call of training a foundation model. So like, how do you decide to build in the application layer?

10:16I'm curious. And then why, why software engineering? Yeah, so I think from my perspective, like going deep from academia, I think throughout throughout all the years of spending time on math and physics, the thing of beauty that I learned to be drawn to was things being fundamental. And spending time doing a research, it was so clear that code is fundamental to machine intelligence. And so I was just naturally attracted to the role that it plays there. And I think that kind of joined quite well with ENOs attraction to the space. You've referred to it a couple of times as a compound lever. Can you unpack that for us and let us know what that means?

11:02Yeah, I mean, so there's the famous Archimedes quote about software. Well, his quote is rather, if you have a large enough lever, you can move the world. And then I think that's been co -opted for software engineering, right? That software is a lever upon the world. And for us, we see AI and in particular, AI code generation as a lever on software, the impacts of that being compounding exponential. And sorry, I think I cut you off. I think you were maybe mentioning how you got to the founding inspiration for factory. Oh, yeah, absolutely. I mean, I think Mattan's story is really indicative of kind of the energy at the time.

11:43I was at hugging face working on training, optimizing, deploying LLMs for enterprise kind of use cases. I was actually working with Harrison on early like lane chain kind of integrations. And it was so clear that the work that was happening in open source was directionally moving towards modeling human cognition with LLMs where the LLM was just one piece of the system. The idea of chains and I think Harrison calls them cognitive architectures or the Langshane folks call it that and seeing that happening and seeing that within the code gen space, the most advanced players were basically looking at autocomplete.

12:25It felt like there was a huge opportunity to take that to the next step and take some of those lessons that were happening both in the fringe research and open source communities and applying them towards kind of massive organization. I realize we haven't said explicitly yet what is factory. So, Matan, what is factory? And then maybe what are a couple of the key decisions that you've made about the way factory is built? And you know, for example, one of them is to start by benefiting from all the ongoing improvements in the foundation model layer. You know, one of them might be the product itself, But can you just say what is factory and what are some of the kind of key decisions you've made that have shaped factory today?

13:08Yeah, absolutely So factory is a cutting edge AI start up Our mission is to bring autonomy to software engineering What that means more concretely we are automating tasks in the software development life cycle And in particular tasks like code review documentation testing debugging refactoring And as I list these off, you'll hear quickly that these are the tasks that engineers don't particularly enjoy doing. And that's very much intentional, right? Like, obviously we are doing code generation and that's really important. But I think an equally important thing too, you know, generating some inspirational and forward -looking demos.

13:55It's also important to understand what engineers are actually spending their time on. And in most organizations, it's not actually fun development work. In most organizations, they're spending a lot of their time on things like review and testing and documentation. Normally, they'll do these things way too late and then they're suffering because they're missing deadlines, right? And so our approach is we want these tools to be useful in the enterprise. And so to do that, we need to kind of meet engineers where they are with the tasks that they are very eager to automate away. We call these autonomous systems droids, and like Eno was alluding to earlier, these are kind of, there's a droid for each category of task, and in this kind of a paradigm where we want to frame these problems as games, it's very convenient that software development has a clearly defined software development lifecycle.

14:52And so for each kind of category of task or each step in the software development life cycle, we have a corresponding droid. So that's kind of a kind of a first pass there. I guess there were I think there was a second part of your question that I missed. Well, we'll get into the rest of it. Where did the name droid come from? It's pretty catchy name. It's like very memorable and distinct to factory. Where'd that come from? Yeah, yeah, so I mean keep in mind, you know when factory started this was like you mentioned about a year and a month ago and And, you know, I actually, I would have thought this would be different 13 months later, but this is still very much the case where agent is synonymous with unreliable, stochastic, demoware, vaporware.

15:38And I think something very important for us is we want to build these systems that aren't just like cool examples of what is to come, but rather valuable to date. And not just valuable for like a hacker on a side project, but valuable to enterprise engineers to date. We felt very strongly that agents just doesn't really capture what we're trying to deliver. And so, fun fact, we were originally incorporated as the San Francisco Droid company, but upon legal advice and given, I guess, the eagerness with which Lucasfilm pursues its trademarks, we changed our name to factory. Fair enough. So is it fair to say then that a droid is sort of like a job specific autonomous agent that actually works?

16:25Is that a reasonable way to think about it? Yeah, okay. It's okay You just said the words cognitive architecture and I know my partner Sony long well enough to know that this is her love language So I'm sure that Sony is mine just lit up with a whole bunch of questions for you So I don't want to get in the way. So yeah, have at it We just had Harrison on the podcast who talked about custom programming architectures as well I guess what are you doing on that front and how do your implementation stuff tell with the multi -joyed strategy that you're taking? Yeah, absolutely. I mean, it's a great question and I think the the way that we think about reasoning and cognition within the the standpoint of these systems there are clearly huge innovations happening on both layers, the foundation model layer, as well as on the kind of orchestration or application layer.

17:20The way that you can kind of think of our technical approach on this is that, you know, traditionally labs like DeepMind and kind of some of these these orbs that are really focused on solving problems that you can model like a game where you have rules and an environment and feedback You can build out systems which model the reasoning of humans and even outperform them. They did this with the alpha series of models, protein folding, go, code. And for us, most of the reasoning work we do is similarly focused on kind of inference time reasoning. Search through kind of decisions and what we kind of think of as, you know, maybe it's something of intuition.

18:05maybe it's something of planning, but we aren't training foundation models yet. And I think a lot of the innovation that's gonna happen at the foundation model layer will be things like latency and context window and kind of performance on some subset of tasks, but any time that you need action and environmental feedback and kind of long -term planning, It's going to be really difficult to build a single foundation model that does that. And I think it's really the application there where those types of innovations are going to happen. Yeah. I thought the Princeton, Sway agent paper that came out last week or so was really interesting as an example of that.

18:51I've like, you can get incredible, authentic reasoning performance on code tasks from small open source models. I thought that was really nice. I've proved point to what you're saying. We love the whole team that put that together and the SWE Benchwork. I think it is a popular benchmark in the space. I think it's clear that a lot of the efforts towards building these systems relies on not just like any one benchmark or EVAL or set of tasks, but rather collaboration across a bunch of different areas, whether it's the model layer, whether it's the tasks themselves, it's how what data are you using to evaluate and ultimately like the overall architecture.

19:33And yeah, they're a really great team where we're super pumped to see that work. Okay, last question on this and then I will pause myself. Any favorite cognitive architectures? Like is it the tree of thought stuff, chain of thought stuff? Any favorite cognitive architectures that you think are especially promising or fruitful in your field? Yeah, I think that's a great question. I mean, I think kind of what I alluded to previously when you have like the almost like the game like problem space where there are kind of simulatable Analyzeable and optimizable Boundaries then that means that you can search through those decisions And there's a bunch of techniques like Monte Carlo tree search language agent research that people have talked about in research papers that I think are interesting approaches here.

20:27I think that in my mind, there isn't a singular cognitive architecture that makes sense for all tasks. And a lot of the benefit of breaking down the software development life cycle into kind of semantically meaningful segments is that developers, when they have these workflows that move from one step to the next, they've kind of defined the boundaries of the game, so to speak. And so a lot of the work we do is figuring out which cognitive architecture, what design makes sense for a given task. You're reminding me of the rich Sutton, but they're less in search and learning or the two techniques at scale.

21:09Yeah, absolutely. And I think you definitely need both. And then you know, you were talking about this a bit, how the sort of the reasoning layer on top of the foundation model, is really the focus for a lot of the fundamental research and a lot of the fundamental work that you guys are doing. But, Tom, you had a line a couple of months ago when we were talking that was, and hopefully this doesn't come across as starkey because it's not nintu, but it was something to the effect of, there are eight interesting engineers that open AI working on my margins for me. Can you say a word about that? Cause I thought that was, first, I was incredibly well put, and then second, pretty good insight in terms of how you're building the business and really benefiting from the work of the foundation models.

21:52Can you just say a couple words about that? Yeah, absolutely. So, you know, there are a lot of companies, a lot of startups that are, you know, pursuing training foundational models or fine tuning models. Then there are a lot of huge research labs like OpenAI and Anthropic, who are also putting a ton of resources behind making these foundational models, better, cheaper, faster. And from our perspective, we don't want to run a race that's not suited to our abilities. We don't want to fight a battle that we know we won't win. Training foundational models, we are not going to win that battle. And similarly, I also don't think it's a particularly unique battle at this point.

22:34I think these companies were incredibly unique and innovative, clearly based on what they're delivering. But now, I think the stages set in terms of training foundational models. And I think similarly with a lot of the infrastructure for fine tuning and that sort of thing, what has not really come to fruition yet is actually making products with AI that people are using. There's so much to talk about. All these foundational models, all this infrastructure, and there's still very few real products that use this AI. In the analogy that VCs like to talk about a lot, We have a ton of picks and shovels and no one's actually going for gold And so the thesis behind how we're building this company is let's first you know use these beautiful black boxes that Open AI and Thropic and Google are spending billions of dollars and you know hundreds of engineers to make Let's use these black boxes and build a product that people are actually using and once we do that Then we can earn the right to do the fancier things like fine tuning and training If you're unable to build a product that people are actually using with these incredible models, then chances are fine tuning and training will not save you, and it's probably just not a good product.

23:46And so that's kind of the approach that we're taking there. And so, you know, we do get a lot of improvements when new models come out. But yeah, we are very much grateful for the work that's being done at these cutting -edge research labs. I want to, you've seen a lot about how what you're doing is kind of like making AI immediately practical for engineers in like an enterprise setting. And so I went through another, I think it's a Moton quotes, and I'm not sure if you were quoting somebody else, but you said, you were talking to us last time, you said, if Jeff Dean shows up at your office and he doesn't understand your code base, he won't be productive.

24:21And, and, and, and, in fact, that for us, like what does it take to kind of make a coding agent that's not just good for anybody that boots up a computer, but somebody that's a full -time engineer at a real software company. Yeah, totally. And yeah, so the analogy here is that, you know, Jeff Dean is the analog of a really, really good foundational model. Let's say like GPT -6 with incredible reasoning, right? But if it comes into your engineering organization with all your nuances and all your engineering best practices, just having that good inference and good reasoning is not enough to actually contribute and automate these tasks reliably.

24:59Some given isolated tasks, sure, you can solve. Like give it some like lead code problem and this give Jeff Dean a lead code problem, I'm sure he will solve it. But if you have some, you know, 20 -year -old legacy code base, some part of it is dead code, the other part of it, the last person who was contributing to it just retired, and so no one else knows what's going on there, you need deep understanding of the engineering system. Not just the code base, but why you made certain decisions, how things are being communicated, what top of mind priorities are for the engineering organization. And it's kind of these less sexy but incredibly important details that were really focused on in order to deliver this to the enterprise.

25:40What about the, I think a lot of these AI code, coding companies are kind of focused on the individual developers productivity. How do you think about the individual level optimization versus maybe the system -wide optimization? I think the important thing to think about with respect to the whole org is when a VP of engineering comes into the room, they're not really focused on whether or not an individual completed like one task, an hour faster. They're concerned about how many tasks are being completed, an aggregate metrics of speed. But if that person completed that task an hour faster, but it's 40 % worst code, right?

26:26It's churny code where people are going to rewrite it on top of it. Or that person took that task and, you know, they did it an hour, but it took them four hours to plan that. And they were blocking five other engineers. And so when you start to actually add the nuance of what does it mean to be successful, measuring an engineering org, you start to bump into a lot of challenges with understanding what needs to be improved and what is a bottleneck and what is just kind of a secondary metric. I think a lot of the initial attempts at making AI coding tools are really focused on first order effects.

27:03How quickly is somebody tabbing to autocomplete statement or how quickly is somebody completing an individual task? But I think that at factory, a lot of what we're trying to do is understand from an engineering leaders perspective, how are you measuring performance? And what are the metrics that you look at to understand, hey, we're doing really well as an org or hey, we need to make improvements and targeting those. And I think metrics like code churn end -to -end open to merge time, you know, time to first answer within the end org. where all of these things are much more impactful to an organization's speed of shipping code.

27:43And so that's kind of how we think about it. I think this really ties into what ENO is just saying quite well, which is, you know, the clearly, we were talking about products earlier as well, like clearly the AI product that has penetrated the enterprise the most is Copilot, right? Unfortunately, with a tool like Copilot, the things that are kind of, the metrics that are really held up as success or things like autocomplete acceptance rate. And the problem is exactly to your point, if you're a CTO or a VP of engineering, how do you then go to the executive team and say, hey, look, our autocomplete acceptance rate is this high?

28:18They don't know what that means. They don't understand how that translates into business objectives, right? And also, you know, Eno was alluding to this. There's kind of a hidden danger to some of these autocomplete tools, which is orgs that use tools like this end up increasing their code churn by anywhere from 20 to 40%. There's some studies that look into this. There's some problems with these studies, but, you know, directionally, what's clear is that, as the percentage of AI generated code increases, code churn, if you're not, you know, doing anything different in your review process, code churn is gonna go up.

28:54And so, our reason for focusing on org -wide metrics is that it kind of divides out all of these concerns. If we look at things like, like how fast are you completing your cycles? What is your code churn across the org or across these different repos? That divides out these kind of like smaller, like intermediate metrics and gives you a sense of, hey, we are shipping faster and we're churning less code. So that's really how we talk about this with these engineering leaders. At the end of the day, the three kind of main axes we look at is saving engineering time, increasing speed and improving code quality.