Factory's Matan Grinberg: The Coming ‘Dark Factory’ Where Software Builds Itself

0:00Matan Grinberg:Bezos at Amazon, it's customer obsession. But in our mind, that's an input metric. You don't want to measure input metrics. It doesn't matter if you're customer obsessed. You could be customer obsessed and they file a restraining order against you because they don't like what it is that you're doing. Our job is to build something so good that our customers themselves become obsessed with us. That is our job. The analogy is, if you're a coach of a basketball team, you don't want to tell your players before they come out there like, hey guys, make sure to sweat. It's like, what? No, score points.

0:29Matan Grinberg:Like we need to score points and in doing so, yeah, you're probably gonna sweat. And I think similarly to create obsessed customers, you probably need to be really obsessed yourself with the customers. But the output is what matters.

0:57we're here in the studio with matan from factory this is our second time with the time thanks for having me you're in the small and elite group of second time training data attendees so thank you Oh, yeah. Matan is the co-founder and CEO of Factory, which makes droids, which are autonomous agents for the art of software development. Yes, indeed. And Matan, we're going to jump right in because I think you guys are a little bit of a dark horse candidate in this world of software development. It is a market that has absolutely taken off. There are folks like Cloud Code and Cognition and others who have a lead, but you guys are coming up strong.

1:33Talk about the competitive dynamics and what makes Factory special.

1:36Matan Grinberg:It's been a wild ride. we started factory three and a half years ago now. So in April of 2023, when the world and the enterprise in particular was barely ready for GitHub copilot, let alone fully autonomous agents. And so I think the first two years, it was kind of our journey in the desert is how I like to refer to it, because we were focused on fully autonomous agents, but engineers weren't ready, procurement teams at the enterprise weren't ready. And so I think, retrospectively, we really like honed our craft and learned a lot about how to build for developers in the enterprise. But you know, it took a lot of time to actually come around to when they were ready to receive it.

2:21Matan Grinberg:And so we're kind of now emerging much more and some of these other players like Anthropic or OpenAI who have a ton of distribution are going in and bringing their incredible tools like CloudCode or Codex, the thing that enterprises are really caring about that we have learned through those two years is they do not want anyone to kind of be their single point of failure. They do not want anyone to kind of control their fate. And so something that really matters is model independence. Everyone learned from cloud, where, you know, back in the cloud days, it was like AWS or Azure being like, hey, you know, come on in, sign this three-year contract.

3:00Matan Grinberg:It's going be so cheap we're going to subsidize it it'll be great and then a couple years later when it came time to renewal they would 10x the the the contract haha data gravity we got you now yeah we got you what are you going to do a two-year migration to go to someone else like no way everyone has scars from that now and so everyone knows look Claude Code is fantastic uh Codex from OpenAI is fantastic we cannot put our fate in any one of these model providers hands also like you just look at the risk profiles of the model labs versus the cloud providers. What's the last piece of drama that came out of a one of the cloud providers versus like the model labs, it seems like there's kind of always some sort of chaos of, you know, internal fighting or getting in spats with the government or, you know, any other entities.

3:44Matan Grinberg:And so if you're going to, you know, build this very important part of your business, you want to make sure that you're robust to any of these changes. And that's something that we've learned over those kind of initial two years is like developers really care about things being modular they want to know that they can customize it to what they want they want to know that if there's a new model that comes out that's faster or cheaper or more performant they can kind of hot swap it in and that's i think one of the biggest reasons why a lot of the largest enterprises are taking the momentum that they had from a codex or cloud code and then are carrying that into factory because they get that performance from these fantastic models but they do it without the vendor lock-in that, you know, the Model Labs directly provide.

4:26And if I'm the enterprise, I'm going to be like, wait a minute, am I now just getting locked into factory? What's the answer to that?

4:31Matan Grinberg:So it's a really good question because that is something that you might think of like, okay, wait, so we're just switching the lock-in point. All of the modularity that we build is such that if at some point you wanted to say, hey, you know what, factory is not staying at the frontier anymore, whether it's like the automations that you build or the skills registry that we help you create, the work that we've done stays in your code base. and any of the automations that we've created, the artifacts also live in your code base. In other words, there aren't really things that we're saying like our tribal knowledge about your org that we're keeping on our side and not giving to you.

5:05Matan Grinberg:And that's part of the relationship that we have with customers is like, we similarly want to make sure we're providing the best experience possible. If we help you arbitrage between different models to get cost optimization, we're giving you that optimization. We're not taking that away from you. And I think that's a really important part of the trust that we're building with these enterprises. You and I were talking probably a couple months ago at this point, and I was trying to give you credit for having the right vision for this market two, three years ago. And you responded with something along the lines of, thank you, but being two or three years early is the same as being wrong.

5:40Yes. Which I thought was a wonderful response in so many ways. Can you talk about those two years in the desert? How did it feel to have this vision that turned out to be right, that nobody appreciated for a year or two? Can you just talk about that journey and what it has done to the DNA of your company?

5:58Matan Grinberg:Yeah. I mean, in the moment, it's really, really difficult because I hadn't had a job before. I dropped out of my PhD to start this company. And over the course of those two years, convinced 20 of the smartest people that I've ever met to quit what it was that they were doing and join Factory and join us on this mission. Um, and these are people with families. These are people with kids who are like dedicating years of their lives to this problem and going, you know, customer after customer. And they like, they weren't ready for agents. They didn't get it. Also, the models weren't as performance, but I think a lot of it was behavioral.

6:36Matan Grinberg:And I mean, even just a fun anecdote of like giving developers an NPS survey. If you ever are giving a developer an NPS survey, they do not like whatever it is that you're giving it to them. because like developers they vote with their feet they are very clear what they like and what they don't like and if you're like hmm i wonder if they like it they definitely don't um and uh but during that time i think there were there was a lot that we were learning there was a lot that i myself was like i'd never had a job before enterprise sales is not something that comes obvious to to a physicist um but at the end of the day it doesn't it doesn't matter there's no you don't get any you know bonus points for being early because like who cares like there's no consolation prize it's either you do the thing or you don't do the thing and that's all that matters and for the team is really tough there were points where uh we ended up getting good at enterprise sales but the product still wasn't good and that's a very tricky position to be in because we ended up you know getting to a point where we were like just under two million in revenue and the product was not good and there was a point in time where we realized this because if you're really good at sales you can sign contracts that's like you can definitely do that but if you're doing that and the developers don't like your product it's like a ticking time bomb because it's eventually they're going to churn and it's going to be really really bad we realized this and we proactively gave all of those customers their money back and i remember that uh was one of the most difficult decisions to make because not only is there a you know a group of you know 20 people who are getting ridiculous offers from all the labs they have these huge you know financial incentives to go elsewhere there are all these other companies that are doing well and they decided to do this and then we're going to say oh yeah hey by the way that you know a little bit of revenue we managed to get we're actually going to give it back because we don't think product is making their developers happy we also had to make that decision you know we sold them on a good vision and convince them that, you know, this is the right team to work with and that we were going to deliver the solution for them.

8:39Matan Grinberg:But we realized that the way that we had sold them on it and the product that we were delivering was not up to snuff in a way that I don't think it would hold true to one of our operating principles. And one of our operating principles that I really like is create obsessed customers. Yeah. This kind of like flips over Bezos's thing where Bezos at Amazon, it's customer obsession. But in our mind, that's an input metric. And input metrics are like you don't want to measure input metrics it doesn't matter if you're customer obsessed like you could be customer obsessed and they file a restraining order against you because they don't like what it is that you're doing like our job is to build something so good that our customers themselves become obsessed with us that is our job it's like you know the analogy is if you're a coach of a basketball team you don't want to tell your players before they come out there like hey guys make sure to sweat it's like what like no like score points like we need to score points and in doing so yeah you're probably going to sweat and I think similarly to create obsessed customers you probably need to be really obsessed yourself with the customers but the output is what matters and I think that it coming back to this the product that we were delivering was not creating obsessed customers and we wanted to make sure like this was a group of the smartest people I've ever met we were getting there like we were getting a lot of intuition things were starting to come together internally.

9:56Matan Grinberg:Like we could see internally, we were starting to become a lot more agent native in how we were doing things. And the product was kind of scratching that itch. But we were kind of ahead of our customers. And we wanted to maintain trust with our customers, so that when it does hit, we can come back to them and say, Hey, guys, this is the real deal, I promise. And to build that credibility, we have to say, Hey, look, you know, even though you were maybe happy to continue, we're going to give you this back and say, three months from now, I think it'll be ready. Give us some time. And I promise we will knock your socks off.

10:27How did your customers react when you had that conversation?

10:30Matan Grinberg:Some of them were like, oh, great. Sounds good. Because I think it wasn't something that they were obsessed with. Some of them were a little bit confused. But I think generally, it's especially enterprises. They're not used to these things. A lot of times enterprise budget, once it's gone, it's gone and no one really cares. Yeah. And so some of them didn't even know if they had a mechanism by which to take back the money. um but you know it's a difficult thing to tell also like investors who believe in you like you know i remember having the conversation with sean um i think sean obviously he's stayed really close with the company so he was very like on the same page but it's kind of a scary thing to be like hey by the way you know remember all those updates and you're saying hey look the you know revenue is going up it's about to go down to zero um it was a scary thing uh and i think it was kind of a leap of faith of like we see the signal internally early of like this is the direction we need to go we need to kind of pivot the approach on the product but i remember that all hands where we told the whole team it was like oh my god that was like one of the worst months of my life like i was just because no like not everyone was going to say like what the hell is this what's going on but it's kind of the looks on their faces where they kind of go a little bit pale and they're like oh boy like is this just the early signs and we're about to sink completely um how'd you keep the team together through that i think honestly the only reason the team stayed together is we were so ruthless about hiring early on where it was like people that are genuinely really really obsessed with the mission which our mission is to bring autonomy to software engineering and like really really caring about that making sure everyone um was also like very clear feedback loops as to like this the fate is in our hands it's not like this is like oh something that I go do.

12:14Matan Grinberg:It's like we all have a part to play in making this work. And I think embracing how much it sucked was also, I think, something that was very valuable. Just being honest about it. Being super honest about like, yeah, this sucks. Like, oh, look, look at those competitors. The revenue is going up like crazy. Like, this is not good. Like, we are in a very bad position. Like, we just had to give back all of our revenue. Like, we need to really get our shit together. And in the moment, I think retrospectively, those are the moments where really the deepest bonds are made. Like, if you talk to people who are like athletes or even like academics or whatever whenever you're in the like stressful period whether it's like cramming before finals or you know in intense like you know we have some some rowers on our team and i think that's an example we always go to like that's a pure pain sport it's pain it's literally just there is one number that quantifies your performance it's just what is your time on your 2k are you timing that um but like embracing that is what creates those enduring bonds such that afterwards, like, we know what it's like to be at rock bottom, we know what it's like to lose.

13:16Matan Grinberg:We know what it's like. I mean, when we first started the company, our valuation was 5 million. Like a lot of our competitors, a lot of the companies out there these days, they don't know what it's like to not be a unicorn. That's like manifestly, that is what they are day one. Whereas like we have been there kind of in those dark moments and not a single person left. Yeah. That makes us so resilient and so strong that, you know, going forward, things are going a lot better now. But there are going to be really bad times. But we have that resiliency in our DNA that I'm not sure some of these other companies do.

13:48I love that. So talk to us about what changed. And I'm curious your comment from earlier that the models getting better is not the most important thing that happens. Because at least in my mind, the models getting better is the most important thing that happens. Yes. Help me understand. Yeah.

14:01Matan Grinberg:So a couple of things. So one is the interaction pattern that we were building for before was too ambitious. Like to your point, we were right in that what we were building for was fully autonomous agents, but it was two years too early, which makes it wrong. And fully autonomous agents require a complete change in behavior from the developer. And we were trying to do that out of the box before they were even using tools like CoPilot. It was just too much of a leap. It was too much of a step function jump. So it's an important day. September 26, 2025 was when we first put out basically the Droid CLI.

14:35Matan Grinberg:And the Droid CLI met developers where they were in a manner that previously these fully autonomous agents did not. And also its performance was like completely state of the art and it was model agnostic. So it could use every model that was out there. September 26 was also two years after we initially started. So the world had gotten much more used to using things like autocomplete. Like by late 2025, most engineers were using an autocomplete tool. And many were starting to, at the time, use like a chat interface to ask an agent to go do changes like wholesale. So like the more agentic interaction.

15:11Matan Grinberg:However, what we see is that like, if you go back now and use in this like agentic interaction, some of these older models, they're still good. So the biggest thing that changed was developers and in particular in the enterprise, like being open-minded to this new way of working. In particular, you know, developers, they've established their workflows over the last 30 years. They can be stubborn. A lot of them were like, no, no, no. Like my craft could never be done by an AI tool. So a lot of it was just like understanding how to work with these tools and having the willingness to go in and try and also the intuition about what are the guardrails that you need to provide in order for it to succeed.

15:51Matan Grinberg:And so I think it was a combination of both of these things. The model is getting better, so you need to do less in the way of providing guardrails, but also developers lowering their guard and being like, okay, you know what, let me go try and do these things. It's going to go do things I don't like. And then also, there's a certain degree to which when Andre Karpathy tweets about something, then every engineer suddenly is like, okay, you know, maybe this is true. And Andre started to tweet about these agentic work. early on he wasn't as open to it and then him being more open to it genuinely just changed some people's minds which is funny but that's some of the things that go into behavior changes like you hear it from people you trust you start seeing it you know from people within your organization who are maybe a little bit more agent native but that's that's kind of these things together is what what changed that and now we're all gonna be on slack we might we might be on we might be pushing the limits of slack which i think is going to be another interesting thing but yeah okay so So September 2025, you launched the Droid CLI.

16:47You said Frontier Performance Soda. What does that mean for you?

16:50Matan Grinberg:There's like the benchmarks, which have a very short half-life. Like anytime there's a good benchmark, it gets benchmarked within like three to six months. At the time, I think the one that we kind of championed when we launched and kind of it ended up becoming a pretty good benchmark was Terminal Bench. so prior to that the one that was kind of leading was sweet bench which was um kind of took some open source projects and some examples of issues that were then solved the problem with that was it was very focused on like python and like scripting or like individual file changes whereas terminal bench was more one it was in the terminal setting so it was things like scheduling runs and things that were not just like changing the code file but general software development tasks.

17:34Matan Grinberg:And that was something that we ended up, you know, having really frontier performance on. Now it's like, benchmarked to the extreme to where it's like, I think, you know, models that come out now are like 90 % on it. And I think there's a very short time horizon from putting out a good benchmark to then it being kind of in the training data. What goes into building a great and is it a great harness? And it seems like there's almost a lot of FUD in the ecosystem of my harness is better than your harness. And, you know, you need to own the model to have a good harness or actually you have a better harness if you don't own the model like yeah what's your mental model for for yeah you know benchmark maxing aside what keeps you at the frontier yeah so a couple so some general things that matter are um the way you do caching so you know cash tokens end up being like a tenth as expensive and so one big piece of performance for a given harness is what what is your like rate of of token caching um another example would be how do you perform while in compression or compaction?

18:32Matan Grinberg:So typically when you're dealing with a long session, you're going to exceed the context limit of the model itself. And so the harness will do some sort of, you know, summarization, compression, compaction, whatever you want to call it. And the way that you perform during that compaction is a big determining factor of how good your harness is. And tests that they do for that are like, you know, they call it needle in the haystack where you have some long thread and maybe there's one piece of information that's really important how often will your harness preserve that through compaction other examples are like tool use or how does it use the environment to validate whatever work that it's doing these are things that you can kind of have individual metrics on and that we kind of have our own internal benchmarks to measure how do the out of out of the box agents do versus how does factory perform i think one thing that um naively everyone believed initially was if you train the model and you build the harness, you're going to make them better together.

19:26Matan Grinberg:Yeah. And much to the chagrin of many of my friends at OpenAI and Anthropic, this is not true. If you build a harness that supports different models, that harness will be better. What's the, like my intuition would be model harness co-design makes you better. Yes. What's the intuition for why it's actually not? It's very analogous to the idea maybe like, I don't know 10 years ago of if you were to be like hey I want to train my personal AI back in like like ML days before like GPT-3 I want to train my personal AI I'm going to give it all of my data because I want it to know me turns out the answer was train it on the whole internet and it'll be so much better for you than if it were just trained on your data so there's a sort of analog that emerges where it's what data is to a model models are to a harness where the more models you expose to a harness, you avoid overfitting that harness to the nuances of that model in particular.

20:21Matan Grinberg:And there are certain intricacies about different models that you can learn from and then improve different models performance in your own harness. And this was why, for example, we kind of stopped doing it because Terminal Bench got so bench maxed. But initially, when like every new Opus or GPT model would come out, it would perform better on Terminal Bench in droid than it would in clod code or codex um which is why like and this is something that you know i think was somewhat frustrating to because i deal like from a lab perspective you ideally want it so that it's better together because then that means you have to use their harness and you can't use a different one but i think the reality is it it's uh you know having that multi-model harness ends up getting kind of frontier on on all those is there a good like example or illustration of that conceptually it makes sense is there like an easy way to Maybe maybe a good example of it is like if you're familiar with the different behaviors of Opus and GPT 5.6 right now.

21:19Matan Grinberg:I am he's okay opus tends to be I mean loosely loosely I mean to be fair honestly these days I'm not doing it as much either But I will say this loosely opus is kind of like that Super friendly colleague where you're like, hey, I want to go do these 20 tasks and they're like, okay, cool hey by the way five of those tasks I realized we didn't need to do it don't worry about it I got other these done did it this way tonight's not a good time let's pick it up in the morning yeah like let's go let's go get a beer afterwards and hang out whatever meanwhile like gbt 5.6 is like absolutely I will do every single one of those and nothing will stop me I'm not going to sleep until those it's like kind of very OCD and you know meticulous but sometimes you know you want one where it's like it actually realizes hey that list of 20 that you gave me actually here's a better way of doing it anyway.

22:06Matan Grinberg:You know, 5.6 is more methodical. If you build a harness for each of those, there are actually different things that that harness will then be good or bad at. So for example, one thing that, you know, typically agents will do is they'll, they'll have a to do list of like, if you have a task, it'll go and generate a to do list. And the Claude code harness can in some cases or and this is maybe less relevant now. But I think earlier this is a just a more illustrative example earlier it was really strict to make sure it would stick to the to-do list because the model itself would typically wander meanwhile codex wouldn't do that because the model itself was really really ocd about that but if you're a user you want to have the same experience regardless like you want to make sure if you switch to a different model you're not going to suddenly lose track of whatever things that you are working on and so there are certain things where like maybe in some cases you really want robust tool use and there are tools that you use to do these to-do lists you want really robust tool use and you want to make sure that no matter what if i'm a user i want to see my to-do list there like there were some cases where it would just like not have the to-do list and so that these are things that kind of improve the general performance and that the to-do list matters because you're doing some crazy migration and you don't have the to-do list and then you're in this long session where there's compaction that might get lost in the summarization.

23:23Matan Grinberg:And now you forgot what your seventh step was. And that could be one of the failure modes. That's kind of an example of how... That's a good example. Yeah. That's a great example. Okay, so we talked about one type of maxing, benchmark maxing. Let's talk about token maxing. Yes. Because it feels like the world has changed a lot. We've gone from token maxing to now cost rationalization. What does that mean for factory? Yeah. So maybe I'll lay this out just so we're all on the same page of the way that we see what's led us to this token maxing. So loosely, there was like this phase one where maybe phase zero was like no one believed in AI.

23:54Matan Grinberg:Then phase one, everyone believes in AI. And then boards were like, Mr. CEO, what are you doing about AI? What's your AI strategy? And Mr. CEO is like, shit, I don't know. Like, what's our AI strategy? CTO, like, make sure everyone goes and uses AI. And so then phase two is, you know, CTO is like, okay, shit, we got to make sure everyone uses AI. Let's start putting it in performance reviews. Let's make public like benchmark or public like rankings of who's using tokens the most because everyone's stubborn. No one wants to use this stuff. They're all skeptical. And then we enter phase three, which is everyone sees these ratings.

24:26Matan Grinberg:They see that it's part of their perf reviews and they're like, OK, I'm going to use AI for everything. And that's kind of phase three. It's this token maxing where people are using like Opus for literally everything. Like what's the weather NSF? Opus. Tell me. I don't know. Like there are banks that we are working with where they are spending literally hundreds of thousands of dollars a month on people asking things like literally what is the weather? Or like tell me about Python. Like trivial questions that you could Google, people are asking Opus. And the reality is this happened because we were so worried about adoption that we overcorrected and we're like adoption by any means necessary.

25:03Matan Grinberg:And I think that's actually it's like a decent approach. like it's probably faster to do that and then curb usage or make usage more responsible than it is to start limited and be like you know you can only use it for this thing because when you have people that are stubborn first you want to just prove that it works and then you can get kind of more mature about it where factory fits in i think one of the most important things that we do is that we have the factory router which allows you to dynamically route to different models based on the task that you're doing so you know if you're asking what the weather is you probably don't need the very frontier of human intelligence to answer that for you.

25:38Or you really do.

25:39Matan Grinberg:I mean, it depends. I don't know. It depends on what kind of answer you're looking for. You know, giving you like a full like down to the like molecular level of what's happening. But, you know, allowing that. But also more importantly, for every enterprise, something that no one's dealing with yet, but 12 months from now is going to be the case is not everyone needs the same tokens. having a blanket kind of token cap for every individual in some large bank let's say makes no sense so every cio is going to need to answer for every incremental token where do we put it and right now it is super not obvious how you would do that like right now we're saying oh you know the pms who are like vibe coding dashboards get the same token limits as like the engineers who are building like critical infrastructure that's probably not the best thing to do or similarly, you might be dealing with COBOL codebases where Opus is not the best model to use, but instead maybe some fine-tuned model on that codebase in particular.

26:36Matan Grinberg:The point of the router is that we can kind of accommodate these different constraints where maybe you say, you know what, this part of the org, they're just vibe coding. They can use Gemini Flash. This part of the org, they're doing COBOL. We fine-tuned this great model to work on COBOL. Let's route to that when we're working on that part of the codebase. Maybe this other part, we really care about reliability. So let's generate the code with open AI, test it with Anthropic, review it with like Gemini, things like that. And we can actually take in your routing procedure instructions in natural language.