How to see in the dark factory | LaunchDarkly's Cameron Etezadi

23:17I had a team of mine that got together. They had an 18 month backlog of items. And you can say, well, you know, why would you keep something on a backlog for 18 months? Because it provides a high watermark. What is better than that to go on the backlog versus be just rejected out of hand? they cleared a third of it in a day yeah so the backlogs no longer become important what becomes important is the roi and so we look at it and say is it worth investing the time and the spend i have a um i have another project here that we rewrote oldest gnarliest nastiest part of our code base using, using Claude.

24:01Uh, we did it. I would have sold it to the board as a one year project with a team of engineers, eight engineers to do it. We all know that's a lie because that's all, that's really a two year project. If we ever actually got it done, I've been through a few of these in my life. I'm sure you have to. And so you take, take the cost of eight engineers in Silicon Valley, run that for two years. You're looking, let's just say conservatively 5 million spend. I had two engineers, a lot of tokens, and they did it in one quarter. So not only did I do that for about a 13th, 14th of the cost, so 14x cheaper, basically, but I also managed, and I spent a lot of tokens to do it.

24:48I spent about the equivalent of their salary and tokens to do it for that period of time. But now I have basically two years plus, a little less, a year and three quarters to get ROI from that project in the marketplace that I also would not have had. So the real thing is I did it 20x cheaper because I'm now making money off of it. So it forces us to become, to go from theoreticians who thought it was very expensive to build software, so we had to get it right when we put it out there to judge consumer response to it, into experimentalists where we just go, I don't want to say YOLO it, but YOLO it into production, test it, get immediate feedback, and immediately make that change.

25:29And so you put a system like LaunchDarkly where we have experimentation and observability, you can all of a sudden rapidly run these experiments, learn very quickly, get your product market fit, and deliver extremely compelling experiences to your customer base in a lot less time. But you have to be data-driven to do it. The last bit of this, I know this is a long answer, but the last bit of this is, is, you know, 20 years ago, I worked at Amazon as an early, you know, as an engineer, there are 1200 engineers in the whole company at that point. And it was a very exciting place to be. And the concept of two pizza teams had already been thrown around at that point in time.

26:06I don't think we have two pizza teams anymore. I think we have two slice teams. where you can feed the entire engineering team that you need to get the job done with a bunch of agents with two slices of pizza. But that forces every engineer to think like a frontline manager and direct work and not actually be the hands-on grind of, you know, code monkey is what we used to call ourselves in the old days, right? We're no longer hiring code monkeys. we're hiring frontline managers at every level. It's just you're managing agents. The great part of that, as somebody who's been in this industry, you don't have to have any EQ to manage agents.

26:47They're going to look right at you and go, you're fabulous. The smartest person I've ever seen. What a great idea you've got.

26:54Cameron Etezadi:They're going to tell you that no matter how much you abuse them. I love the man who's going to come on the show and say you don't need an EQ to work with agents is also not going to give the agents pizza. Wow, what a man I'm dealing with right now. So I completely resonate with what you're talking about. It has completely changed the math of what people should focus on. And it's changed the economics of what I can build and how fast I can deliver it. You're right that not only could you deliver faster and get the market value faster, but you did it earlier when tokens were cheaper. So then you saved even more than the 20x that you're thinking of.

27:28Cameron Etezadi:And when all of it is going to, you know, when all of the tokens fall, so to speak. And another part of this too that really I loved is, like I said, I've been asking this question of a lot of folks who come on this show who are in your kind of similar position of how do they inspire the change and how did that change take shape within their organization? You fall within the category of what I would call the Gordian knot leader, where you give them the impossible challenge, something that they just cannot solve with what they have available today, or is like an unattainably big goal. You set it up on a high shelf and you're like, get something out of that cookie jar, right?

28:04Cameron Etezadi:And so in that kind of inspires an extraordinary creative new usage of the tools. People find shortcuts and you realize that that road you were traveling is no longer one you need to trot on anymore. And those are the kinds of transformations that I think get a lot of gain really quickly. And other than being obviously putting everything together into an internal ecosystem. So folks are sharing skills and MCP servers, which are the new UX for how all the departments talk to each other. The sooner those get centralized, the better. And so like it's all a strategy in terms of aligning those resources so that you can think like, oh, what kind of software can I deliver?

28:46Cameron Etezadi:because I have all these parts kind of like lined up now from how we're thinking about the true cost of creating software. Because, you know, I'm just going to wear one slice of pizza and a few engineers are going to eat it and none for the agents, apparently. So, yeah, well, it's good that they don't eat pizza, but they do seem to take up more energy than a pizza oven. So that is true. So they are getting theirs. You said something there I just wanted to pick up on that was, I think, very, very smart. You said the MCP servers were the new UX between departments and teams. That builds on something that the best companies have been doing now for the last five, 10 years, which is they've gotten away from data lakes.

29:27They've gotten away from data warehouses, obviously. Many of them built data swamps because that's what the lakes turned into. But the best ones built data fabrics where data was vended internally between teams and data was thought of as a product. Those companies, those people who thought that way are light years ahead because sticking an MCP server on top of that is a much easier problem. And that's where the organizations get that 20x return that you were talking about, that speed.

29:57Cameron Etezadi:Exactly. We got to find all those fuzzy context areas that make our job really hard as engineers or as, you know, you called them, we used to be like the code monkeys, you know. But what really makes our job hard now is all of the messy context for meetings that we're dragged into and the focus time that we lose and the things that pile up on Asana and in our JIRA tickets and the half-completed PR reviews. And so, like, getting all of that into one space and MCP has been a great kind of, like, it solves for distribution. Obviously, it helps with the uniformity. If you have the composing layers that you need within it, you can get a lot of internal momentum really quickly.

30:33Cameron Etezadi:And so like, that's like a baseline change that happens between teams. Now you get this wonderful MCP interchange between departments and behind those departments, the software and the technology that they're picking up and they're integrating into their technology and on your own engineering team, the software that you're buying or thinking about integrating or using to deliver your service to customers, like all of those now, every line item. And I know this is something that you're feeling as a CTO becomes this build versus buy discussion of why did we buy this? Do we need to buy this? And every future thing that you pick up becomes, is it cheaper to buy, build this myself with tokens?

31:12Cameron Etezadi:And that then kicks off, obviously, a series of questions about like, is this worth my time? And where should I focus? And how do you think about that as a leader right now? It's very salient. I think it's salient for us as an industry. We've talked about the Sassiker or the Sasspocalypse. The Sasspocalypse. We're covering that a lot here. Yeah, I like Sassiker, but it just feels more brutal to me. I think it's because the industry got painted with a broad brush because of this problem without doing the deeper dive. And we're seeing that come back out. Valuations are sort of starting to fix themselves where there are companies that deeply got beaten up in there and deservedly so.

31:53There are companies that are in the middle that are tailwinds from this. And there are companies that, of course, frontier model builders, those type of things. I think the build versus buy decision calculus has definitely changed, but a lot of folks are still approaching it very naively, ignoring the operational costs of this. So, you know, I'll pick on something like LaunchDarkly. If you ask an agent to do feature flags that are super, it's super simple, right? Agent will put them in for you. You have a config file. Great. I have to redeploy a config file every time I want to launch it. It takes time because the physical speed of light network bandwidth problem, you can't ship instantly.

32:32You can't roll something out in 200 milliseconds. It's physically impossible to do it, but you can implement the basic functionality. But when you start to look at, I'm running a worldwide consumer brand that needs to do this in 200 milliseconds when something goes wrong before the damage gets excessive, then you start to see the value of the companies that have infrastructure and economies of scale to provide this. And those don't go away with AI. Those, if anything, are multiplied. The second part of this is while the agents are very good, they still have context windows that are better spent on the things that matter.

33:09Your code, your project, your business, and the things that aren't fundamental to that where somebody else can do it with the economies of scale still have tremendous value in the space. And I think that will hold true for a long time. I can't for the cost of what the average customer pays us, even put an offshore engineer who can keep it up with the same level of reliability and definitely not at the speed that we deliver. And so there's tremendous value to me in that. I think, I think though, as a CTO, you know, I am looking for things. Are you just a tool that visualizes something? You know, there are a lot of those in the marketplace.

33:45I don't need a visualizer. Are you just, are you a CRUD service that has my data that you're storing in a CRUD database because I track, you know, a project from A to B to C, right? There's not a lot of value in that because it's small scale for me. It's, it's very simple to implement. It's very easy to replace. And so we're looking, I mean, we look ourselves at vendor consolidation as a very real thing. Of course, the money still gets spent. We're all just blowing it on tokens, right?

34:16Cameron Etezadi:Right. That's where it's all going. But we are consolidating a lot of things behind the scenes where this is duplicative with something else except for this one feature. Well, I can build that one feature and then use the vendor that provides most of it for us. The last bit of this is as I look at a vendor, the number of people who come and want to demo a UI to me, I don't care anymore. Like none of us should care about the human UX other than the first setup and the help file because the MCP server and good sets of documentation about how it's supposed to behave. They all just get fed into my coding LLM.

34:57I don't, I don't spend, you know, three weeks learning somebody else's SDK anymore. I feed the, I put the docs, docs and cloud code and say, here's the docs. Here's the other side of that. Connect the two, please. And I have an answer in 15 minutes, hopefully. Hopefully I can one shot it. If not, okay, I'll iterate a few times.

35:18Cameron Etezadi:you're smart to call out how you should spend your tokens on things that are providing value to you and your customers that the economies of scale favor you to do don't fight the system that you're that you're in in this regard and there are definitely going to be entities and organizations and technologies that are better aligned to serve that maybe particularly niche for you kind of business need way better way more efficiently than you can build it because It's just building, it's one side of it. Then you also have to maintain it. We all hate the cost of keeping the lights on. Why would you want to magnify that cost for stuff that is falling so outside of your domain and what matters to your customers and what makes your engineering renowned?

36:02Cameron Etezadi:It just becomes a distraction over time. Don't bet on distractions because those distractions become more expensive as inference becomes harder to get. And so it really kind of influences that decision. Like it's kind of like an external pressure as well. It's the same. It's, I mean, the business fundamentals haven't changed. Focus on your core business, focus on the outcomes. We're all outcome focused now, which I don't think, I don't think everybody was in the past. We've all had to become outcome focused, which means we're all now goal seeking because that's how the tools function. They're goal seekers at the end of the day.

36:38And we all need to do our part to, for whatever business we're in to get to that outcome. the tooling behind the scenes will fall into place.

36:47Cameron Etezadi:So what are some of the problems you're thinking of solving in the future? Maybe that your customers or users aren't even thinking about solving yet because you are on that forefront as the domain expert there. Yeah, as I mentioned earlier, the two things that really stand out to me are the control of the indeterminate or the probabilistic runtimes where that agent control at runtime is important. And it's not the agent framework. There are a lot of really good ones out there. Anthropic has theirs. Amazon has AgentCore themselves, which is excellent. There's some open source ones out there. That's not so important as corralling the behavior to make sure that what's going on there manages to match what you want from your own strategy to achieve.

37:34So we are determined to help customers through that and help them optimize their environments for that. The second bit of that is the dark factory. Regardless of what your dev tool chain is, we all have our favorites, whether you're GitHub or GitLab or Mercurial or whatever you like for source code, whether you're CircleCI or ArgoCD or whatever you like for a build system, GitHub actions, whatever pieces you have along the way, the paradigm of a dark factory still applies. You can look back to Henry Ford, for instance, getting away from everybody building a car single piece by single piece by single piece and changing it into, we have the steering wheel expert, we have the tire expert, we have the welder.

38:22Getting into that assembly line factory for producing factory, that assembly line looks different for every manufacturer. But the fundamentals of mass production are universal. And we aim to be the best provider of the instrumentation to help folks let their agents and let their agentic workflows run through the path of a dark factory. If you speed up one part of the engineering lifecycle, the bottleneck just moves somewhere else. Writing code used to be the slow part. Now it's the pileup of pull requests waiting on review and the testing that has to clear before anything ships. AI made your developers write faster, but turning that into faster delivery is the harder problem.

39:06LinearB is the engineering productivity platform that reads the signals in your Git workflow to show exactly where AI velocity is getting stuck. And then it automates the fix so that faster code means faster delivery. See where your delivery slows down at linearb.io.

39:22Cameron Etezadi:That's such a fascinating way to envision it as well because, you know, in the factory space, in the robotic space, that was the problem that they were fighting. the whole time that we got to live in our nice, like, you know, we're in zeros and ones of space as computer scientists. And if we want to engineer something new, it's ultimately just the same light on our, you know, some rocks that we tricked into thinking, right? But the people that are making factories, like they had to like, they got to lay down the conveyor belts. They got to do all their sorts of stuff. And when that next iPhone comes out next year, they got to tear down the whole thing and respec it and build it back out and figure out where all the people and parts go.

39:58Cameron Etezadi:And you're right that like, Like the science of that mass production doesn't change at scale. You get these incredible business scientist folks. Like you go to a factory in Japan and like on the Ford or like on the Toyota assembly lines and they have the circles that are drawn on the ground. And the person is supposed to stand there for like, you know, an hour. And then someone comes back and says, like, what did you notice that was wrong? And if they said nothing, they stand there another hour. So like that world that they lived in allowed them to produce at scale, created the physical world that we live in.

40:33Cameron Etezadi:And now because all of we used to be that, you know, Ford assembly line where we're all going to hammer the one car together and build it. And then voila, here's our product. We're taking it to the market. Now you're so right. We have to think about how do we compartmentalize it, break it down into this virtual assembly line. We have to think of it as like a factory where stuff is coming in and then coming out at one end in a finished route. And as much of that in there is automated and has a clear process. And when it's at this station, we know exactly what station it's going to next. And I think that's going to be a big fight for folks that have really enjoyed living in the wild, wild west of software engineering.

41:16Cameron Etezadi:So I think that's a big challenge to get people on board with. I agree wholeheartedly. I think, too, we get to be the digital end-on cord, to use your Toyota metaphor, which I love. The end-on cord is one of my favorite pastimes, as is Kanban, and being able to look at these things. I also think, as we come back, the dark factory is real. When I go buy a car or whatever, I don't really think about how it's made. Now, I bought a car and I got from the manufacturer, this is some years ago, the manufacturer sent me a video of it going down the assembly line, like my car in particular, which was really cool.

41:59Cameron Etezadi:As a nerd, like it was super exciting to me, like not going to lie on that one. But I don't really think about it, to be honest. I don't think about there are 17 steps in the paint-by process. It doesn't matter to me. And I want software for people to be like that too. I want to provide the systems that make it so ideas literally materialize. And when they materialize, they're robust, reliable, scalable. At the end of the day, that's what business is all about. It's about solving a problem that exists for people in a way that they wouldn't have thought of themselves or couldn't do at cost themselves.

42:35And that's how you create value. That's what we believe we do for companies. And our customer base seems to back it up. And I think every business is in that space.

42:46Cameron Etezadi:In that video that they sent you too, it's as fun as it is, it's like a token and a memorabilia of the thing you bought from them and oh, I'm one of their customers, right? If you think about it, it's also a flex because their engineering system is so well harmonized and the end and completely instrumented in a physical space that they're like, all right, we can't at this point, we're just champagne problems. Can we get a video? every individual customer's car getting made and then create a marketing campaign to send it to them. And the answer ended up being yes. And that's like a perfect merge of an executive and a marketing and an engineering all on a very high level executing a goal that to you is just like, oh, it's this fun keepsake of when I bought my car, but it actually sings to, it's a testament to the system that they've all built together.

43:38The funny part about that, this sort of follow on that I wasn't going to share, but I'm going to anyway. Now, three years later, I got a new car. Same. In fact, same company, same model, just three years newer. I was leasing them at the time and, and the new, the newer model did not come with the video and I was crushed. Crushed.

44:00Cameron Etezadi:See, now they've set these unrealistic expectations and you see they are dealing in the physical factory world. So maybe when they had to tear the factory down to make that car model factory for three models later. They just couldn't get the camera up on the assembly. I have no idea what went wrong. I wasn't, I wasn't gonna, I still bought the car. The video was not the make or break on that one. All right. Well, on unnamed video car executives, if you're listening to this and you've scrapped a video production system recently that goes to your customers, maybe revisit it because it's a special, you know, cherry on top.

44:33Cameron Etezadi:And, you know, Cameron, this has been such a really fun conversation. It's been insightful for me and really great to get kind of like in your head about how you're thinking about these problems. I think the space in which like LaunchDarkly is working in order to keep our world from falling apart at the seams is so critical. And it's great to hear and see how forward thinking that your team is. And like if folks want to get more engaged with what y 'all are building, learn more about how you are putting guardrails on the probabilistic world that we live in now, Where can folks go to learn about the latest and greatest?

45:06Well, of course, please come to our website, launchdarkly.com. And we're going to host a seminar in just a couple of weeks with replays available on our agent control platform. We called it AI configs internally originally.

45:23Cameron Etezadi:So agent control platform. Those pesky names, they really stick. Oh, don't let me leak the code names or I'll get sued for something, I'm sure. But the agent control framework has a wonderful set of demos. I've got a great talk coming up that'll be broadcast with Edith Harbaugh, our CEO and founder on this, along with Mark Polis, who's our head of AI. And the three of us discuss what it does with a great demo. And I'm super excited about it. I'm going to share the last little bit as we had a demo internally where we taught it to play SimAnt, which we all remember from the late 90s. I love SimAnt.

46:01And so we built agent fleets that could play the game and we used our control platform to optimize for the right outcome. And it was it was a super cool internal demo with a game that's as near and dear to me as Super Mario was on Nintendo.

46:16Cameron Etezadi:No, seriously, you can't leave me on that note. I'm like right here like, oh, and me, I want to boot that up. I want to play SimEarth now. OK, well, you're going to send me a GitHub repo when we get off this call. And I really want to learn more. So thanks for sharing that. And also to those listening, you know, we're going to make sure that we get all of these links in the show notes so you can go and learn about these resources, but also go and listen to Cameron and learn more about like what the leaders at LaunchDarkly and how they're thinking about the space. It's great to be here. Thank you for having me.

46:45It's been a lot of fun.

46:47Cameron Etezadi:It's been a blast for us too. And just to wrap things up at the very end here for those listening, you know, we also have a newsletter that comes out with this episode. So if you're listening to us wherever you get your podcasts, be sure you pop over to LinkedIn or Subsec so you can chat with us there. Cameron and I are both on LinkedIn and we would love to hear from you as well as hear how you're building for the future. So Cameron, yes, I had so much fun with you. I can't wait to have you back on the show and best of luck at your upcoming talks. I know you're going to crush it. Thank you. Thank you, Andrew.

47:17It's been a pleasure.