Successfully coding with AI in large enterprises: Centralized rules, workflows for tech debt, and training your team | Zach Davis (Director of Engineering at LaunchDarkly)

0:00Zach Davis:Vibe coding is not an acceptable enterprise development strategy. I love it. I can do 100 commits a week by myself on my side project on my startup. But when you're working on a code base and a platform like LaunchDarkly that powers trillions and trillions of experiences every day, you can't take the same strategies and tactics that a vibe coder could take. One of the things that I realized is what's good for humans is also good for LLMs. And so I really started with how do we make sure that the repo is well set up for humans to know how to work in it. So we have front end organization, we have accessibility, we have a JS style guide.

0:37So all is this very detailed documentation that we've put into the repo itself rather than have it in other places. And this way LLMs can access it, humans can access it, et cetera.

0:47Zach Davis:I think all the engineers out there are like crossing their fingers and hoping that there's one rules protocol to rule them all that shows up. And I think what you've shown is you can just create that yourself. And then that makes it much more scalable.

1:03Zach Davis:Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive here on a mission to help you build better with these new tools. Today, we have a great episode for anybody trying to deploy AI agents in a real engineering team with a real code base, not just vibe coding. We have Zach Davis, Director of Engineering at LaunchDarkly, who's going to show us how he sets up centralized rules and docs for all his AI agents, uses AI to burn down tech debt, and keep his hiring bar high. Let's get to it. This episode is brought to you by WorkOS. AI has already changed how we work. Tools are helping teams write better code, analyze customer data, and even handle support tickets automatically.

1:46Zach Davis:But there's a catch. These tools only work well when they have deep access to company systems. Your copilot needs to see your entire code base. Your chatbot needs to search across internal docs. And for enterprise buyers, that raises serious security concerns. That's why these apps face intense IT scrutiny from day one. To pass, they need secure authentication, access controls, audit logs, the whole suite of enterprise features. Building all that from scratch? It's a massive lift. That's where WorkOS comes in. WorkOS gives you drop-in APIs for enterprise features so your app can become enterprise-ready and scale up market faster.

2:26Zach Davis:Think of it like Stripe for enterprise features. OpenAI, Perplexity, and Cursor are already using WorkOS to move faster and meet enterprise demands. Join them and hundreds of other industry leaders at WorkOS.com. Start building today. Zach, I'm so excited to have you here because I feel like I maybe turned you into an AI fiend at this point. Before the show, we were talking about how many tools you're now using. So before we dive in, can you just tell us a quick list, maybe not so quick list, of all the AI tools the technology team at LaunchDarkly are now using? Yeah, absolutely. Let's see. So on the design side, again, we're exploring a bunch of things.

3:15So there's going to be a bunch of tools. So Lovable, V0, Figma Make, we're using on the product side, obviously, we're using ChatPRD. And then on the engineering side for code, heavy cursor users, heavy Devon users. We're also using now like Cursor's background agent. I personally use Windsurf because I like Windsurf, but most of the rest of the work does not. we are using we're trying augment we are looking into cloud code we're doing all the things we're also looking into PR review so we use copilot for code review and we use cursor for code review as

3:50Zach Davis:well okay and I feel like 18 months ago we were using maybe a little github copilot but not much yeah not much more and one of the things that I really liked that you and I did together and you were a champ in coming along this journey is we really decided that in order for AI to be really effectively adopted by a team like LaunchDarkly's engineering organization, which is over 100 people, we really needed to put some concerted effort behind it and put a person in charge. And lucky you, you drew the either short or long straw, however that works. What do you think about teams approaching this kind of engineering-wide transformation?

4:32Zach Davis:And what kind of organizational and cultural things you need to do to make it possible. I do think having a person who's kind of, I don't know if in charge is the right word, but whose responsibility it is to drive that kind of change. And I think that having someone who's close to the code helps a lot because you don't really know what's working and what's not working unless you're in the code, at least on some basis. And so that can be a manager, that can be a director or someone, but it has to be someone who's actually trying these things, I think matters a lot. And, yeah, you I think you were looking for someone to take that role.

5:08And I was, I think, skeptical, right, of how well things were working really well for you over on your side job on chat PRD. But when I tried to do the same things in our code base, I was struggling. And so I really came at it from a standpoint of I want to understand what works and what doesn't. and either be able to push back on you and say, hey, you know, like it's great over there, but on this, you know, larger code base, it's not working or be able to actually drive change. And now I'm at the like, I'm on board, let's drive change. I think that matters a lot.

5:46Zach Davis:Yeah, and one of the things that I think is really important for our listeners, especially ones that are at growth stage or larger companies is vibe coding is not an acceptable enterprise development strategy. Like, I love it, right? I can do 100 commits a week by myself on my side project on my startup. And, you know, I can recover from quality issues, you know, the maintainability of the code, the issue right now, that's not my big business issue. But when you're working on a code base and a platform like LaunchDarkly that powers trillions and trillions of experiences every day, you can't take the same strategies and tactics that a vibe coder could take and bring those into not just an individual developer's workflow, but an entire team's workflow.

6:35Zach Davis:And so what have you kind of discovered as you try to figure out how to make these tools work for a larger team? So with smaller teams, you have more flexibility in terms of how you approach these things. With larger teams, you have more enablement and stuff like that. We're kind of in the messy middle. And I found it more difficult to sort of like operationalize that, make everyone successful. And what I found was everyone was on their own journey to try to be successful with AI. And that just doesn't scale very well, right? So, yeah, you really need to come up with a system in order to make everyone more successful, right?

7:15What I want is when those skeptical engineers jump in and they try cursor, they try whatever for the first time or they try it for the first time in a while. I want them to be successful so that they get that aha moment. And if you just leave them on their own, then you're not going to get there.

7:32Zach Davis:Totally. And we had, I think you and I had mutually had the experience of developers having their first experience actually be negative, whether it was with Cursor or with Devon. It was like, see, I knew this was never going to work. And here's my first pass proof that it didn't work. And so you really did a lot of what I appreciate about you is you did a lot of technical work to make sure people were successful. We'll definitely talk a little bit more about some of the kind of culture and operations piece. But I actually want to get dive into what you did in the code base to make it easier to work with Cursor and Devon.

8:04Zach Davis:So can you walk us through some of those things that you did? Yeah, absolutely. So in my IDE here, you can see one of the things that I realized is what's good for humans is also good for LLMs. And so I really started with how do we make sure that the repo is well set up for humans to know how to work in it? And so we have this docs directory, and I pulled a bunch of stuff from Confluence, from Google Docs, from other places in the repo, and I put it all in here, right? So we have front-end organization, we have accessibility, we have a JS style guide. So it's this very detailed documentation that we've put into the repo itself rather than have it in other places.

8:44And this way, LLMs can access it, humans can access it, et cetera. In addition, we had these, we had cursor rules before. We had a CloudMD file, and I wanted to consolidate that. And so instead of a.cursor rules, I have this.agents rules. And the idea is to kind of centralize all of this knowledge in one place. And so you can see here I have something like TypeScript Essentials, which has a really kind of like quick, like the quick hits of what's really important. And then it also links off to like the comprehensive docs and says, hey, go if you want to find out more, you know, go look at the JS style guide.

9:24And so then our cursor rules actually just point to that, right? So with our cursor rules say, hey, if you want TypeScript guidelines, go find this file in.agents. And then I talked about augment earlier, we were trying augment, I set this up yesterday. And I had, I asked the augment agent to just create this, I pointed at the cursor rules, and I pointed it at our agent's rules and I said, can you just create this file? And so it did the same thing. And this way, we don't have to duplicate everything across multiple tools or tool files. And it's much easier to get stuff working well by default.

10:04And the whole idea with this is, again, I want people to be successful out of the gate and having this kind of centralized place, having all this documentation in the repo just makes it way easier for tools to be successful by default.

10:18Zach Davis:Yeah, one thing that I hear a lot is people are really frustrated with the tool specific rules. They're like, why do I have a Claude.md? Why do I have a cursor rules? Why do I have these GitHub rules, especially if you're experimenting with the number of tools that you're trying? Each tool has isolated their rule set in an individual file structure. And I think all the engineers out there are like crossing their fingers and hoping that there's like one rules protocol to rule them all that shows up. And I think what you've shown is you can just create that yourself, create a directory in your repo, put a consolidated set of rules, make your sub rules for each of those tools point to those rules.

10:59Zach Davis:So say when you're looking, you know, when you're working on front end, reference these rules. And then that makes it much more scalable. The other thing that I want to call out for you is, I think what you said at the beginning, you're using, I don't know, a dozen tools, probably like three to five IDEs across the engineering organization, probably within any individual engineer. You're testing like I want cursor today and Devon tomorrow. And if you don't have your rules set up for all those tools, then you're starting from scratch every time. And so I really like this idea of a rule setup and then consolidation.

11:35Zach Davis:I'm curious, do you feel like the rules have improved the quality of the outputs significantly? Yeah, yeah, absolutely. So here I'm looking at our feature flagging rules. And it's interesting because we are a feature flagging, we have a lot of feature flagging code in the code base. And one thing we noticed was that some of the models, some of the tools would get confused about whether we were asking it to create feature flags on the like launch darkly product or whether we were actually trying to get it to do stuff in the code and so there's there's a bunch of stuff that i did to just be really specific about how to make how to be successful when creating feature flags i want you to return a link that kind of stuff and it really has it has made a difference yesterday literally uh one of our product managers was doing a task with devon and was able to tell it to put a flag, put the feature behind a flag.

12:27And Devin went and used the MCP and hit the flag and everything worked.

12:33Zach Davis:So I have another question about rules because LaunchDarkly's giant monorepo, 10 years old or something like that, it's got a lot of code in it, front end, back end, tests, all this stuff. What do you think, if you had to give some advice to peer engineering leaders who approaching the same problem, what are the must have rules from your point of view in the code? You know, I saw like a lot of front end stuff. But what you know, what are the quick hits of what you think should belong in a kind of cursor rules or a rule set? I would say the best tip is ask the agents, right to get you started. And so Devin actually has a great wiki that hat for each repo that Devin works on, it creates a wiki and it has a ton of really good information in it.

13:15And so I actually started with Devin and I said, hey, can you like this is what I'm trying to do. Can you create basically the human readable docs for this? And so Devin did a pass and created a bunch of docs, suggested some structure. We went back and forth and I kind of tweak things. And then I took that output and I went through it with kind of like a fine tooth comb because I think it matters. Right. It matters to get those details right. And then once I had the human readable docs, I went to cursor and I said, hey, can you take these docs and can you take your existing cursor rules and can you turn those into a dot agents file?

13:52And so it was a combination where you can kind of lean on the agents a little bit to help you get unstuck and get started and then also use your knowledge of what's important in the repo. And the other thing is that I was looking at where are people getting stuck? I knew that people on the front end would struggle with getting like testing, basically writing unit tests. It would write just tests and we use V test and stuff like that. And so putting in specific rules to make sure where people get stuck, we have rules to help the agents be more successful.

14:28Zach Davis:Would you mind pulling up Devin and actually giving an example of generating a rule? and I have an idea for you, which is like rules around generating data visualizations since we've done so much and just, you know, see what it comes up with. Here's an example of both asking Devon Wiki a question and then also using Devon to create a rule. So we can say to the Devon Wiki, we can say, what are the libraries used for charting on the front end? I'm curious while this is loading, how long did it take you to set up Devin's environment? It's something that, you know, everything's easier with a little Vibe code and Greenfield app, except for setting up Devin's environment.

15:11Zach Davis:It's just as hard. I'm curious what your experience has been configuring Devin to work in a large repo. Yeah, I would say to get up and running with Devin, I got started pretty quickly. And we have kind of, we have a separate flow for front end and back end. And we have a concept of sort of like front end only mode, which proxies against another back end. And so I was able to get Devin's machine up and running pretty quickly in just front end only mode. And then I was able to take on front end tasks using Devin. One of our other engineering managers actually came in and saved the day on the back end to get the full, like running our whole end to end up and running with Devin.

15:53And that took him, I think, a little bit more time than it took me. But the nice thing is you can do that sort of incremental, you know, you do what works. You don't have to have Devon running your full app locally in order to get value out of it. And so it's just about kind of like doing it piece by piece. And again, if it's hard to get Devon up and running, it's probably hard for your human developers to get up and running. So there's always incentive to make those things better.

16:19Zach Davis:Yeah, I will give just because you said it. You said, you know, Devon's environment doesn't have to be running for you to get valued. my number one Devon prompting trick is don't run this locally. Just give me the code and I'll test it for you. So sometimes I bypass that process entirely. Okay. So you asked this question of Devon Wiki, what are libraries used for charting on the front end? And it gave answer recharts, some other things. And so what would you, one, is this information pretty accurate? And two, what would you do with it? Yes, it is accurate. We are using multiple libraries. And so that was one of the things I was curious about is, is we've brought in several libraries and we're kind of trying to figure out how to consolidate.

16:58And so it picked out that we're using Recharts, we're using VizX. It lists eCharts as a secondary library. I don't know if that's strictly true, but generally this seems very correct. And so I like to use Devon Wiki to just sort of ask basic questions about the repo, make sure I understand what we're doing. But if I actually wanted to create a rule, so you can't take action from Devon Wiki. So what I want to do is I want to create a new document, a human-readable, human-centered document in our doc slash frontend about how to use charting libraries. And then I want to also add a rule to.agent slash rules.

17:38So I'm just going to give this to Devin, and I am going to see how it goes. So Devin's going to spin up a new session here.

17:45Zach Davis:And one thing I want to call out for folks listening on the prompt is you specifically said you wanted to create a Markdown document. Markdown is every engineering agent's favorite file type. So that's a good way just to give a little bit of structure to your code. It also tends to pretty print and be human readable and easy to view in GitHub and all that stuff. And so what you're doing here is just asking Devin to make those docs for you. And this is one of my favorite use cases of Devin. I think you know this. It's my favorite Devin hack, which is I have a GitHub action on every PR that writes docs for the PR and adds to a change log programmatically with Devin.

18:32Zach Davis:I found that it's a very good technical writer. You know, sometimes the code is OK, but the technical writing is very clear and very good. Yeah, I think that's exactly right. The Devon wiki is very good. It knows a lot about your code base. It has this very explicit way of learning and understanding your code base. And so it is very good about kind of describing that back and, as you said, doing it in a solid technical writing way. Yeah. And then one of the other things that I want to call out for people that are maybe listening and not watching is we are chit-chatting because we're waiting for Devon to spin up a virtual machine.

19:07Zach Davis:So for those that don't understand kind of how Devon works, it actually spins up a virtual environment that reflects a development environment. It's going to open it up. It's going to read your code base. It's going to do all this stuff. And so, you know, it takes a minute to actually boot into an environment a little different than running something like Cursor locally. Yeah, that's exactly right. Where these other tools are just using whatever you have locally. Devon is running its own machine, which has a lot of upside. It can run a browser and see a browser. It can do a lot of things that don't come out of the box with these other tools with the downside that it takes a little time.

19:45Depending on your repo, it takes a little bit of time to actually set that machine up and get it running.

19:50Zach Davis:But like you said before, if your machine is slow to cold boot for Devin, it's probably slow to set up locally for an engineer. So again, align incentives on getting your repo to work well for both your agent co-workers as well as your human colleagues. That is my favorite thing is all the things that have been hard for humans forever. And we have just kind of swallowed it and said, well, that's the way this works. Become even more important today with these LLM tools to solve and improve. Well, I think it becomes more important to solve and improve. And then I also think it becomes easier to solve and improve them.

20:31Zach Davis:If I said, you know, two or three or four years ago, Zach, go document everything in the repo. High quality human readable docs. You just go do it by yourself. It would take forever to generate high quality docs that, you know, really reference our code and understand the nits and details. And I think the fact that even you can spin up docs so quickly is so transformational to how you can. And I know we'll see this in a little bit, like burn down tech debt, make your engineers happier. And so I do think, you know, a lot of engineers have the skepticism that adoption of AI tools is really about moving faster, shoving more junk in the code, like just getting feature bloat.

21:13And I actually do think for mature engineering organizations, it is also an opportunity if you approach it correctly to take care of some of the things that you have just hated forever in either how you run your software, how your team operates or the code itself.

21:27Zach Davis:And so that's one of the advantages I think people underestimate in larger organizations because they blur the line in their mind of AI assisted engineering with vibe coding, which is not what we're talking about right now. No, not at all. And technical debt is my favorite use case for AI to supercharge like a medium-sized organization. Okay. So what we're seeing here is Devon's looking through your repository, accessing its knowledge. Actually, I'll take a pause here. Have you set up the knowledge in Devon explicitly, which is for folks that don't know, little snippets of kind of rule. It's almost like Devon's rules in some ways, little snippets of knowledge and rules, have you set those explicitly?

22:12Zach Davis:Or have you simply accepted and approved the ones that Devin suggests? Yeah. So as you mentioned, one of the things that I really liked about Devin, especially as we were first getting started, is it builds up this knowledge on its own in some ways. So as you're interacting with Devin, it will make suggestions for additions to its knowledge so that it gets sort of like, quote unquote, smarter every time, where some of these other tools have memories. Because Devon's starting a fresh session every time across different users, it has a centralized knowledge, kind of like repository. And so it's been a mix.