Many tokens make all bugs shallow & open source’s new maintainers | Chainguard's Dan Lorenc

0:04Welcome back to Dev Interrupted. I'm your host, Andrew Ziegler. And here on Dev Interrupted, we've been talking about the agentic transformation and how it's been coming for engineering. And there's also a darker side to that speed as well. All of the security debt that we rack up underneath all of that progress in an internet that frankly just wasn't built for autonomous agents. And joining us today is someone who spends his weekend stress testing the Claude Code and all the places that you can take agentic engineering, but then also spends his weekday securing the global software supply chain.

0:38He's the co-founder and CEO of ChainGuard, Dan Loring. Dan, welcome to the show. Thanks for having me on. Yeah, a lot is changing right now. I think everyone is scrambling to try to figure out what that means and where we're going to be a year from now. Everyone's guess is as good as mine. I'll start by saying that I've got a lot of guesses, but I've never seen software move and change this quickly. So, yeah, we got to take guesses and see what turns out to be right. Yeah, taking guesses. And I think it's about experimenting all the time. You can't be standing still. And part of experimenting and trying new things is maybe throwing away assumptions about how engineering is supposed to be done or how it can be achieved.

1:21And we've been following a lot of your work, Dan, about the way that you've been working with agents and thinking about them in like a parallelized way and otherwise getting to like a state of eventual determinism through a series of gates and checks that allow like an agent to drive itself forward. And this is something we've actually talked about a lot on Dev Interrupted, the idea of chaos going in and progress going out. And there's a lot of our listeners who are all on different stages of this journey still, of getting to that point of trusting the machine and having the security guidelines and the safety in place to go fast.

1:55So I want to just start there, Dan. How do you think about the multi-clawed philosophy, for example, and everything that you've been publishing and releasing to the world? Yeah. So I think if we step back to kind of what's happened in the last 18 months, I would say agents and coding tools and AI autocomplete and stuff like that aren't new, right? Copilot and VS Code from GitHub is actually the first consumer LLM product out there. It predates ChatGPT. A lot of people forgot about that because the space has moved so quickly. And it was pretty good for a while. that kind of complemented existing development flows though it really was a smarter autocomplete where we've gotten to in the last year i think it has really started to change the way people are writing code right it's a change from you in an ide writing code yourself maybe the llm saving you a little bit of typing to that flipping llms are writing the majority of the code that people that are using them anyway are shipping today the cloud code tool itself i think it's one of the best software tools or programs ever written and released.

3:01It was in a crowded space of all of these IDEs that people were using and cranking out. And they kind of inverted it and said, no, no IDE. It's just a terminal app that's going to write all the code with bash and grep and said, just like the creators of Unix intended, right? No IDEs. Your IDE is Unix again, and a tool that actually knows all those flags to a level no single person does. But it still didn't really change things until probably about six or four months ago now, which is like a lifetime in AI speak. But the models actually got really good and the tool calls got really good and the harness got really good to the point where it was just kind of, in the beginning, it was just kind of mesmerizing to watch it slam all these grep flags and stuff and write code that way.

3:41But it wasn't really any faster and the results weren't terribly good to now when it's better than any person writing code that I've seen. You still have to prompt it. You still have to steer it. But you can crank stuff out in hours or days that would have taken weeks or months before. And I kind of equate this to like power tools, right? Like imagine that we've all been doing, you know, woodworking by hand for decades. Every engineer has some weird fantasy of like retiring, turning off all of their electronics and doing hand woodworking or something like that. And that's kind of how we've been writing code up until now, even with some of these fancy LLMs and IDs like Cursor and Winsurf and stuff like that.

4:20People are still writing every line of code. but now it's like we handed the whole industry circular saws and we're like go try to use these things with like no safety course or anything like that yeah it's a lot more powerful people are going to get fingers cut off you're going to make a lot of mistakes it's a lot easier to mess something up but you're going so much faster and so that's sort of the shift everyone has now of like how do we do this safely and then at the same time this stuff is also getting good enough where you can build entire factories around it and we're starting to see that a lot more too instead of people writing and reviewing and shipping code robots are doing that and if you go with that same analogy of like you know hand work woodworking to power tools yeah this is now full assembly line mode that we are either able to create now or just on the cusp of it and people aren't even going to be operating those circular saws they're just going to be operating this actor itself i like that you go to the metaphor of going from like the hand working tools to suddenly getting power tools i think that's really powerful when when jeffrey huntley was on our show He also made a woodworking analogy, but he said, you know, the idea that you have to have, you have to prove your worth before you can use the table saw.

5:28The idea of understanding the bounds and the constraints of the workshop you're in and how to keep people out of danger. And that becomes the real job now of engineers of how do we create these working environments, this tooling, this process, these rituals that allow us to capture all of these new gains and this new way of working, but also has a fundamental level of trust and understanding to it. And so in that world, you've described it as a factory. I agree with you. That's where everyone is going. We're going to be stacking all of this until we get to the idea of a assembly line kind of style output, where all we need to do as engineers is align on the intent of what we're trying to achieve.

6:15And then the rest can happen downstream. But just as well as we can use that to create, others can use that to look for weaknesses and to deploy maybe bad actors as well. And so in this world where you get these two lanes, I think there's an unfair advantage for the attackers. They can parallelize a lot of probing and looking around and harm. But when you're on the defense, you don't know where to look to protect yourself. So how do you account for the idea of that almost like a losing arm race between those two sides of the coin? Yeah, so I think we're in a really interesting state. And this is like we're moving up an exponential curve very quickly here with these capabilities.

7:01Big enterprises, security teams, they're usually the slowest to adopt any new technology because they should be, right? They're not YOLOing every new app into prod where you've got your bank account data and stuff like this. or your health records. They let everyone else try things out, see what works, see what doesn't, see what broke, see where they got hacked. And then they move it into their environments. But when you're on an exponential curve like this one, if you typically adopt things six or nine months after everyone else, or one or two years after everyone else, that gets farther and farther behind every single year as we move up this curve quicker.

7:34Like before you, maybe you were two years behind the rest of the industry. Now that's two decades behind with how fast things are moving. Attackers don't have that same set of constraints. And so they're now going to be two decades ahead of you instead of two years ahead of you. The ways of thinking about how you're going to secure your system are wrong and they're not going to work unless you try to get as close to that as possible. And I'm not saying everyone has to go run open claw and prod inside of their banking infrastructure today because that just came out a month ago. But you do have to consciously try to get closer to the bleeding edge.

8:07Otherwise, that gap in that exponential curve is going to make it impossible for you to secure anything that you're running today. Yeah, we can't let attackers have the fancy new stuff forever. In this idea where they can get that far ahead of you in terms of like attack vectors, right? What do you think are some basic ways that maybe a company that is typically going to lag on that adoption curve? I think a lot of our listeners are at those kinds of companies and their leaders there where they have to grapple with the realities of the bureaucracy and slow moving enterprise adoption of these tools.

8:42What are the things that they should be doing to be proactive and protect themselves in that environment? Yeah, one tactic I've seen work pretty well in some of these larger companies is to set up whole sandbox environments. Get your developers new laptops that don't have access to the same code bases and that kind of thing and carve out time to get them playing with stuff. because if they're not even aware of the state of technology, then that's half the problem and they don't even know what they're missing out on. And when you finally do bring something in, they're going to have a six to nine month learning curve to get comfortable with these tools.

9:13Like I'm really, really, really good at cloud code today, right? And that's because I've been using it for a year. As the tool has gotten better, I've been able to understand the capabilities and can kind of press that limit. That learning curve isn't going to go away. And the more time and the more ways you can get creative to let people experience that without also sacrificing your security posture and opening up your entire tool chain to open flaw overnight. It's better, right? You have these constraints. You're not going to be able to get rid of those, but you need to figure out a way to get your workforce and get your engineering teams and get all of your leadership aligned.

9:44This is where we're going to go as soon as we can figure out how to do it and be ready for that time. I really like that your answer for that was going straight to the human element. It wasn't, you know, a lot of people would be obviously leaning into the more of a technical way of fortifying yourself. But no, the best way to protect yourself is to upskill your employees, make everyone aware of the realities, create that shared space where folks can experiment and understand the bounds of it. Because like you said, it's just like a daily motion. You didn't start riding this bicycle till a few months ago.

10:15That's how easy, that's how convinced we all are that it's teachable because we all learned it, you know, rather so quickly. This is moving so fast. So in that world, what are the kinds of skills that you think are most important for a senior engineer right now? Yeah, I think it's sort of intuition, right? Like, I mean, all engineering is intuition somewhat at the end of the day, but understanding what the model capabilities are and what types of tasks it's going to be able to do without supervision and which ones are just going to cause it to go on a loop and go crazy and self-destruct and know where those limits are and how to scope things and break them down so they fit into context windows.

10:51and then when you get a new bigger context window, see what it can do with that one before things start going off the rails again. There's no real concrete skill here because it's changing so fast. If somebody were to publish a course of like, become a master of this tool, all those Twitter influencers that were posting, like here's this magical prompt I built that can do anything in one shot. That goes out of date a month later when the model changes. That kind of skill is not going to last very long. the real one is just building up that intuition and keep pressing on it and keep testing it that's what's going to last yeah uh in that part of the intuition too is part of it is experimenting with new tools and agentic ways of working and operating with the world as they come out obviously sometimes this is a little bit like taking a sledgehammer to like 30 years of security practices in order to extract some value or maybe even in some cases novelty and something we've been talking about a lot on this show is OpenClaw, which you just mentioned prior.

11:51And it's a grapple on the world and how it's kind of truly left the original audience and is very mainstream now. It's the most widely starred GitHub repo. We covered that on our show just the last two weeks. And so you have a high visibility point from your perspective at Chain Guard. What are some really dangerous or scary things that you've seen agents do in these kinds of environments that kind of like keep you worried and keep you at trying to solve this problem? Yeah, I think, you know, assume every agent is like an intern that you just gave a laptop to. And that intern is going to make a mistake, right?

12:28No one gives their interns laptops with keys to production on them. Because, you know, if that intern accidentally runs the wrong command and deletes the database, like it's not that intern's fault. Yeah, they shouldn't have run that command, but it's your fault for putting them in a situation where they could have run that command. And that's the same way people are actually getting results out of these on the positive side. The teams that have these amazing CI systems and test frameworks and harnesses and continuous deployment and all that stuff that we've known we should have been doing for a decade anyway.

12:55If you're confident that when those checks come back green, you can press merge and it's going to go to production in 10 minutes, then you don't have to worry, right? These agents are just going to push VRs to that repository. No one's touching production. No one's SSHing in and debugging things. The agent's not going to be able to do that either. And then you don't really have to worry. Is the code good, bad? It doesn't really matter at the end of the day. If it's bad, just tell the agent to fix it later. You need those automated signals in that a really, really, really strong pipeline where you can ship code confidently.

13:24And then at that point, the code doesn't matter. That's how you can go from writing 500 times as much code to shipping 500 times as much code. The people without that, where they're scared to deploy on a Friday because half of the deployments crash and break things and you don't want to page someone. Now you have 500 times as much code, but you can only release things at the same rate. So the bottleneck has really just shifted in your process. I love that you call out the Friday deploy. We talk about the Friday deploy a lot on the show and the phenomenon of embracing it. And so I really agree with that.

13:56Also, the idea of going back to the basics of the pipeline, right? Having a really clear, structured system that can gate the work that your agents are doing, just like how you said, it should have been gating all the work the humans were doing the whole time, right? So like building that kind of baseline, is that like, is that where you think engineering leaders should focus on in order to extract the most value from getting started with this kinds of stuff to shipping it, you know, not going from experimenting, but to shipping? Yeah, I think so. And especially if you're in one of these regulated industries where you can't roll this stuff out yet, the best investment you can make to get ready is to get rock solid deployment pipelines that you can trust today.

14:40Because once you do have these agents, they're going to love them too. An analogy I like is it's from bowling, right? So I'm a terrible bowler. If you go bowling and you put up the bumpers, you can still have fun if you're a terrible bowler. You don't really have to look. You just throw it down. It bounces off. It'll hit the pins. But now if you take 100 bowling balls and run up and slam them down as fast as you can in every odd direction, right? They're not going to get down any faster. They're going to be bouncing off each other. The bumpers are going to crash. You're going to break the bowling alley.

15:07And I think that's sort of how CI systems work, right? Like if half the tests flake and you're running 200 tests every time and everyone is just sitting there hitting retry, hoping everything gets green and then half the deployments that go out still fail, that's kind of where you are. You have these gates, but they're not really helping you get down faster or those bumpers. And I think as you start to pull them in, and I think that's really going to be the role of engineers in this future is getting those gates rock solid making sure all the intent is captured making sure all the performance stuff is in there everything you need to be confident if you can start to pull those bumpers in tighter and you know get them to exactly one diameter of a bowling ball then you can throw a hundred of them down that track as fast as you want they kind of turn from uh guardrails into guide rails there's no way for them to get off track and start bouncing around and not make it down oh i love that it's just your pipeline that way self-teaching they need to teach yeah yeah there's only one way things get to prod I don't know if it makes it through there.

15:59It's good. Yeah. And so the idea of, you know, the guardrails is just something, stuff bounces off of or you can't trust is, you know, you can't build off of that. But understanding why the guardrail is there and entrusting the guardrail. And then like you said, letting it guide you. I think that becomes a natural way where you get to that level of eventual determinism that you've written about, like with Multiclawed and your article about the Brownian Ratchet, which is a great read that we're going to include for folks. And so I want to shift here, though, to another problem scope that you have a really great view on as leader at ChainGuard.

16:36And that's the software supply chain world. And that's something that's been fundamentally altered by the arrival of AI and agents and agentic software. And the thing about it is that it's an iceberg. Like so much is built on top of it. But so much of it is so deep and down underneath in the murky depths that, you know, people like you, like me or a lot of our listeners don't have a lot of visibility on what's going on down there. But I know this is something you spend a lot of time and focus on at Chain Guard. So I want to understand from your perspective, how has the software supply chain evolved in this world and how have the stakes changed?

17:14Yeah, I think it's still early, right? Agents are around, they're getting used. their effect on open source as a whole, I think is still early. And it's hard to say too much has changed one way or another. People are feeling it. People are complaining. There's stuff happening. It's going to change, but it's too early to tell exactly what's going to happen. Right. Daniel from the curl project, Daniel Stenberg, he's been complaining for years that people are using chat GPT to basically denial of service attack his vulnerability report process. Everyone grabs chat GPT, something like that says, find a CV and curl.

17:51It's been something back that looks kind of, uh, same before you read into it too much. And then the email has private list. So it went from like, you know, a couple hundred reports a year to a couple hundred reports a week. And 99 % of them are just garbage because the people submitting don't know how to review this stuff. And that's a security vulnerability in and of itself. If you can't find the real one buried in there with 99 garbage ones each week. and so he's you know basically shut that off completely no one is allowed to use ai for security vulnerability research and curl anymore because it caused too much of a problem for him but at the same time you see things like google's deep sleep research where they they found a bunch of really good really valid zero days and open source projects that no security tools were able to find before and disclosed them and got them fixed and all that before it was out but agents can do this stuff now.

18:41And open source is kind of going to be front and center in it because it's a lot easier to point an agent at open source code than it is, you know, your bank's locked down code. So we're kind of just going to see more of everything. And some things are going to collapse under that and others aren't. And I think my prediction is open source is going to stick around, right? There's a lot of people saying it's gone now. What's the value in it anymore? Or if you can one shot every library you need, why are we reusing libraries? I don't think we're going to get to that world. But I do think it's going to bifurcate, right?

19:13There's going to be a whole group of people that just don't want AI pointing at them. And I understand why. It's just a whole bunch of noise you have to deal with as an open source maintainer. And then there are going to be other projects that embrace it. And we're going to see what happens there. But if things go well, the projects that embrace it are going to start moving a lot faster and shipping a lot quicker. And we're going to see that bifurcation happen in real time. So it's really like you think the social contract on open source will evolve and you'll get these two different types of groups who exist for different reasons.

19:43In the short to medium term, yeah. There's going to be a bunch of projects that just say, no, we can't deal with this. And some that say, no, let's go only agents, you know, committing instead of people and see what happens. You mentioned, too, the idea of just creating your own software. And so why would I use open source? And I've been reading a lot about this, too, about the folks doing these clean room experimentations where they have an agent implement something with no outside resource. Obviously, there's a huge grain of salt because the LLM itself is an outside resource. But all of that is to say it does kind of change the economics for why companies would pick up software.

20:21But at the same time, it doesn't for certain groups because a lot of parts of adopting software for SaaS is I don't want to maintain it. I want someone else to. So how do you think that balances out? And what do you think that looks like? Yeah, I think, you know, I was asking Claude this earlier this week, what it thinks is going to happen to the space. And I think no one's going to vibe code a database, right? And ship that to production, you know, something like Postgres or MySQL or those layers, right? It makes absolutely no sense. Even if it could one shot something like that, it's too much risk, right?

20:51There's going to be a bug somewhere. All software has bugs. If you point enough agents at it for long enough, yeah, they can probably squash most of the bugs. But people are going to keep using battle-tested pieces of software down there. And some of those are probably going to adopt this and start moving even faster. And then those are kind of at the bottom layer, web servers, databases, that kind of thing, where you just need them to be battle-tested and solid. And the only way to do that is for other people to run them for years and run into all these edge cases. I can't really speed that part up.

21:20And then at the top level, agents are amazing at front-end development. You can just tell it you want a website and it builds this amazing looking one. And that's because they're trained really well on these DSLs and things like React and these high-level libraries that deal with all the crazy DOM nonsense. And they can keep context windows small and move really quickly. I think there's going to be a lot of innovation at that top level, too, that let agents go fast. Libraries and things like that that they're optimized for and trained on and trained around. But that middle section, all those little middleware libraries and routers and Postgres client libraries and things like that, I can start to see people pulling in a lot more of that into their own stacks and maintaining that kind of thing yourself where, yeah, you can use this library, but you have to rewrite, you know, 30 other ways in your app that you call things and restructure to use that library.

22:07And it's not that hard to write in the first place. That area I can start to see getting hollowed out a bit as agents get better at doing that glue in the middle. Yeah, there's almost like a math equation for, you know, is it more convenient or is it more reliable for me to just use the tool? or is it cheaper for me to use tokens to build a replacement for it? And there's probably a threshold there where the usefulness of the tool way exceeds what you possibly will do with the level of tokens you can do to get a baseline version of it. So therefore, you keep it. Those are the economics, I think, that shift.

22:40And you're right that projects will fall on different sides of them. So it'll be interesting to see how that evolves. Yeah, and the stuff where it's really hard to get the edge cases right and the cost of messing up is really important, like databases and web servers. that kind of thing. We're all better off if we point our tokens at one solution and make that better over time rather than everyone pointing their own tokens at their own solutions. I love that because it's kind of an analog to the whole, like, you know, eyes make all problems shallow. The idea that everyone's tokens could make those problems shallow too.

23:09Yeah, I was working on a version of that. Like, yeah, it's Torvalds' law. Many eyes make all bugs shallow. It's like many tokens make even more bugs even shallower. That's amazing. And speaking of, you know, this ecosystem is going to evolve and change. It's going to be interesting. But the thing about open source is that it, you know, sometimes lacks its guardians, its champions. And sometimes that can be hard for it to come by. And that's what can make open source and all of the gains from it so tenuous and something that we take for granted a lot of the time as an industry. And so there's like an element of like, how do we sustain the development and the proliferation of open source in the future?

23:51How do we find how do we discover these new forms that open source is going to take and the value exchange that both sides are going to have? But part of that too is that just like a lot of modern code bases in the world that we live in relies on open source. But there's in this world where you're describing like long, long standing projects can't even accept contributions or pull requests anymore, get inundated with security features. They might spend hours staging up a good first issue just like for a human to never be able to come along and discover it because of the new world they live in.

24:25And so then how do they hand off the project to someone else? How do you develop a community around that? I think that becomes the real challenge. How are you thinking about that at Chain Guard? Yeah, I think there are pros and cons to what agents can do in this world. There are a lot of projects that are just plain done. No one ever wants to call them done because they're always open and you can always come up with something new to add. But for the most part, they're feature complete. They're done. They're tested. They don't really need much extra work. And we see a big sustainability crisis kind of at that end.

25:03Those also tend to be the most widely deployed projects, too, because they've been around and stable and haven't changed every six months for the last decade. So they show up in super low levels of the stack. They're everywhere, even places you wouldn't expect to see them. And that's hard because the maintainers need to be around if there is a security incident or something like that. But it's not a ton of steady work. So it's hard to fund that work, too, because it's not a full-time job, even if you were to try to hire someone and pay them a full-time salary. It's a couple hours a month. Maybe one month out of the year, it's a whole week.