[AIEWF Preview] Containing Agent Chaos — Solomon Hykes

0:03Hey, everyone. Welcome to a latent space learning pod. This is Alessio, partner and CTO at Decibel, and I'm joined by my course works founder of Small AI. Hello, hello. And today I'm so happy to have Solomon Hikes join us. Solomon, you're most famously the creative docker. Hi, thanks for having me. You started Dagger six years ago and I think originally it was pitched as some sort of infrastructure provisioning thing. I'm sorry, I'm probably totally mangling this in front of you. How do you introduce Dagger today? So Dagger is, I think six years, yeah, six years, I guess sounds right, yeah. Yeah.

0:37It's a workflow engine. It's an automation tool for software teams that would deliver software faster, more efficiently. And it takes all these workflows that are usually semi-automated with artisanal scripts, your builds, your tests, you kind of end-to-end pipelines, and it turns them into robust, modular workflows that you can drive with codes, and it all runs in containers. So it's highly portable, highly isolated. You can run them locally or in CI, which saves a lot of time. And we're an open source platform. We've got a very active and engaged open source community, mostly made of platform engineers.

1:16You know, those systems engineers that actually design the factory and run it and enable the developers on the team to be more productive. So that's our core community. Yeah. In some ways. Yeah. Sorry. I was going to say, just to make that clear, these are both pre-development. So spinning up an environment for people and then also between you're done writing the code and getting ready for production. Are those basically like the two entry points? We started mostly post-development. So anything that happens after you've saved and you're ready to see what happens next, you know, build tests, you want to take that live.

1:51So there's that delivery loop, right? We've been focusing on making that delivery loop more efficient because it's really terrible in most places. And there's a lot of inefficiencies that can be cleaned up. In a lot of ways, it's a cobbler's, you know, the cobbler has no shoes situation. Like those platform engineers, they spend all their energy and their significant experience helping developers get the best possible tooling and the best possible experience. But they themselves, for their own tooling, it's sort of like, okay, we got to cobble this together with Bash scripts and YAML. So we've been focusing on that post-development.

2:26although recently we're getting pulled into the dev loop in part because of this crazy change that's sweeping the whole market right with agents yeah so i think let's kind of run right into that obviously there's a lot more of context on dagger and docker that you've done prior but we you know we are an ai podcast so like why not let's just let's just go right into it a few months ago you messaged me and you were like i think this is the biggest thing i've biggest idea I've had since Docker itself. And I was like, what? Docker is very big. What is your context going into the AI builders? In some ways, I've always said basically like to us DevTools people, it is just more of the same.

3:07Everything that you wanted, you just need 100x more. So how did you approach this? Yeah, we didn't think of ourselves as an AI company. We got pulled into it by our community, our users, because although Dagger is primarily used to build CICD pipelines, historically, we've never thought of ourselves as a CICD company. We like to build platforms from first principles, and then we encourage our community to go and apply it. So what we have is an engine for automating workflows, making them reliable, portable, and giving them very clean environments to run in. And the environment is key. And of course, we use containers because that's what we know.

3:47And the container tech still today is underutilized, misunderstood. It can do so much more so that our community pulls us into this AI space because of agents, because these platform engineers start messing with agent loops. They want to insert LLMs into their workflows. And now it's becoming more popular to run agents in the context of your CI, to automate more parts of your delivery. And then they started showing us that everybody wants to use these coding agents now. So if you're a developer, increasingly your job is not going to be to actually develop, but to manage and enable these coding agents.

4:23And we're at the very beginning of this. You got this one agent in your IDE helping you, but now you want more than one, right? You want a team of agents sort of doing the work for you. And that transition from a team of one to a team of multiple coders, that's basically what our community deals with, these platform engineers. So what we're witnessing is developers becoming platform engineers. They have to learn how to enable others to be productive. These others, of course, are AIs. And to do that, they have to give them environments to work in. And you can see the problem when you see someone live streaming their vibe coding, right?

5:01Everyone's vibe coding. but you can kind of make one set of changes at a time. You're kind of, everyone's sort of messing with this dev environment that really isn't cleanly isolated. So what we're doing is we're taking this technology that we invented for CICD and bringing it into the coding agent's environment and giving your agent basically a perfectly isolated, reusable, and portable environment so that you're not completely locked into this one app connected to this one model running on this one cloud infra provider, right? You want the environment where the agent does its work to be decoupled, to be its own thing that you can manage and look at and then move to another platform if you want, right?

5:49That's sort of what happened with Docker in the previous wave when everyone was adopting cloud technology, right? Everyone was building these big platforms and they had everything. but they were highly fragmented. They tried to kind of cobble everything together like a monolith. So you didn't have this portable environment that you could carry around with you. You were trapped in one big platform. And the same thing's happening now. So we want to use our experience from the past to enable this new generation of developers to really unlock the potential of these coding agents. That's what I'm excited about.

6:23Yeah, I think the scope of what people do for coding agents today is maybe kind of like single VM, let's call it. Especially the coding agents from the big labs, they don't even have internet access. They have like pre-installed libraries and like it's very, very limited. And it could be so much better if there were standards for it. I think obviously the standard needs to be open source. But there's a question of like the design constraints that you are coding for. I think that, so for example, Gitpod is another demo and personality. market participant I've seen where they're like, yeah, like we, we really need to isolate like this kind of like almost like a mini VPC.

7:06Like you need the, you need the networking sorted out and you need a storage, everything, right? Like just all the fundamental units of compute. So like, I guess like, what is the container here? Like what is that concept here? The unit of actual containers as the base layer. That's the, that's my, the first insight here. Like the, you don't have to reinvent to the core technology that already exists, but you got to rethink the tooling because the tooling as it stands, I mean, honestly, it's, it's a little frustrating for me because we, we busted our ass on Docker and we built this whole ecosystem and we invented the Docker file and Docker compose and the format and the CLI.

7:42And then, you know, we, we kind of messed up on the follow through. And, and at some point Docker basically stopped innovating and I left and, and the ecosystem continued around containers, but it didn't actually pick up where we left off. Everything towards applying container tech to development kind of stopped. And, you know, there's still a lot of interesting experiments, but it never had the unity of purpose that this initial movement had. And so you have fragmentation, right? No standard really emerged beyond Dockerfile, Dockercompose, and that's it. All the effort went into infrastructure, you know, Kubernetes, scalable storage, scalable networking, you know that kept moving a lot and if you go to kubecon or any of those events like you see all the infra people applying containers to solving that problem but for development for dev environments it really hasn't moved that much use containers would be the my first recommendation but then yeah i just i just think you gotta you know design a solution from first principles it's just a difficult design problem and i'm very excited because there's an opportunity to go through this design process again from first principles.

8:51So yeah, maybe, I mean, I'm aware of Gitpod, of course. You know, there's dev containers as a standard in IDEs, and there's a million other products out there, but none of them have convinced the majority of humans to develop in them with their tool. It's very fragmented. In fact, most people don't develop in a container still, right? They just develop on their laptop. That's just a failure of tooling. So yeah, I think It's anyone's game, how to apply the container technology to the perfect developer experience. But the criteria are, it has to be well isolated. You should be able to have a bunch of agents working in parallel, and they don't miss each other's workup.

9:30It has to be portable. It should not be locked to a model or a cloud provider. It should also not be locked to an IDE. That's crazy. If you think a whole team is going to standardize on the same IDE forever, you're deluded. It should be fully observable. You should be able to see everything that happens in that environment end to end. Everything from what the model's doing and thinking and saying, all the way to what are the tools actually running and what's the state of the environment. You should be able to see everything in one place. And you need a strong multiplayer element. You need agents and humans to both be able to interact with that environment so that you can tell an agent, do this.

10:05And when the agent says, I did it, you can go and verify. Okay, did you do it? Give me the keyboard for a second. You need all those things. And right now, I'm not seeing us heading in that direction. I'm seeing us heading in the direction of highly integrated, very vertical end-to-end monoliths. And I don't want to name names, but it's definitely a market trend. If you're selling an IDE right now and people are asking for more customization for the coding agent's environment, you're going to add some proprietary way to customize the environment. You're going to add your own observability solution.

10:44And when people ask for a way to have the agent work in the background, you're going to add your own hosting solution to run the agent in the background. That's a monolith. That's what fragmentation looks like. And that's exactly what we were up against in the early days of cloud that led to the creation of Docker, right? So we're going to need some sort of a standard, not on everything, just on the environment, right? Just that little piece that connects all the other pieces. It's not the most powerful piece, but it's sort of, it's the linchpin that connects everything else, you know? The environment in which the agent does work.

11:22That should be independent. What do you think are like the biggest limitation today? So I use Docker to develop and I force the coding agent in the IDE to run commands in the Docker environment. I would say if I had to give feedback on it, it's like one, the agent cannot make changes while inside the container and propagate them back to like the Docker file and Docker compose. so it kind of like spends all the cycles and then all the work is kind of lost and there's no like ai native interface i'm basically just doing docker compose exec and running the command in the container instead of like having a more native way to do it how do you think about that changing with like the new dagger approach i mean yeah i you just got to rethink there's just a design process you gotta i mean docker file was something we designed as a stopgap prototype thinking, oh, we'll clean this up later in 2013.

12:14You know, it's been more than 10 years. Compose was a clone of a clone that we acquired into the team and like stitched on top. And then as soon as we launched it, you have to understand there was so much excitement and demand. People didn't want it to move. You know, you would build platforms on top and then you'd be like, don't change the syntax. And so that stuff has been frozen in time. And I commend Docker for maintaining it and keeping it alive and just like doing the hard work and maintaining it. But it's not it's not agent native. It never will be. So there's like there's just got to go through a good engineering and design process of understanding how people how people develop with agents and understand what's the best UX that they need and try to design that on top of technology and components that is that are as standard as possible.

13:05you know you don't want to reinvent the wheel where it's not necessary but we've got a bunch of standards to work with we've got the container tech i mean it's there it's universal we've got git we've got the llm you know the open ai um api spec right and and its derivatives and now we've got mcp so that's pretty good that's a pretty good set of standards we can work with that but But yeah, it's got to be a new UX, in my opinion. And I mean, not to plug Dagger, but obviously Dagger is our vehicle for going through this design process. So if you want to see my particular opinions and how things should be designed and how you want to balance, for example, simplicity versus modularity, then in my case, look at Dagger.

13:52but um yeah it's normal that it's normal that you can't just tape existing tools as is on a new workflows and hope it to be i hope it'll be perfect that's that's totally normal something i bring up in my work history a lot and i think you know it's also worked in workflow orchestration and temporal yeah and like this migration of like let's say like a custom language like a docker file or like an aws step function type thing into like a more programming language like a typescript or go So that is the exact same journey I took. Yeah, I mean, it's really hard to find the right balance. I think of it as Lego because it's a hard problem to solve, these workflows and these environments, because no two workflows are the same.

14:31No two dev environments are the same. It's like factory design. Every great product has its own factory that's unique. No one goes and buys a factory at the factory store. You design it and you build it alongside your products. And that's what these things are. It's like a factory. And it's really hard to provide tooling for that space because if you make it too complicated and too customizable for no reason, then you're wasting people's time. It's just, I buy this. I'm just going to do everything myself from scratch. But if you try to streamline and simplify too much, then you're restricting choice and then it becomes useless.

15:10You know, like, well, this doesn't fit in my factory because my factory uses this system and I can't reconfigure it. So it's a really hard area of design and engineering. And I mean, to me, the GOAT in this space is Lego, you know, the actual Lego. And it's used as an analogy so much that it loses its meaning. But if you really think about what Lego really is, it's really hard to, it was in reality, very difficult to design and engineer the Lego brick because it had to be just right. And it's designed as one component, but it's designed with a much larger system in mind. So there's sort of this two layers of design.

15:50That's what's so hard. And that's why Lego is genius and has stood the test of time. That's what everyone in this space should be trying to do. Build a better Lego system that is worth adopting. It's expensive to adopt a new standard in your stack. It's one more of everything to worry about. right so it's got to justify the cost by actually saving you know it's got to save you something effort money whatever that's what lego does when i play i like to play with lego because you know i can assemble things quickly etc so yeah that's that's the challenge in the case of temporal i mean i think these these systems are very well positioned also for running agentic systems right because it's an agent always has some sort of loop that's triggered by events.

16:35It's asynchronous. You got to run that somewhere. Yeah, I would say temporals focus more on the runtime applications and then other systems are more focused on CICD. Honestly, you could use one for the other, but... Yeah, but they're converging because your CICD will soon be nothing more than runtime infrastructure for your workflows. And all those workflows will become agentic, right? It's going to be either workflows running LLMs, LLMs running workflows, all the way down. And CI is just, it's also events, a job dispatcher and compute. So I think these things will converge with coding agents being the domain of application where everything meets, right?

17:12Because when you're running a coding agent, you're running an agent. So it's a runtime problem, but it's a very specific area of application. Like it's not real time. You don't have to worry about voice and video and things like that. You worry a lot about artifacts that are being produced and are they repeatable? Can I trace how they were created? Is this binary created by an agent that the model that went rogue, you know, things like that? I think one element that I see a lot for these kinds of like, we've, you know, we've talked to Bolt and like there's all this universe of let's call it ephemeral apps that people are making or vibe coded apps, single use apps, even just because it's so easy to create an app now that you're like, I just, it doesn't matter.

17:55So the speed and the setup and the teardown is pretty significant. And then obviously the resource usage and cost. These are all things that I'm hearing the founders in these companies all trying to solve for and all finding the current set of tools lacking. Because we just don't have the, we cannot subdivide things that small or that cheaply. We can't start things up that fast. And but like the users always want this. That's why they're doing the shortcuts. That's why they're not using containers. Because they're like, I don't know how to do this with containers. Yeah, and there's a lot of duct tape.

18:30I mean, if you want to use containers yourself, you've got to duct tape a bunch of tools together. And it's not just containers. It's also like the file system isolation. You know, everyone's playing with Git work trees to try and get multiple instances of the agent working in parallel. That's the same problem, right? It's not just containers. It's something container-based. It's containers for isolated execution, work trees for isolated files, and you've got to connect it together somehow. I do think what I'm seeing, I'm seeing a lot of vendors, of course, think about that and try to find the right solution for their customers.

19:07I think one mistake everyone's making, again, I'm making a historical parallel with the rise of PaaS. Only you can do this. Hopefully, I'm not the only one left. But I think it's going to be very apparent very quickly that when someone who sells a commercial cloud product, a cloud-centric product in the AI space, they're hosting your stuff, right? You log in and they have your data. They have your traces. They run the model or they proxy to model whatever. Everything solution they come up with will be very infra-centric, right? They're going to think of another hosted feature, another hosted service.

19:43So when they think environments, they think, how can I run these VMs on my infrastructure as fast and cheaply as possible? You know, my scale is going to be X. I have this many customers. The pricing is X. And that's excellent. But developers also want to run stuff themselves locally. And they don't really want that to be an afterthought. But right now, it is an afterthought. And it's the same mistake CICD vendors made. CICD has no good local story. There's some open source projects that tries to simulate, say, GitHub actions locally. And it's like, it must be a nightmare to maintain this project because compatibility is just so hard.

20:27But my point is just local execution. It's not everything, but it's a good test. Whatever solution you're imagining, does it support local execution? Will developers be able to run it locally and enjoy it? If the answer is no, you're solving part of the problem, but you're not fully solving the problem of standardizing dev environments for coding agents. It's not going to stand the test of time because it cannot be ubiquitous. It'll be a great commercial solution. You'll make lots of money with it, but it's not going to be ubiquitous. So I'm going to ask a really hard question maybe, but what does it take for one of the big clouds or big labs to adopt Dagger?

21:08Yeah. So for example, I've had this exact same call, same problems with the Microsoft team, and they're pushing DevContainer. And obviously DevContainer is not enough, but whatever. They want to build around it. They want to extend it. And I'm like, okay, but everyone's working on their thing. When do we get some consolidation in this space? yeah well who knows i mean honestly everyone should just give it their best shot and design the best possible solution yeah honestly i think with open source to some degree well it depends on the the the area the domain but i think for this problem scale doesn't really help as much like if you're if you're going for like foundational models to take the extreme example.

21:51Sure, let the best design win. But also I know who's I know who one, you know, I know the 10 names of who's, you know, I'm saying like, who's going to win, it's going to be one of those 10, like startups don't really stand a chance. This is very different. Because it's not about scale. It's about developer experience. And it's about designing the interfaces in a way that actually help people be more productive. So there's a lot of leverage for small teams to just design the best possible solution. And then you have to convince others to adopt it and build on it. So community and ecosystem are extremely important.

22:29If you're large, like Microsoft, you have an advantage on that. Obviously, there's a Microsoft ecosystem, it's massive, but you still need to have a solution, right? So like we're talking to Microsoft, we're talking to a whole bunch of people, but there's a lot of in this particular problem that we're targeting, you know, standardizing the dev environments for agents, you don't really need permission as a startup. You just go ahead and do it and build momentum. There's less gatekeeping, I would say. Yeah, awesome. I think the other thing that does come to mind sometimes is that there are different layers in infrastructure.

23:01I think you're very focused on, I don't know how to call it, virtual hardware. Is that like a term that resonates? Like virtualization? no like that when i look at let's say what i can do with dagger it doesn't for example have off or billing right the higher level still infrastructure yeah yeah yeah but there are like recombinations of things that are more sort of business facing than than just like the hard metal facing yeah it's it's so we're i think that's what you're describing as a consequence of the Lego approach? Like we're focusing on a platform problem. You know, how do you create a modular system that can run anywhere, you can connect to anything and allows you to compose the ideal environment, the ideal workflow.

23:55And it can be ubiquitous because it can be integrated into pretty much any existing system. In order to do that, you can't go ahead and design a complete end-to-end solution, right? That's the price to pay. So Dagger will always be a component of a bigger platform. It will never be the complete end-to-end platform because in order to do that, you have to force your customer to adopt your authentication and your UI and your storage and your networking, et cetera, et cetera. And we're doing the opposite. We're telling these platform teams, hey, we will adapt to the stack that you have. And so, for example, we started this conversation with CI-CD because that's our traditional use case.

24:38Dagger makes your CI-CD better, but it does not actually replace your legacy CI platform. It runs on top of it, and it allows you to simplify it and turn it into basically dumb runner infrastructure. But it's still important infrastructure. So we don't come in and seek to replace what you have. We try to integrate with it and make the overall system better. and so as a result it's just a different shape of a product right and we definitely don't solve off you know you gotta you gotta you gotta rely on others to help solve off but whatever you do we'll integrate with it yeah yeah um cool um i think that that's plenty for an intro to like where daggers at and you know the pool that you guys are seeing i'm looking forward to your talk to be honest like i think i literally haven't seen like a good like solomon hikes like ai talk Well, I don't know anyone has.

25:29Yeah. So I'll see one next week. Yeah, I know. I know. You know, let's let's, you know, feel free to lean on me to prep and all that. But thank you. I'm excited. And I think like when I saw the workshops, the mission come in, you guys had a really good like, let's let's build a code, code, code agents. And like, here's all the things that really tied it together for me. That was like, OK, this is where what that is going. This is how like, like ultimately, like I need the LLMs to start generating their own infrastructure, right? Like generative infrastructure almost. I feel like that's like prone to a lot of spend.

26:05Yeah. But it's going to happen. Controlling the environment is like, okay, how much control, how much freedom really do I want to give this agent, you know? I mean, we're all past the stage where like these things are just running wild on the internet, so why not? But like, you know, to give them my AWS account and just go, that's like, that's tricky. But, you know, just, you know, depending on how things go, You know, there's like a 50 % chance that for my keynote, there will be some like brand new original stuff to show and talk about that is not yet out. Like on top of what we're showing in the workshop and everything.

26:45So it depends if it's ready in time, but we may have like even more, even more fresh stuff to talk about. All right. Well, we'll let you get back to it. thanks so much for your time thank you yeah thanks guys i'll see you next week