Building Durable AI Agents

0:02Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show.

0:41Daniel Whitenack:Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard and really excited for today's episode because it fits right in the theme of our show, which is practical AI, focusing on some things that are actually useful and practical. Have with us today Hamza Tahir, who is co-founder at ZenML. And they have a new product, a new project out, Kitaru, which is focused on agents and making agents durable, which is super interesting. And Hamza is joining us today. I think you are out at the AI Engineers World's Fair, right? Yeah, I am. It's like 7 ,000 people.

1:27All of our crowd gathered in one small, like, big hallway. So it's just fantastic to be in San Francisco when the energy is so high.

1:36Daniel Whitenack:Yeah, that's awesome. Always inspiring. And really cool to see also growth in that from Swix and others who've really built up an amazing community over time. Friends of the show. So if you haven't checked it out, go ahead and check out what they're doing over there. But yeah, excited to dig in today, Hamza. Maybe just to set the stage. I know your co-founder of ZenML. There's kind of some background with that. project and product around ML ops. Now you're getting into agent, agentic things. I love your perspective on maybe first off kind of the world that you have been inhabiting around ML and ML pipelines as now we're all thinking about agents and generative AI and all of these things.

2:31Daniel Whitenack:Like what, from your perspective, before we get into agents specifically, like what role does the more traditional ML models, training pipelines, et cetera, play in our world moving forward from your perspective? Awesome. That's, I think, a great one to start with. And thank you for inviting me on the show. I appreciate the opportunity. So I co-founded ZenML about five years ago. So this was really almost at a point where MLOps was really reaching fever pitch on, And, you know, there was all sorts of chatter about how to productionalize AI and machine learning workloads. And I had done four or five years of that in my previous job where I was co-founding another company trying to deploy ML models in disparate, you know, compute backends and all over, especially out of Germany where I'm based.

3:29So that led me to having a framework internally that we used that you could write workflows and DAGs and you could deploy them on these different backends. And that turned out to be ZenML. We open sourced it. We got a bit of traction at the beginning and a few projects and revenue and we raised. And that's been the story so far. And smack dab in the middle of this, from then to today, we had the agent renaissance. And it felt a bit funny because in MLOps, it was like DevOps reinventing itself. And with agents, it's like MLOps reinventing itself. And at the end of the day, it comes down to these very basic principles of how to write good software engineering code that runs non-deterministic code in a way that's safe and reliable and retryable.

4:18tribal. So I think if anything, even if you throw away every other tool that we ever used in MLOps, the principles and the learnings that we took from productionalizing these applications at scale still translate and are being rediscovered. Even at the AIE WorldSphere, I sometimes hear docs. I'm like, I seem to remember I've heard this talk before in the MLOps conferences. So yeah, I'm happy to chat deeper if you're interested in a particular.

4:43Daniel Whitenack:It's interesting, like you're talking about the workflows, DAG DAG pipelines, there was very much this phase, and at least this is how it occurred to me. I don't know if everyone had this perspective, but there was this phase with generative AI around workflow automation, and there was this very much workflow focus for some time with things like N8N or whatever. And those tools are still very useful, of course. But there is like this DAG focus. And now it seems like people have thought, well, and I remember having Jeffrey from Noose Research on the show and he's like, well, with like their Hermes agent or whatever, it's like, well, I don't want to impose my workflow into this, but there's still a workflow under the hood.

5:33Daniel Whitenack:Like there's decisions made, there's a workflow executed. It's just like you're not defining it. And so from the human perspective, you actually don't see that that workflow in like a visualized DAG, but it it sort of exists under there. Is that partially why you think like some of these principles carry carry over or reinvented in a new way? Because like at the end of the day, there is some flow of things being executed. Right. Yeah. I mean, a graph like all software is essentially a graph, right? Like when you write code, you have if this, else that, execute this, execute that. That's like the sequence of steps you're doing in a graph.

6:13I think the way we thought about workflows before were more deterministic, obviously. For example, I come from the world of machine learning pipelines, so it's like load your data, preprocess your data, train your model, evaluate your model. So we sort of knew the steps ahead of time, and that lended itself to a graph structure. And, you know, graphs introduced order to the chaos of just willy-nilly scripting things. And then, obviously, DAGs are directed acyclic graphs, but they are graphs, so they're directed in one direction. They're acyclic. They don't have cycles. They don't loop back. So in the world of agents, the acyclic part gets very tricky because it cycles all the way down, right?

6:56It loops. So it's where we had to reinvent ourselves as well. So back in like 2023, four users started hacking our pipeline engine to run agents like dynamic steps, conditional branching, state through artifacts, store workarounds. And then, you know, we were like, OK, they're starting to fight our abstraction. And so we introduced a new dynamic mode. And I think the main, the key difference here is really, as you said, everything is a workflow. So everything is an agent, in my opinion, is just an unrolled graph. So it's like LLM call, tool call, LLM call, tool call. Sometimes you do the tool calls together.

7:37So you're just, it's like a tree structure. So the trace, I guess, is a graph. And we just needed to make our system more capable of having graphs that are defined in real time versus statically compiled at the beginning. And we did that quite early. And since then, the same abstractions have worked, except, yeah, and we can get into this. There's this whole new workload that we need to think about different things about, like durability, state management, retries, and how those things work and replays. And those are the things I've been working on in the last two years.

8:14Daniel Whitenack:Yeah, I would be curious to know maybe as we get into those things, maybe just some of the stories or types of failures that you see. There's like one piece, which is like how we handle those, how we instrument things, how we, to your point, make our agents more durable. But that's assuming like they need to be made more durable. They are currently not durable. they're fragile, right? So what are the, I guess, help the audience understand some of those main categories of why agents in our world today are not durable or reliable or however you define that. Yeah, so I think in order to follow that thread, we need to see where the world is going, right?

9:07So if you look at how agents, most people, when they think of agents, honestly, still outside of our little bubble of these 7 ,000 people in the Moscone Center, they think that agents are these local cloud code instances or, you know, like Hermes or something. Yeah. And I think that because we come from this revolution of local processes that run on your computer and you're token maxing, you know, in your local machine, I think that it's very hard to then understand durability because that durability, I guess, and recovering from failure in your local machine turns out it's a simpler problem than when these things migrate out of your little machine and go into a sandbox or somewhere in your computer, in computers running in the cloud, defined by or controlled by your company that are executing arbitrary code and doing all sorts of things like MCP calls and, you know, like loading skills.

10:05Daniel Whitenack:And would this be like, to kind of go off of your point, This would be certainly people are running cloud code or whatever on their local machine, like you're talking about. But ultimately, at least the way I've heard companies express this is to really lean into this element of how the future is going to play out. there will be this digital workforce, however you want to think about it. Agents that are operating and actually taking action within your enterprise infrastructure, right? That aren't tied to someone's laptop. And they maybe then eventually are not just acting, you know, in a single agent type of way, but eventually there are many of these agents that are operating, which I'm sure adds another level of complexity.

10:59Yeah, these fleets or swarms or, you know, the multi-agent architectures that keep coming up, I think they're getting very real now. So, I mean, the companies certainly that we work with are deploying these things. And what ends up happening once you do that is there is no limit to the scale that you can reach technically, right? A disregarding token spend. And, I mean, why would you not arbitrarily scale that to 100 ,000 agent executions per, I don't know, hour, if you could afford it, once that you loosen the yoke of the laptop, right? So I have the feeling that, you know, as things, as infrastructure gets more and more mature and architectural practices get more mature, we can't really predict how much volume of agents would be running once they're not running locally.

11:52And that migration is well and truly underway right this year. And people who get there tend to just it's like this sort of slow adoption. And then suddenly, boom, there's like a crazy increase. And then at that scale, of course, you have different sorts of problems with agents like and I guess that's what we we can spend a bit of time talking about.

12:12Daniel Whitenack:And these agents that would run in your cloud environment, you know, disconnected from your local environment, you know, people might have some ideas of some of the local ones, like a Cloud Code or OpenCode or OpenClaw or whatever we're talking about. Could you give us some examples of kind of the other type of agent? Like what are they built on top of? Like what are these? Do you envision these as mostly kind of software vendors that are building vertical focused agents and they're deploying, you know, into companies or companies building them themselves? I'm just trying to get like a bit of a vision for people of like what these things might be.

13:00So I think this is a big battle in the industry right now, who owns that part of the stack. So, you know, let's start with the most easiest part of the stack. It's the harness, right? So the harness and the model providers have been for the last year tightly coupled. And I like maybe we should spend a little bit of time talking about what the harness is. Yeah, yeah, go ahead, please.

13:23Daniel Whitenack:we've mentioned it on the show but i think always it's like this is one of those concepts that it has popped up and it is often very subtle for people like what we're what we're talking about right it's very ephemeral anyway because what used to be an agent is now a harness yeah so uh the definition i cling on to in um july 2026 is is basically um you know we we we had this a notion of an LLM model, right? The LLM model is simply a token generator. So in itself, it doesn't do anything. It doesn't do action. So the way we did action was we started three years ago with structured outputs and tool calling.

14:02And suddenly, what is structured outputs and tool calling? That's simply imposing policies on what type of tokens can the model generate to predict the next action inside the environment in which it's playing. So you can, ahead of time, you give it like five tools and you say hey um you have get weather or you have like ping hamza or like ping dan and like you have all these bunch of tools to send emails or do different things and here's the definition of the parameters of these tools and then whenever you want you think that you need more information given that description you can return me a bunch of tokens which i can parse and actually map that to my code and execute it and that while loop like that's the agent, right?

14:47And suddenly we started having a program that runs on your machine at the beginning, which mapped those tokens back to the tool calls. And turns out that's harder than it looks. So it's, I mean, I still remember when I did it the first time, like three years ago, it was like, you literally have the, literally do a exec, like an eval in Python, which converts the string into code and runs it. I guess that's still what's happening under the, under the hood. But basically three years of software development later, we had things like CloudCode, which started doing more things while having that while loop.

15:22So, you know, things like compaction when the context window gets too big or things like ensuring that the tool goals are, you know, have the right parameters. Indexing memory. Yeah. Indexing and memory. So that software program is called the harness. So it's basically the thing that gives your brain a hands, like a body. So your brain is just spewing out tokens and you're converting those tokens into actions. And the outcome that comes out of combining the harness and the model is the agent, right? So that's how I think about it. I will obviously be very embarrassed in a year when I'm listening back to this.

16:03Daniel Whitenack:For what it's worth, I mentioned Jeffrey and Noose, like he gave the same, a very similar metaphor of like brain and body. So at least we're in, yeah, we're in safe, safe, safe territory for now until the industry decides to flip this. Yeah. So, so, so, so obviously the most famous harnesses being like some, some interesting phenomena started to happen because what we ended up figuring out is that the models started to asymptote. performance and what the model providers quickly found out given their valuations is that if you just make this program better and you manage context better and you manage memory and all those things general purpose tasks get easier to solve and therefore you could just make a better program right the better program will win given given the model stays static and and then they started a weird rl loop so what they what they did was they like cloud code for instance has a very you know, CloudCode is the harness and underlying it is Opus 4.8.

17:07And Opus 4.8 by now, unlike Opus 3.5, understands what CloudCode itself is. So it's self-aware in the way it's that it's running inside CloudCode. So when it calls a tool like edit file, it actually uses certain parameters and the tool calling is very accurate. If you drop GPT 5.5 into CloudCode, the harness, that would not be as accurate. It wouldn't perform as good because simply the two things have coupled together. So the harness with the reinforcement learning loop that has gone on for the last year and a half has coupled deeply with the models. Now, on the same time, there has been this renaissance of open harnesses, right?

17:50So we had Pi. We have frameworks like Lengraph or Piedantic AI. And those harnesses are of the opposite opinion that we should have an open standard that shouldn't be tied to the model because we don't want to be tied to the model. I mean, if the US government decides to ban and unban models, that shouldn't affect our business outcomes.

18:10And that has been an underlying tension in the industry for the last year and hasn't resolved yet. So I wouldn't know whether, my intuition says that an open harness will win, which standardizes those things. But on the other side, I mean, you know better than most people that, I mean, a cloud code works, right? Why would I use anything else? So that's another thing.

18:32Daniel Whitenack:Yeah, I would guess that there's going to be a mix for some time. You see this also with the existing software vendors who are trying to figure out how they show up within agents, like a NetSuite or a Salesforce or whatever. You see on the one side them officially supporting MCP interaction with their platform because they see maybe, well, the value of our platform is how we manage this data, how we create a functions on top of it, the ability to do actions within this environment and the interface through which people do that, like an agent, really like the value is in the platform and that data, not so much the web interface through which people interact with the product.

19:24Daniel Whitenack:But then on the other side, you see the same individuals, you know, the same companies creating proprietary agentic things in their web, you know, web app, which you can't swap out to your point. You can't swap out the, you know, in many cases, you can't swap out the underlying endpoint to change out the brain to your point. Yeah, because the brain ultimately is commoditized. It's like electricity. So what else are you going to make the next whatever? a trillion dollars on um and and that's the whole stack right and we we haven't even gotten to the point where so this is all local right so this this revolution is happening december 2025 we come into this era of suddenly this works right excuse me for the language um and then it's like okay so what has started working ah so we have pi and we have open claw and the innovation is that it's running all the time and it has a heartbeat and it's like it's managing memory somehow better and okay, what did we actually change?

20:22Oh, we changed the RNS. So somehow like Peter Steinberger and Mario and Armin, like those guys, they had better ideas how to manage that program, which just caught fire. And yeah, and you know, we haven't even gotten to the point where let's get these things out of the computer and into a Kubernetes cluster or something.

20:42Daniel Whitenack:Yeah, yeah, super, super interesting. Sometimes the gap between AI-generated ideas and production-ready work can be really frustrating. Agents can produce things that aren't editable and don't live in your core workflows. That's why I really love what our partner Framer is doing with the way they're integrating agents into their website platform. Agents work in the same place where the real site is designed, managed, reviewed, and published. It lands on the canvas, stays editable, and can be published when the team is ready. Framer is a complete website platform, not just a builder. So teams can launch and keep improving their sites in one place.

21:25Daniel Whitenack:Learn how you can get more out of your website from a Framer specialist or get started building for free today at framer.com slash practical AI for 30 % off a Framer Pro Annual Plan. That's framer.com slash practical AI for 30 % off. Framer.com slash practical AI. Rules and restrictions may apply. Well, Hamza, it was really good intro to, I guess, just how to think about or categorize some of these things and the shifts that we're seeing in our mind. Let's go under the assumption that, you know, maybe there's even people listening to this podcast that have envisioned products that have agents in them.

22:11Daniel Whitenack:They're maybe building agents that they intend to sell into the enterprise, or maybe it's part of people in a larger company that are building their own agents for operational efficiencies or internal tools. And they are picturing that future that you described, which is, hey, I actually want these things to run off my laptop. I want to be able to take my laptop, go into a meeting and not have to keep it up all the time for my agent. So in that sense, then, assume we have some of those agents running in that type of environment. What are some of these, I guess, fragilities or points about agents in terms of how people are architecting them now, how they're building the agent harness piece that make them fragile, not durable?

23:01Yeah. So I think first, we have to understand the different types of agent workloads that can run in the background. So if you're talking about a chatbot interface, or a voice agent, that's largely different from a personal assistant that's running or a deep research or an auto research that's just doing a single loop and always on and achieving some outcome. So I think that for all of those different types of use cases, there's different infrastructural pieces and different ways to think about it. So I would say that if it's a simple agent that's running online somewhere, I think that I would personally default to something which is very easy, whether that's using the model providers like Anthropic has Anthropic managed agents.

23:54they have inbuilt like routines and all those things inside the harness now that you can that you can deploy and they take care of all of that infrastructure for you. Now when it starts getting more complex where your workflows start looking a little bit more business process-y meaning you have a workflow which is doing things like okay fetch an order here and do some processing with an LLM which is an agentic loop here and then do some post-processing there. So then it starts looking a bit more complicated. And if you do that at scale, then you might want to own that infrastructure. So the first thing you do is sort of have an agent platform internally as an enterprise.

24:34And this, I highly encourage people who are scaling beyond single teams to really invest in. Like similar to MLOps, where the people who build the best MLOps platforms like Uber, they won their markets. I'm not saying the MLOps platform was the reason Uber won the taxi game. But it was a big reason that they made, you know, things like search pricing and, you know, they were the best product out there in the market for a while. And I think the best companies will be the ones that invest internally to build platforms that can run these things at scale. Because simply the act of doing that informs you so much internally of what works and doesn't work for your particular business context.

25:15Yeah.

25:16Daniel Whitenack:And in that, I guess the set of things that you need to have in place to support that, let's say I did want to go down that path, I want to own it. Like, what are some of the main sets of things that like need, I need to be thinking about, I guess? Yeah. So the first thing is you need to pick your harness or you need to give your teams the ability to create, you know, using an agentic framework, whatever, let's say you're using LandGraph or PyDantic AI or something like that. Then you need to deploy it onto some compute target. Let's say you picked something, probably you have other applications running.

25:54So probably you use the same thing, whether that's ECS on AWS or whether that's a Kubernetes cluster running somewhere. And now you have an API, so you have a REST API. So I'm more of a Python guy, so I'm going to give Python analogies. So let's say you have a fast API application. You have a post REST call that kicks off an agent. And obviously, the first failure mode that will happen is that, you know, you can, these things are not quick requests, RESTful, that just execute in milliseconds. These are very stateful processes that run for a long time. So they can't run in process, right? This is where you start to farm out things architecturally to things like workers and task queues.

26:36And when you start thinking infrastructurally like that, then you need to think about, okay, what is a task queue? Is it a pull-based system? Is it a push-based system? So this is just where in the realm of how do we actually execute this so that at a certain scale it keeps working? then obviously you have this a very simple task you would be so imagine you have an API and you just directly from the API you spin up a worker like a salary worker well you don't want to be doing that because what if the worker goes down because at scale the workers can go down for a number of reasons like network failures or maybe the pod didn't exist at that point because somebody was using your underlying compute for something else.