Practical workflow orchestration

27:14So if you have something that's working locally, you don't have to be like a DevOps engineer to get it working on Kubernetes if that's your flavor. But honestly, if you're like, look, I've got this thing working in this Docker container locally on my machine. I want to get it working on Amazon's ECS. It's really like a one-click deploy in order to get it working somewhere else. And the last piece is around observability. We haven't talked about it, but observability is really this pair that comes with orchestration, where if orchestration is really the practice of like, crap, how do I get this stuff to actually run and how to react to it when it fails?

27:52There are some times when it just cascades and fails all the way down. And no matter how much you account for everything, everything goes wrong. And you need to be able to like, see what went wrong so that you can replan, right? Like, you're not going to be able to handle the entire universe's amount of complexity ahead of time. So when it does fail, you need to be able to learn from those mistakes as a human being and actually write in the logic to handle that for next time. So observability is really this element of this thing failed. How do I provide the breadcrumbs to figure out why it failed?

28:25Which data provider was the one that caused me to fail? Was it a parsing error? How often am I seeing that parsing error? If it's OpenAI's fault, now there's a bridge between observability and orchestration. If OpenAI has failed more than 80 % of my requests in the last 10 minutes, now I'm going to switch to Anthropic and we make it easy to switch those two things out for each other. And so I would say that between just adding your most native retry caching transactions, making it dead simple to submit to infrastructure, and then the last piece around really having a clear understanding of how things are working, and importantly, how to figure out when they fail, those are really the core, I'd say, value adds of Prefect.

29:07Yeah, and I would definitely recommend people check out the Prefect docs. There's a quick start in there. Just to give people like a visual of this, I see you have, you know, this kind of converting your Python script into Prefect workflow type of thing. That's really helpful. there's sort of these definitions of like a workflow related to getting information about a GitHub repository, like contributors and repo information. And then, you know, those are in Python functions, you know, like definition, get contributors function. And then above that, you kind of put these decorators, task and flow decorators with options to convert this into a Prefect workflow.

29:52So part of my thought on, you know, follow up on that just to kind of really hone in practically to give people a sense of what it's like to do this. So let's say that I've, I've converted my Python code into some, you know, one or more Prefect workflows. I can run that in my understanding, if I'm looking at the docs, right, I can run that just Python, my workflow. But obviously, like you said, there's a deployment element to this. So could Could you talk a little bit about, you know, I know one of the things that's always a struggle in my past is maybe getting something working locally. And then because I'm going to run it in a different environment or something like that, you know, then I deploy my workflow on top of other infrastructure, right?

30:42And it's hard to just connect that like local development, debugging and development staging production environments. So practically, what does that look like in the Prefect world in terms of going from that Python command to run your workflow locally to, you know, watching things flow through your workflow and a nice dashboard that you have pulled up in production? So I'll do my best to do this without a screen to share. Yeah, it's hard. So you had said, talk to me the experience of taking my Python code and converting it to a Prefect workflow. For folks that are listening, it's as simple as from Prefect import flow and then at flow on top of your function.

31:27That's all the work it takes to take a Python function and give it superpowers, basically, to add all the stuff that we've talked about. And that doesn't take away any sort of attributes of that function. If I take my hello world or my ETL flow that runs locally and I start adding prefix things on top of it, it still runs locally. And so it's not like we don't sort of require additional infrastructure just to have your stuff run. We detect whether or not you're running it locally or not. And if you're running it locally, we execute it as if it's regular Python functions. Now, there's kind of two stories of, well, this thing is working on my machine.

32:09How do I get it to run somewhere else? We have two ways of doing this. And I'm sorry for the vocab lesson, which is, let's say that what's the crawl, walk, run for anybody trying to get something to run remotely? it's you start off on your machine and then it's like well i'm gonna go get a server somewhere and i'm just if i can close my laptop and go to bed and this thing is still running on my server that is what counts for me as remote as running it remotely uh that's like you know the next step in in the hierarchy of needs so for that if it can run locally on your server right like you You go to EC2, you SSH in, you've got your console, you pull your GitHub repo, you pip install your requirements, right?

32:54You're still able to execute that flow on your machine. And now if you execute that function, you hit, say, like.serve. Now in the same way that you would like start a fast API application, it is now running. You can specify a schedule that it runs on. it exposes an HTTP endpoint that you can call out to if you want to invoke it on demand. And it listens to events. So maybe you don't want to just run it on a schedule. Maybe sometimes you just want to be able to hit a specific endpoint to manually invoke it, maybe from your Django app or something like this. Or maybe you want to emit an event into the world that says, look, when my user signs up, when my data set is ready, I want to have this ETL flow go and operate on it.

33:40Because my data is not always ready at 6am. Sometimes it's not uploaded till 6.02. So sometimes you want to have things be a bit more dynamic. That's what happens in a flow.serve world. So where are we in the hierarchy of needs? I had it on my laptop, but it stopped running when I shut it. I go and I get an EC2 instance. I have it running there. Now I can shut my laptop. It's still running? And then now how do you do this at a massive scale? And how do you do this auto scaling with respect to the amount of work that you have, right? If your EC2 instance, you hit it 10 ,000 times, and all of those things, you know, every invocation requires downloading some data, that machine is going to fail over quick.

34:18So what happens when you want 10 ,000 things to spin up just as many machines and you want to fan out, that would look like my flow.deploy. So I'm here my laptop. I've got my hello world flow. I would literally just say, if name equals main, right? Like I'm writing any other script, flow.deploy. And the sort of autocomplete in your IDE is going to tell you like, all right, what do you want to name this thing? Where do you want it to run? Do you want it to run on ECS, in Amazon? Do you want to run in Google Cloud Function? Where do you want it to run? Can you point us to, do you have this in a Docker container somewhere?

34:53If so, we'll just like pull it from whatever Docker registry that you have, or do you want to point us to a GitHub repository? And one of the nice things is when you build a tool like Prefect, that is treats failure as a first class citizen, you see the failure modes of tens of thousands of people trying to get their code to run. And so this really allows us to build kind of an ergonomic experience that's like, look, here's the minimal stuff that we need to know in order to have like a high reliability guarantee that this is going to work when you shut your laptop, when you submit it to a Kubernetes cluster, what have you.

35:26So the experience is really like pretty dead simple. It is, if it's running locally on your machine, you can hit.serve. Now it starts a process on your machine. Or you want it somewhere else, it's.deploy. And we can guide you through exactly how to point it at remote infrastructure. Does that answer your question, Daniel? I hope I'm not glossing over anything. It does. Yeah, yeah, that's great. I just have always felt this as a pain in my own life. So I often like to ask the selfish practical questions on this podcast when I get the chance for them. Yeah. One thing that you had said, which I did gloss over is like, you had said, what's the experience of deploying?

36:07And then like, what's, you know, tell me about like a shiny dashboard that you see or something like this. So once I go through all the work and I have my workflow running out in the world, I have this really beautiful UI that I can log into. And when I access that UI, what it displays for me is it basically tells me like it's almost like standing on the platform above the factory floor, right? You can see everything that's in progress. You can see the box of widgets that everything has produced at the end. And you can see everything that was broken in process. And so if you're the type of person that treats your workflows as cattle and not as pets, it's very easy to see like, yeah, you had 10 ,000 jobs that succeeded.

36:47Here's the 10 that failed. You can easily click in, see, OK, the 10 that failed. Why did they fail? If you're on Prefect Cloud, which has a very generous free tier you can run a business on, we do AI summaries of the errors. And so we will summarize like, look, these workflows are failing for these reasons in natural language so that you don't have to dig through like a 10 ,000 line stack trace just to find out that you had an out of memory error.

37:25well there's no shortage of helpful ai tools out there but using these ai tools means you gotta switch back and forth back and forth between yet one more tool so instead of simplifying your workflow it just gets more complicated but that's not how it works when you're using notion notion is the perfect place to organize lots of stuff tasks tracking your habits writing beautiful docs collaborating with your team knowledge bases and the more content you add to notion the more this cool thing called notion ai can personalize all of the responses for you unlike generic chatbots notion ai already has the context of your work plus it has multiple knowledge sources.

38:09It uses AI knowledge from GPT-4 and Cloud, and that helps you chat about any topic. And here's the kicker. Now in beta, Notion AI can search across Slack discussions, Google Docs, Sheets, Slides, and even more tools like GitHub and Jira. Those are coming soon. And unlike specialized tools or legacy suites that have you bouncing between different applications, notion is seamlessly integrated infinitely flexible and beautifully easy to use so you are empowered to do your most meaningful work inside notion from small teams to massive fortune 500 companies these teams both small and large use notion to send less email cancel more meetings save time searching for their work and they reduce spending on tools which helps everyone stay on the same page you can try notion for free today by going to notion.com slash practical ai that's all over case notion.com slash practical ai to try the powerful easy to use notion ai today and of course when you use our link you're supporting our show and i know you love that again notion.com slash practical ai

39:43So, Adam, I want to ask you a question. I understand that you have a friend named Marvin. I'm wondering if you can tell me a bit about Marvin. Yeah. So Marvin means a lot of things at Prefect. Prefect, if folks don't know, is an homage to Ford Prefect from Hitchhiker's Guide to the Galaxy and the genius alien that, well, I think Marvin's a bit more of the genius, but the alien at the beginning of the book that's just like, come on, let's get out of here, introduces him to the world or to the universe. I just reread that last week, by the way, just completely randomly there. I just had to say that.

40:21Amazing timing. Yeah. For folks listening, this is not like paid actors, I promise. And then Marvin is his like terribly depressed Android companion that's like just kind of has the curse of knowledge, knows everything. Everybody hates me. Exactly. And so Marvin started off as a mascot at Prefect. And we were first to ducks. So we had Marvin the Duck, which was this lesson of like, often the fastest way to triage failure is by rubber ducking. So we would send like ducks off to new customers to put on their desks so they could literally rubber duck with Marvin. And then Marvin was the name of our first internal like LLM powered Slack bot, which is still alive today.

41:09It serves a community of 30 ,000 data engineers that show up in our community Slack every day who have questions about Prefect. So, you know, if somebody is trying to configure particularly complex behavior, maybe that's a blind spot in our docs. Marvin is hooked up to all of our docs, all of our GitHub issues, everything that's ever been written about Prefect. Marvin is hooked up to. And so in Prefect today, if you're a community user, you can show up and you can say like, hey, Marvin, I saw this stack trace and I thought I'd configured this correctly. What's going on? And Marvin would be like, oh, well, actually, if you configure it like this, here, maybe you want to check out this doc or this video.

41:50And so Marvin has really created this personalized interface to learning prefect that docs never can be, right? Docs are always kind of shooting for like the middle, I don't know, 80 % of technical prowess, the middle 80 % of use cases. And LLMs, just as they've done for pretty much every other product, create a human interface to PETA entry, right? So it's somebody can show up as they are and get a personalized interface to our docs. The last piece is Marvin is our opinionated LLM framework. So Marvin is incredibly popular project. It's got 5 ,000 plus stars on GitHub. And it got started in early 2023, where we had just built our internal Slack bot and we had felt that like existing options in the world didn't allow us to write LLM workflows in a way that felt particularly natural or ergonomic to us.

42:49And as a company that more or less was like, look, we think that writing dynamic workflows and configuring infrastructure, those are hard problems, but we should take on creating a good ergonomic interface to them And so you don't have to face that complexity. Prefect is obsessed with giving people access to sort of the complexity of the world without having to face it themselves. They can opt into it whenever they want, but you shouldn't have to have like a CS degree just to get a script running on another machine. And similarly, you shouldn't have to like, you know, import a thousand agents and a thousand different data loaders.

43:27And you shouldn't have to learn like a new common expression language just to get LLMs to do the work that you want. And so Marvin is like, you've got a Pydantic model. Guess what? You decorate this thing. And now this thing that was responsible for validation can now, you put a document into it, and now it can extract data from it. You've got a typical Python enum. You decorate this thing. And now you've got a classifier. You've got a function. You write in a doc string that says, here's what this function is supposed to do. Now you can get strongly typed outputs because we're just reading from the function signature, its return annotation.

44:05We're basically trying to figure out how to make writing LLM workflows feel very Pythonic and very ergonomic. And it's been a blast to build and it's just welcomed so many AI engineers into basically our ecosystem who saw a really sane way of writing Pythonic LLM workflows that they just didn't have in any alternatives. And I guess if we take that maybe a small step beyond, I know that you also have control flow as another piece of this stack. Could you kind of highlight maybe the differences, overlaps kind of distinction there? I know it's really interesting, I think, what you said before in the sense that a lot of people are trying to build these agentic workflows.

44:52And a lot of those things are very flaky and hard to debug. Right. That's the main thing that I hear in the main, which is often why I think people are decomposing their agentic workflows into maybe static kind of workflows that aren't really agents anymore. more, but they can be debugged and, you know, they can check errors and that sort of thing. So yeah, I would love to hear about your kind of approach on the agentic side, given the sense that you're treating those sort of failure states as first class citizens, that sort of thing. Yeah, yeah. No, so I think one thing I want to talk about is like, what's the value of agentic workflows?

45:29I feel like a lot of folks just hop into it being like, cool, now I can make this thing agentic, but for what reason at all? So just like little context here is like, I truly believe that LLM workflows and just to make sure we have the same vocab, it is like very deterministic workflows that say, first, I'm going to call out, I'm going to extract this data. If I see the extracted data, now I'm going to classify this piece. And then if that now I'm going to, I don't know, send an email. Those are things where the human is really writing the logic. If we were to take this workflow and we were to go back to like 2016 or 2017, it would look the same.

46:08But instead of calling out to an API, we would say, now I'm going to have like some hidden Markov model that's going to go and do my entity extraction. That's my step one. If I detect this entity, now go to my next thing. Now I might have some like interpretable logistic regression model, try and classify the output coming from it before. So LLM workflows are really like, how do I take traditional workflows that existed in the world of ML, but now instead of having to go and train models, now I can essentially wish models into the world through natural language. And those workflows are amazing.

46:44That's like, frankly, I think that they solve most business problems that you need to. That's like a very reductionist, like broad brush that I want to paint with. And they're incredibly easy to debug. They're incredibly easy to observe. because the logic is something that you wrote. If your first if statement fails or something like this, now you're able to say like, well, I knew exactly where in my logic this thing failed. And so now I can go debug this tool that I called out to. It basically, it's easy to debug because it's not changing our mental model of how to build data pipelines. And so all of the tooling that we built in the last decade around like building robust evaluations, collecting data to try and figure out how often they're succeeding, these are tried and true.

47:29So we're able to basically like do our own bit of transfer learning from, you know, decades of ML research and evaluating ML pipelines onto LLM pipelines. Now, there was a paper that came out from Alpha Kodium a few months ago that was basically like, look, if you want to do, say, like build your own automated software engineer, there is only so far you can get with like, say, deterministic LLM workflow that's built on top of like GPT 3.5. They were able to show that, you know, even though it's all just kind of like function calling under a big while loop, right? They were able to show that if you adopt this paradigm where you start turning over some of the planning logic to the LLM itself, now it's able to outperform the like top of the line frontier models on specific tasks.

48:17So for things where the decision space is very large, like what piece of code do I want to run next? Stuff where the information space is very large. I've got a giant code base to reason about. Agentic workflows really tend to thrive because if the decision space isn't known to you at the very beginning, agents are able to discover that dynamically, create and refine on a plan that you as a human, it would take a lot of ifs to get to the same style of performance. So that difference between LLM workflows, which are basically prompt engineering, calling out, calling a function, single shot stuff for single shot tasks.

48:54That's what I call LLM workflows. The stuff where you turn over behavior or orchestration logic to an LLM itself to decide its own plans, maybe even create its own tools, that's agentic. And that's really that distinction in problem space is also how we distinguish between Marvin and control flow. I promised I would get to control flow. So Marvin is really good for saying, like, you want to call out to an LLM and extract something. We're going to make it feel the absolute most natural that you can. And we're going to make this so that if you know Python, you know Marvin. If you know Python, you know how to extract something super easily.

49:29If you know Python, you can classify something super easily, summarize things super easily. So Marvin is really a prompting library. It's a way of creating dynamic prompts using classic Python functions or Python objects. Control flow is really now when you start turning over the wheel to LLMs to formulate their own plans, this is where we can't rely on the tools of the past around observability. we can't rely on the tools of the past around evals you can still build evals that say like i put in this thing at the beginning i got this thing at the end how often did it succeed on my test set but if you're trying to figure out like how often did this thing take the specific path through reasoning space to get to this outcome that's uh now much harder to design tests around and it's harder to debug if something fails and you didn't write in the orchestration logic you You can't really point to a place and say, ah, this is where I messed up in how I designed this thing.

50:24And so control flow is really how do you ergonomically express agentic workflows? How do you express dependencies between tasks? How do you tell an LLM when it has the wheel? And then how do you explicitly tell it that it no longer has the wheel? And now it's going to cede back to human written logic. That is what we make it extremely easy for you to express in control flow. And since we have a pretty opinionated developer experience of how to express this, we're able to give folks much better observability. And so this is built on Prefect3, which means that you have an orchestration library sitting behind every single action that an LLM takes.

51:09If something fails, you've got retries. If something takes too long, you've got timeouts. If something fails and you don't want to go through all the work again, you've got caching. if your agent needs to create a sandboxed code environment to run untrusted code, it can now deploy its self-created function to remote infrastructure. So it's really how do you give agents durability and resiliency? Because those are often the biggest reasons that they fail. And so that's how we distinguish between it. But I realize I'm monologuing a little bit. So I'll turn it back over to you, Daniel, if there's some stuff that you want to double click on.

51:46It's all good. I'll jump in. We've covered some ground here, and I appreciate that very much. With Prefect Core producing the open source that you're building around, you guys have the managed workflow orchestration platform that is Prefect Cloud. We've talked about ControlFlow. We've talked about Marvin. That's a lot. What are you thinking about next as you're looking in the months and the next few years to come? you've accomplished so much, but as you look at what you think the space is going to turn into, because it's evolving so fast, do a little bit of crystal ball gazing for us. Right or wrong, tell us what you think the future is going to hold and how you'd like to play into that.

52:33Yeah. So it's a great question. And it's something I obviously have to try to think a lot about. I would say that right now, there is way too much emphasis on single machine, local LLM or agent workflows. Here's what I mean by that. It's like you pull up any framework in the world to run LLM things or to build an agentic workflow. And it's like happening in a local process on your machine. it spins up like an input field in your terminal, and then you have to like type your answer to what your favorite color is. And then it goes and writes you up like a poem or something like this. But businesses who are building LLM workflows or building on agents at the model level, we see that frontier models are accounting for, sorry, frontier model providers are accounting for a lot of the core sources of failure.

53:29So structured outputs from OpenAI managed to wipe out just a whole host of your classic resiliency issues. And the resiliency issues, I think, are going to be around planning. And so the ability to add item potency and transactions to LLM workflows is really what we would invest more into. You had a plan that executed 10 things in a row, you found out at the 11th step, the plan was bad, but you've already done 10 things. How do you walk that all back? You would do that with transactions. Where I am really interested in long-term is when Daniel asked earlier, like, talk to me about how I as a human go from like a locally running function to something that's running elsewhere.

54:14I think that the me as a human part of that is going to be, is to go away, right? And that it's often going to be an LLM in the course of solving a problem decides that it needs to create and massively parallelize a function to call out to. And so now you need to be able to give an LLM on demand the ability to create and provision infrastructure and submit a whole bunch of jobs to it. And so while we've built Prefect to be something so easy a human can understand, that's going to play into a lot of strengths of how do we expose an API that can really be taken advantage by an LLM, if that's the intended audience of how to provision infrastructure.

54:55The third piece is at companies now, you'll have so many teams that are basically building LLM workflows in parallel to each other. So you'll have like the conversation team that's like trying to build in LLMs into how it talks to customers. And then you'll have like the platform team, which is trying to use it to like give feedback on internal pull requests. You'll have teams that are trying to use a bit more of like, you know, in human in the loop for like commenting on like a design doc, something like this. But fundamentally you have tens of thousands of parallelized executions or calls against LLM APIs.

55:36So how do you solve that coordination problem across the programs that are trying to invoke LLMs. And so really trying to figure out if I have 10 ,000 agents all trying to access the same resource at the same time, how do you govern that in a way that doesn't cause them all to crash? If I have 10 ,000 engineers that are all trying to access the same LLM API, how do I make sure that they don't all consume the same token budget all at the same time? So that's a bit more on the practical side of like, you know, future of agents, future of LLMs, Aside, there is still a very, very interesting fundamental engineering question, which is how do you get tens of thousands of concurrent API calls to OpenAI or Anthropoc all behave sanely and not lean into a commons problem of everybody cannibalizing the same data resource?

56:31And so that's like the very boring thing that I like to think about it. But orchestration is one of these like very fun, but ultimately like worst case scenario disaster planning style things. Well, we love the boring stuff here, which is actually not boring. And I think a lot of people want to think more about because they do have the desire to put these workflows into production, right? which is, yeah, of course, on your all's mind and what you're digging into. And yeah, just really appreciate you taking time to join us, Adam. It was a great conversation and I would encourage people, we'll include the links in the show notes, but go check out the docs for Prefect and Marvin and Control Flow.

57:15Try out some of these things on your own. Like Adam said, you can pip install Prefect and be off to the races. So no reason not to try it out. And yeah, thank you so much, Adam, for taking time and joining us. My pleasure. Thanks so much, guys.

57:37All right. That is Practical AI for this week. Subscribe now. If you haven't already, head to practicalai.fm for all the ways. And join our free Slack team where you can hang out with Daniel, Chris, and the entire ChangeLog community. Sign up today at practicalai.fm slash community. Thanks again to our partners at fly.io, to our Beat Freaking Residence, Breakmaster Cylinder, and to you for listening. We appreciate you spending time with us. That's all for now. We'll talk to you again next time.