993: How to Build AI-First Organizations, with Jacob Miller and Jeremy Mumford

23:17Jon Krohn:That was crystal clear, Jacob. Such a nice response. So nice, in fact, that it's the first question I've asked that now both of you has chimed in on. Jeremy's just stunned by how perfect your answer was. It was perfect. No need to jump in. Um, later on in chapter three, you guys make the claim that you should build the hands before the brain. So the workflows before agents, which is an interesting claim to make because today it seems like we're hearing everywhere that we should just be diving right into agents. So what are your concerns about people in our field, skipping what you view as a foundational step, getting the workflow right first.

24:00Jon Krohn:And maybe this is actually a good opportunity to also compare and contrast the differences between a workflow and an agent. Yeah. The fundamental difference between a workflow and an agent is the orchestration. For an agent, you give it an objective, you give it a set of tools and you say, use the tools in the correct order of whatever you think is right to achieve this objective. In a workflow, you say, use tool A, then B, then C, and then maybe there's a branch in that tree. and you're giving it the path already. And so one of the points that we make is if you can't get a workflow operational and being happy with the results where you're telling the exact path to give, why would you be any more happy with an agent that's just got to make it up on the fly?

24:44Now, with that being said, there is this natural progression that people are using right now where it goes like skills, then workflows, then agents. And so you use, instead of mapping out a process, because that can be tedious or complicated, or I don't really know process mapping because you're trying to get subject matter experts involved here. They can create a skill real quick. And it's just a markdown file. Describe it. They can use the skill, see how well it performs, and then update the skill and iterate. And that becomes this natural landing point for a lot of teams. We're just going to build skills.

25:16At a certain point, you say, why call these skills manually? That doesn't really make sense. Turn this into a workflow, and then you can get them into a workflow. It's then taking the job. And then if you need to dial up the capabilities, you transition from workflow to agent. And so that's the natural path that we see at a lot of companies right now. It's start with skills, very accessible, very simple. You then take those skills, package them up into a workflow. And if you need it, roll it up into an agent. I think there's a few other problems with helping people understand this distinction between workflows and agents and helping them understand they should start with workflows.

25:52One is the boundary is very fuzzy. Right. You can have agents calling workflows, workflows calling agents. It can be fuzzy on where exactly the line starts and ends with which one is which. And then I think the other thing that's challenging is the industry naming products, naming techniques. You see a lot of stuff marketing itself as an agent builder, when in my personal view, I view them more as workflow builders. And so it can be very hard then to try to like go and explain this to people and say, oh, congratulations. you built a workflow and they're like, well, I thought I built an agent. And you're like, well, I guess we can call it an agent, but it's, it looks like a workflow and it talks like a workflow.

26:34So let's call it a workflow. And like, I am very pro workflow. I think there's so much power in having a deterministic process that you can rely on, you can count on, and it makes things faster, more reliable, and often cuts costs. Think about like setting something up with cloud code that's just on a routine that everything is orchestrated by the agent. It's all passing through the context window. I can get really expensive. There's a lot of power in building out a bash script, you know, and you can have the agent write the bash script for you. But like, do we have to have everything go through the context window?

27:09Does the agent have to handle everything? That's where you start to see these horror stories of people saying like, it went off the rails in week three and deleted my production environment. It's like, let's stick with deterministic workflows first. Once we've kind of built out our skills and we're confident, then we can kick it up to the next level and go tackle the kind of craziness of non-deterministic unpredictable agents.

27:34Jon Krohn:Yeah, this makes a lot of sense to me. Having an LLM involved in a workflow that you could imagine, how you could code this up as something more deterministic is a really bad idea. From a cost, from a latency perspective, from a risk perspective, I think there's a risk for those of us working in data science and AI that we can kind of end up in the situation where instead of kind of typing that really short email and just saying, yes, that sounds good, we're like copying out of Gmail into Claude or using Claude Cowork to be coming up with an answer to the simple thing instead of just typing it up ourselves.

28:19And so because LLMs are so widely useful,

28:24Jon Krohn:we can end up throwing them into any situation within an organizational workflow when it's serious overkill. And yeah, we end up with all of these downsides with cost, latency, risk. So try to use your brain still, people. It takes effort. And if I can share just a quick story, I like from just an example. I remember we were working on this project and it was all AI, all workflows, just a huge mess of AI everywhere. And we kept getting user feedback that they were not happy and we kept fine tuning the prompt, fine tuning the prompt. Finally, I just turned around and I said, OK, we're just going to rip the AI out of this part.

29:01We're just going to replace it with some canned statements and user complaints disappeared overnight. We just replaced it with a decision tree of like, if it is above this score, we'll say this if it's below the score we'll just have some sentences where we fill in different variables and user satisfaction just like shot through the roof and so i'm like i feel i feel like half my job as an ai engineer sometimes is ripping ai out of places and people should not be afraid to do

29:26Jon Krohn:that sometimes really great example you guys have had so many concrete examples so far in the episode it's been a delight and and so easy to understand what you're describing when you give these clear examples. You mentioned there that you're an AI engineer, but you talked earlier in the episode, Jeremy, about how you were previously a data engineer. And so this next question is related to that specifically. You can tell me about Titan logistics. You can tell our audience about the Titan logistics story. This is from, we're kind of now in the middle of your book. We're like tackling chapters four to six here out of, I guess you have 12 chapters in the book.

30:02Jon Krohn:If I'm remembering correctly, I'm getting some head nods. So yeah, we're getting to kind of the halfway point in the book and the halfway point in the episode. Wow. And so, yeah, tell us about the tight logistics story where they were losing money on both the hedge and the physical supply chain at the same time until an AI firm asked for the boring data. So fill us in on more about that. But I think this is an interesting example because I think this happens a lot where organizations are looking for advice on AI, but what they really need is an intervention on their data engineering. Yeah, I think this is such a great example that we have in the book.

30:44And it's so powerful because I feel like a lot of people in data science have maybe already felt this story in their personal lives, but maybe people getting into AI engineering or discovering it for the first time, I feel like I've heard so many stories from people in the data science industry, where they talk about, I got hired at X company, and I was asked to do data science. And I walk in on my first day, and I ask, where's the data? And they go, what data? And they dive into the company data, and it's a mess. And they spend the first two years of being a data science doing data engineering.

31:19And I think this is the big key thing like you have to have solid data in order to make ai work effectively i think data science has kind of figured this out over the years and now we're hoping that we can help kind of pass that lesson on to ai engineering which is that like data is the foundation of everything we've got a quote that i really like in the book uh from the movie uh ron burgundy like uh anchorman

31:48Jon Krohn:and it says from what was that what was that Shakespeare play oh yes Ron Burgundy yeah yeah so Anchorman Legend of Ron Burgundy we've got a quote in the book that says to the to this effect of remember whatever you put on the teleprompter Ron will read it and so whatever data you put in front of your AI they'll take it as truth and so data engineering is critical with AI because the data you put in front of your AI is what they will take as the truth And so really, a lot of AI engineering starts to fall back to data engineering. The difference between before and now, though, is previously we were dealing with structured data.

32:27You know, in the case of Titan Logistics and our data science friends, it was a question of where are the numbers? Where's the data? Now, with this new generation of AI engineers, it's a question of where are the PDF files? Where are the images? Where are the Google Docs? And which ones are authoritative? Which ones are the truth? Which ones are throwaway junk? How do we chunk it? How do we embed it? How do we search it? All of these questions are now forming the bedrock of AI engineering in 2026.

32:57Jon Krohn:So yeah, thank you for that clear example, Jeremy. And this actually brings me perfectly to the next question that I was going to ask about. So you talked about how with that anchorman legend of Ron Burgundy example, where the agent, and LLMs in general, they know what they know from data that they've been trained on. And so you argue in your book that hallucinations are usually gaps in what you call canon, so the body of information that an AI system accepts as authoritative truth, not flaws in the model. So the AI is only improvising, this is a quote from your book, because we fail to give it the right script.

33:41Jon Krohn:And that's an important reframe. How do you get engineering teams to stop blaming the model and start treating hallucinations as a data curation problem? With hallucinations, it's often, like you said, this categorical error that we see over and over from AI engineers, where if the model hallucinates, they just assume that the model messed up, when it's most often for the model to hallucinate when it didn't have any good reference. If you tell the model, like, I live on the third floor, what floor do I live on? It doesn't say you live on the second floor, but if you haven't mentioned what floor you live on at all, and you haven't given any of that context, and then you ask it, it might make something up, or it might try to infer from information that's somewhat unrelated.

34:26And so when you look at a hallucination with modern models, at least like in, you know, 2026 models, what that indicates is actually we didn't provide any good context about that question about that task. And the fact that it hallucinated is actually throwing up a flag to say, we are missing critical context here, which means we either don't have that context at all, or we failed the retrieval entirely saying that was out there, but we just didn't go out there and grab the right one.

34:54Jon Krohn:It kind of reminds me how earlier when you were talking about Zillow offers and the world changes. And so, you know, the model that was trained is no longer relevant to the world that we're in, it kind of seems like something similar can happen here with context where you could be, say, you're an HR manager and you're like, sweet, we're going to get this agent set up so that it can handle all the common questions that I have to deal with every day. I get dozens of emails every day on simple HR questions that people could be looking up in a guide. So let me throw all of the HR PDFs that I can find, all the guides that we have, I'll just throw that all in as context.

35:36Jon Krohn:But within that context, there could be conflicting information. Maybe you've thrown in an HR guide from this year as well as one from two years ago, and a policy has changed over that time in some significant way. It's now the inverse of what it was two years ago. But maybe this document didn't have a date on it. And so the AI model, it might pull out that information from two years ago and confidently give an answer that is the opposite of what's true today. And so you guys, in your book, you talk about a, you suggest a new role type in the organization that I had never come across before, I guess, because you guys invented it, which is a role called Canon Manager.

36:24Jon Krohn:And so this Canon Manager is making sure that the contacts that AI agents are working with, or LLMs, I guess, in general, are working with in an organization are correct. Yeah, the Canon Managers in general, like we came up with this term, but there are many, many different titles for it across the industry. And every organization is just kind of sprinkled their own and so it's like well we'll just consolidate them into one single title of canon manager of the person who is in charge of what is true not because it can't change not because that document can't change but whatever you tell burgundy is true is is true he will read the prompter he'll read the teleprompter so the technically not not ron burgundy though is it is it ron burgundy in it that does that no it's another character right well i think someone pranks him, putting it on the teleprompter.

37:17Oh, right, right. And then he incorrectly reads it.

37:19Jon Krohn:So it is Ron Burgundy that does it, yeah. Somehow I was thinking, you know, it's been probably 20 years since I watched that movie. I was thinking it was the I Love Lamp guy that would read whatever. My bad, my bad. I would say that the anti-pattern of a canon manager, so to describe what it's not before to describe what it is, is let's say there's critical context for an agent to behave correctly. And I would say the most common mistake that we see is that it's put somewhere in GitHub in a repo saying, hey, you should follow these exact instructions that are within this repo and you will do what this engineer has placed there.

37:56And then if it goes wrong, that GitHub repository is not very public within the company. It's not very public to say the product managers or others who are responsible for its performance. And if they want to update and change it, it's up to the engineer. So someone would say, hey, it's up to the engineer to make sure that this context, that this canon is correct and up to date. But we argue, no, they are not the subject matter expert. They're not the one who knows the most about it. And actually you have to extract that context out from the deep recesses of some GitHub repository and move it over and put it in front of the subject matter experts that actually know the most about it and provide them the tools to minimize that friction and proactively identify those conflicts so that they are accountable and responsible and they feel that they are the ones to make sure that this agent or this workflow performs the way that it should.

38:48Yeah, so it boils down to how do you make it really easy for anyone at the company, people in HR, people in leadership, people in engineering, no matter what your role is, how do you make it really easy for them to publish and maintain and update knowledge that is accessible by agents that are answering questions for other people across the company? And it's hard. It's hard to make that experience easy for people, to make it accessible, powerful, and accurate. And for us, it's been so easy for, you know, the olden days where it's like, hey, use the company wiki and no one uses the company wiki, right?

39:22It just doesn't work. And it's this new world where actually, if we had a super well-functioning company wiki, we could do a million things today that we otherwise couldn't. And so the responsibility for it dials up so much. the impact, that accountability. And for us, probably the biggest unlock for our company and many others that we've worked with is AI tools to minimize the friction to contribute to these wikis. And so it's like, oh, here's a conflict. AI can recommend how to consolidate that conflict and recommend changes so that they can accept the diff or they can adjust the wording in one little place.

40:02And so minimizing that friction in this new world, we found is absolutely critical.

40:06Jon Krohn:Great. So now that we've talked about data, let's talk about the models themselves. So this is chapter seven and eight in your book are focused primarily on models and an interesting analogy, a sobering story that comes up in those chapters is the fine tuning trap that Bloomberg ran into with their Bloomberg GPT. So they spent an eight figure sum training a model, Bloomberg GPT, which for a few weeks was a really big story because they had trained this model on all their financial data and it outperformed GPT-3 by a lot on financial queries for the most part. So it seemed like an example of this kind of this niche model that was going to provide a lot of value.

40:52Jon Krohn:And then a few weeks later, GPT-4 came out. And nobody's used Bloomberg GPT since. So eight figure sum for a few weeks of utility, not a great investment probably. So given how rapidly frontier models advance, what's the checklist that our listeners should run through before committing to a fine tuning project today, especially if it's going to end up being expensive? Yeah, I think I have to be careful what I say here because I have really strong opinions about fine tuning, but I'll just keep it really simple. More people do fine tuning than they should. I'll just leave it at that. You should really think twice about committing to fine tuning before you get into it.

41:39And so let's talk about that checklist. What should you check off? First off, you've got to make sure, am I an organization that's ready to take on this challenge? Do I have the necessary talent? Do I have the bandwidth to commit to something like this? Because yes, it seems simple on the surface. You're like, I just throw some data at it. Boom, it's an expert in the problem I care about now. But there's so much that comes into that with all the things we've talked about earlier with feature drift and keeping the model up to date and the challenges of new models continually getting better and better.

42:11I think it really comes down to do I have a really narrowly defined domain or problem that will not change over time? I think an example of this is anything to do with classification. This is a great place to apply fine tuning. You're like, I have an extremely straightforward classification job that I need to do at scale. I'm talking millions or billions of data points. If you have a problem like this, it may make sense to do fine tuning. Outside of that, I would really encourage people to hesitate before getting into fine tuning. I think in 2026, most people have figured it out by this point.

42:50Two years ago, I had this opinion and it was a very controversial opinion. Today, it feels less controversial, but really think twice before you jump into it. One of the points that I bring up with teams that are considering fine tuning is the opportunity cost. like look you can spend all this time building this fine-tuning model and you might be able to get it from 93 to 95 accuracy is that worth the amount of time and maintenance to support that over time versus all the other projects that you guys have been wanting to do and so yes if you had unlimited budget and unlimited tokens and unlimited resources like fine-tuning makes a lot of sense but if you're looking at like where's the delta in terms of the the improvement in performance yeah classification you definitely see that and there's other use cases where you see fine-tuning worth it but in a in a world that moves as fast as ours and the opportunity cost of all the projects every time we choose a project we didn't choose five others it better be a really really good project and for us fine-tuning will probably make more and more sense over time as we say the technology gets better and the infrastructure gets better.

44:01But generally, it's like more people are doing it than should.

44:04Jon Krohn:I think there's an instinct for those of us who historically, as data scientists, for example, have been really excited about training and deploying models. For us now, most of us, you know, 99 % of us in the AI space aren't working on frontier models, training huge LLMs from scratch. So instead we're kind of like, okay, well I can be using some kind of LoRa adapter and fine tuning this relatively cheaply, but your time is still very expensive relative to a lot of other options out there. And so just curating the data, the weeks spent probably at a minimum in most cases to get things right in terms of fine-tuning and get that 93 to 95 percent delta in performance, exactly as you say, Jacob, it's going to be the case most of the time that we should have just used something off the shelf, which is kind of disappointing for, I guess, like, you know, we're looking, we want to be doing something really cool and really meaty.

45:14Jon Krohn:But as you guys point out in your book. And we're actually getting pretty close to the question where this is going to become the main point that speed is more valuable than anything these days. And it's pretty hard to do that if you're developing custom models, unfortunately. All right. So cool. I kind of got the final word on that topic. Although we're getting to the speed thing again later. So you guys can chip in again. I've got one last question from the midpoint of the book before we get to that speed thing, which is, well, actually, you know, this relates to speed as well, but it's kind of, it's less about organizational speed.

45:48Jon Krohn:It's more about models. Uh, so you, in your book, you talk about the intelligence cost latency triangle. So if you're listening to this in an audio only format, I'm just going to say that one more time. Imagine a triangle. Don't imagine this too crisply if you're driving a car right now, but imagine a triangle with three points. Yeah, try that. And on each point is a different word. So we've got intelligence, say at the top, cost in the bottom left corner, and latency in the bottom right corner. I'm kind of assuming that people by default when they imagine triangles, it's like equilateral and there's a point up.

46:27Jon Krohn:I don't know why. You guys, did that happen for you guys too?

46:34Jon Krohn:And yeah, so this intelligence cost latency triangle, You talk in your book about how you can buy any of those two corners, but not all three. So you've got to pick. You can have high intelligence and low cost, but then your latency is going to be high. And you guys could work through all the other possibilities. But basically, you can't have a lot of intelligence, low cost, and low latency all at the same time. However, despite you saying that, you then go on to provide techniques that you describe for cheating the triangle. So things like dynamic routing, streaming, semantic caching, ensemble methods.

47:21Jon Krohn:Which of these techniques do you think is the most effective and maybe the most underused? Yeah, I think it's fun to talk about cheating the triangle because it kind of makes you feel really clever. You're like, ah, I cheated the triangle. I think probably the most important one is streaming, right? If you look in the world of software development, so much matters with latency and, you know, how quickly can you get something in front of a user? I can't remember the exact statistic, but it was something about like every 100 milliseconds you add, you lose like 10 % of people, something crazy like that, right?

47:59And so every second, every millisecond counts when you're building applications for users. And nobody wants to wait 20 seconds for a response to generate. So this is probably the most effective. But I think it's one that's also probably the most commonly used. Caching as well. Caching is so easy now, especially with the model providers often just doing it automatically for you, whether that's OpenAI or Anthropic. They'll just automatically handle caching on their end. Of course, you can layer in some more complex semantic caching if you want to get fancy. And that can be a very effective tool as well.

48:32Probably one of the most underutilized one is ensemble kind of methods. And I think I'm starting to see this get more and more popular, especially after Andrzej Kaparthi a few months ago kind of released his council of LLMs. So it's exciting to kind of see that kind of pick up more interest across the industry. I think people are more interested in that now, which makes me really happy to see. Probably the hardest one to get right is routing. On the surface, it seems so obvious, like hard problems should go to a smart model and easy problems should go to a dumb model that's really cheap. But then it's like, how do you classify a problem as easy or hard?

49:15And you see people complain about this where Chachi, BT and Anthropic added their adaptive thinking, right? And this is a huge complaining point for a lot of people where they're like, I don't like this at all. They're like, you know, having it instantly respond bothered some people. having it think was bothered some people, but having it try to guess which to do bothers everyone. And so this is like a really tough one to figure out. And so on that one, I'm like, I'm going to sit back and let the labs kind of figure that one out for a bit while the rest of us watch. Now, that being said, you may have in your case, in your company and your problem set, a more deterministic way of determining how to route problems.

49:56And in that case, I'm like, go for it. If it's really straightforward, use that routing system, it's so effective to just figure out which type of model should solve which problem. But if you're trying to figure it out on the fly, that's where you should not use that method. There's this aside that we put in the book around designing the user experience. Of course, our book is not about optimizing designing user experiences. It's primarily on architecting. but that being said it is amazing how some designers can get all the information they need for the ai go kick off the ai shepherd you through something else really quickly and then it appears instantaneously and they don't realize that the ai responded to them while something else was going on and so rerouting these user experiences and and pre basically you're running all of your AI jobs behind the scenes and then designing the user experience flow in such a way that they don't notice that we hid the latency.

51:02And there are like products UX people who are genius at this right now, where you see it throughout these products. So in the products that we use, we notice when someone does this and it's like, oh, it appeared instantaneously. And we realized that it had kicked off, you know, steps before that. And you know, this is like so fun because it's almost like we're back in the early 2000s where you hear the story of Instagram, right? Where they talked about how when a user was uploading a picture that they wanted to share on Instagram, the moment they like added it, they start uploading it in the background to the server while the user types out their caption.

51:37And when they hit post, it's just instantly there. And then it's like technology has gotten faster, the internet's gotten faster. I feel like, you know, us as engineers, maybe we've kind of just gotten accustomed to everything just works fast. So I don't need to do all these little hacks. But now like we've rewound the clock, we've got to deal with these slow, slow LLMs. And it's, it's time to relearn those techniques and lean back into them.

51:59Jon Krohn:For sure. Really well said guys. Great example around latency with respect to our models. Let's now, this is going to be my final technical question for you. And it's around latency of humans in developing AI solutions. So in chapter 11, which is the penultimate chapter of your book, you take on a question every AI leader is wrestling with right now, which is if everyone has access to the same frontier models, where does durable competitive advantage actually come from? And your answer is that it's not the model. It's not even the data exactly necessarily. It's velocity. How fast your organization can run the loop of build, measure, learn, improve, And to capture this concept, you invoke the gingerbread man.

52:48Jon Krohn:So run, run as fast as you can to create a moat. So velocity itself becomes your moat. And I think this ties together a lot of ideas from over this episode. We have been talking about this to some extent, but I feel like it now deserves its own focus. I think a lot of our listeners are wondering where this mode can come from. And yeah, if it's not models, if it's not necessarily data, then this idea of velocity itself, I think it's a great solution. How fast are you guys moving right now? We're trying to move as fast as we can. If you look at how the labs are operating right now, Anthropic and OpenAI and others, they're all in on this concept, right?

53:32The idea of self-recursive improvement and just like trying to close cycles as fast as possible. This kind of like goes back to the critical idea of the singularity. But the good news is like the rest of us companies can take advantage of this concept as well. This is not just limited to people building AI models. This also works for people utilizing them and trying to accelerate their company as quickly as possible. And so everything you can do as a company to complete cycles faster helps you leap ahead of the competition, whether that's shortening the time to build products, accelerating the process of, you know, going back to what Jacob said about define, build, and feedback.

54:16Anywhere you can accelerate each of those three steps is helpful. I think at this point, we've accelerated the build step a lot. There's still more room to be made there. But I think accelerating the define step And the feedback step are going to be enormous for companies to continue to increase the speed of this cycle. And if you're just making that 0.1 % improvement every time you finish a cycle, then over time that compounds. So on one side, it's the velocity of how fast you're going through these. And on the other end of the spectrum is how much do you improve with each cycle? And I think this relates to the concept of taste that people hear about or value and are understanding.

54:55And so you take a certain subject matter expertise, a taste and understanding of what's truly valuable for your end users and say, hey, if we run, if this is what we build, what would we learn and how much would the product improve and being able to pick and choose those? Because if building is not the bottleneck, then velocity and taste, which is the quality improvement on each one of those cycles, is the determiner.

55:22Jon Krohn:Yeah, for now, at least we have a little bit of an edge on taste. We'll see how long that lasts. There was a really popular article, Something Big is Happening. I don't know if you guys have seen that. Yeah, that blog post where he talks about how I think that article came out around the Opus 4.6 release and his use of cloud code where he said, you know, I felt like I had this edge on taste and I had to kind of get the algorithm to go in the right direction there. But now, at least for some use cases, maybe relatively simple, but probably more and more, we're going to find that the post-training reinforcement learning that these models have are allowing them to have our taste.

56:07Jon Krohn:Anyway, I don't mean to be saying you're wrong, but it is interesting. There is also, there's an edge that we can have, at least for the foreseeable future, around the right context for that taste. So, you know, I think that's something key here where even if we're trying to have Claude have all the right context, you know, we're taking our meeting recording transcripts and putting those in, there's still going to be some things that are probably missing that we as a product owner or a system architect, we have some extra context that's critical that the algorithm doesn't. Cool. All right. So that is my journey through your book, Architected Intelligence, available from wherever you buy your books now, published by Wiley, a great publisher.

56:53Jon Krohn:Congrats, both of you. For both of you, it's your first book, right? Outside of textbooks, yeah. Right, right, right. And I understand you're going to start working on a new book soon as well, right? I have started working on it and my publisher is not happy with my pace. But yes, I think I announced this. Did you guys hear this on the podcast or? Yeah. On the most recent episode, we heard that you were working on a book with someone named Edward Donner, right? Ed Donner. Exactly. Many of our listeners will be familiar with him. He actually had an episode on this podcast some 18 months ago. we are going to have another podcast episode.

57:34Jon Krohn:So we're going to, I don't know if you guys have this with your book release, but with, so Pearson is my publisher and Pearson publishes into the O 'Reilly platform. And something cool that you can do with that is we're going to be releasing rough cuts of each chapter as they're available. So I'm hoping that not too long after this episode comes out, I'm going to have the first chapter done and available in the O 'Reilly platform for people who have a subscription to that and then yeah we're going to have probably about a dozen chapters like you guys had so it's going to take us a while to get the whole book out but yeah Engineering AI Agents is the book and yeah looking forward to making that I think Ed is tremendous he He has amassed, he started creating video courses on Udemy about 18 months ago.

58:33Jon Krohn:And he sold 500 ,000 copies of courses that he's made. And yeah, people can figure out how much that's worth because they sell for at least 10 bucks each. You know, it's like, that's a non-trivial amount of revenue in an 18 month period. And he's done that by being so engaged with his audience. and yeah, I'm basically just, I'm kind of taking his ideas, his video content and working with him to repurpose it into a coherent textbook. And given how valuable his courses have been, we're hoping this book will be a hit too. Nice. We look forward to it. Yes, yes. All right. Well, thanks for listening to the podcast, guys.

59:17Jon Krohn:That's nice. We love it. Before I let you go, since you are podcast listeners, you must know that I always ask for a book recommendation. Do you guys have one for me today? And you can have one or even more each. I'm not, I'm not a very, I'm not a very strict interviewer. Deal. Um, I think for me, uh, I'm going to cheat and I'm going to recommend two books, uh, observability engineering by charity majors. Uh, fantastic book, uh, was a huge, huge inspiration for us as we were writing our book and kind of thinking about how can we write a book that, uh, is focused on higher level concepts. So that was super inspiring.

59:57And then LLM Engineers Handbook, I believe it's by Paul Hudson and Maxine Le Bon. Very more practical kind of code-based guide for those trying to get into AI engineering. So highly recommend that. Yeah, besides Charity Bajer's book, another book that was an inspiration for us as we were writing, centering in on how can we communicate something that's changing very quickly, but communicating principles that are durable and are still relevant five years from now. Fundamentals of Data Engineering from Joe Reese. He's someone that we know. I remember going through his book for the first time a few years ago, and that was an inspiration for me because there were so many concepts that were enduring around the principles of data engineering.

1:00:47And as we were tackling this book, we thought how could we approach AI engineering and architected intelligence in a similar way.

1:00:56Jon Krohn:I love that. And you guys have definitely accomplished it. I love how this book is going to last for years and years. So even if you're in the 2030s or beyond listening to this podcast episode or given our efforts to be a user agnostic podcast, maybe you're an agent listening in in the future. Exactly. If architected intelligence is not already in your canon, then put that in your canon, agent. Exactly. Nice. Fantastic. Thank you so much. This has been a great episode. You know, every minute had an invaluable nugget for us to take home with us after the episode. Jacob, Jeremy, for folks who want to glean more invaluable nuggets from you, other than obviously purchasing your book, Architected Intelligence.

1:01:48Jon Krohn:How can people follow you online after this episode? We've got a website called the whybehindai.com. So you can go check that out. And similar to how you're going to be publishing your content through O 'Reilly, that's where we publish thoughts from the book in kind of a rougher format or other thoughts that we have around kind of the why behind AI. It's a very kind of fun place for us to share our ideas. We tend to be a little more casual in that environment where there's more memes. So not as many memes in the book, sadly, there's, there's still some fun nuggets in there, but if you come to our blog, you're going to have a lot more fun as far as memes go.

1:02:27So we make it a lot more entertaining and we're both on LinkedIn and that's where we, you know, post updates in terms of things that we're doing and tackling. But yeah, Wiley of course, doesn't like as many memes as we would have liked in the book. And so those, those are all, those are on the blog. It is a more fun way of, of, uh, ingesting the material.

1:02:48Jon Krohn:You do have a lot of stick figure cartoons though, that it's still add that, that lighthearted element to it. So that's really cool. All right, Jeremy, Jacob, thank you so much for taking the time to be on the show. And also this episode would not be complete without me thanking John Bowne. How do you pronounce his last name? Jonathan Bowne? Yeah. Bowne. Yeah. Jonathan Bowne. Okay. Nice, nice, nice. Uh, yeah. Thanks so much, John, for recommending Jeremy and Jacob as guests. John has been a listener to this show for years and years and one of our most active social media commenters. And yeah, maybe at some point we're going to have to have John on here for an episode as well.

1:03:26You should. He's a great guy.

1:03:27Jon Krohn:Yeah. And I know that at the time I was recording this, he is in Boston at ODSC East giving a big two-hour long tutorial. So there's probably some content in there he could be sharing with our audience. so yeah so thanks John yeah all of these J's from John to Jacob and Jeremy thank you other John for this jovial Jepisode yeah this was really good thanks listeners for bearing with me and all of my J's and yeah thanks a lot Jacob and Jeremy for being on the show thank you man oh man And that was a great episode. Really appreciate the well-thought-through insights of Jacob Miller and Jeremy Mumford.

1:04:14Jon Krohn:Don't forget to check out their new book, Architected Intelligence. In this episode, we basically only talked about stuff from their book, including the user agnosticism tenet, which means designing products and processes so they can be executed equally well by a human, an AI agent, or any hybrid combo. We talked about the shift in the define build feedback loop now, where building is no longer the bottleneck like it always was, which means definition and feedback are where teams win or lose. We talked about why workflows are deterministic, predictable, and cheaper than agents, and why the natural progression is skills first, then workflows, and only then agents.

1:04:55Jon Krohn:We talked about why data engineering is the true bedrock of AI engineering, illustrated through the Anchorman analogy. Whatever you put in front of your model, the model will read as truth. And we talked about why Velocity is the only durable moat in a world where everyone has access to the same frontier models. All right, that's it. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Jacob and Jeremy's social media profiles, as well as my own, at superdatascience.com slash 993. Thanks to everyone on the Super Data Science podcast team, our podcast manager, Sonja Brejevich, media editor, Mario Pombo, partnerships manager, Natalie Jaisky, researcher, Serge Massis, and our founder, Kirill Aramanko.

1:05:45Jon Krohn:Thanks to all of those fine folks for making another sensational episode for us today for enabling that super team to create this free podcast for you. We are deeply grateful to our sponsors. You can support the show by checking out our sponsor's links, which are in the show notes. And if you'd ever like to sponsor an episode yourself, you can get the details on how by making your way to johnkrone.com slash podcast. Otherwise, there's tons of other ways you can support the show and we really do appreciate if you do. It is the only way we can grow. So please do share this with anyone who would like to learn about how they can be building AI-first companies themselves.

1:06:29Jon Krohn:Review this episode on your favorite podcasting platform or on YouTube. Subscribe if you're not already a subscriber, but the most important thing above all else is that you just keep on tuning in. I'm so grateful to have you listening and I hope I can continue to make episodes you love for years and years to come. Until next time, keep on rocking it out there and I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.