1. Podcasts
  2. Practical AI
  3. Abundant Intelligence at the Midwest AI Summit
Practical AI

29 Oct 2026 · 54m · 12 chapters

Episode artwork

Abundant Intelligence at the Midwest AI Summit

In short

Practical guidance from the Midwest AI Summit on building “durable” AI-augmented workflows (mechanism selection, data grounding, guardrails, evals, observability), plus how to drive adoption of AI agents with frontline workers, and how to structure team workflows for AI-assisted software development (avoiding “dark factory” misfit).

Guests (backgrounds)

  • Heidi Porter: Principal at Moyer Group AI; decades in networking, data analytics, ML/AI, and on-prem/cloud deployments; teaches AI education/strategy/consulting.
  • Chris Schenkel and Andrew Dunlap (SEP): SEP builds client projects across aerospace, medical/healthcare, finance; Chris is Director of Innovation; Andrew is a senior AI engineer.
  • Kevin Solario: Staff software engineer at Focused; builds custom agents using the LangChain stack; focuses on production adoption.

Key claims

  • AI pilots fail when LLMs are asked to do too much; choose the right mechanism per task and enforce guardrails as real code/human gates, not prompt text.
  • Ground LLMs in systems of record at runtime; evaluate beyond averages; add evals and observability (replay/drift).
  • Agents without evals are “proof of concept.”
  • Team “dark factory” automation depends on constraints; AI amplifies existing team issues.

Notable examples

  • Manufacturing “rubber hose” defect detection: vision model triggers diagnosis; LLM drafts actions using knowledge graph evidence; rule checks verify evidence support; human escalation when thin; monitored 52 weeks.
  • Agriculture: Syngenta CropWise AI for seed selection; AWS identity authorization; Bedrock agent routes to knowledge bases/APIs and uses existing agronomy models; LLM explains in plain language; reps decide; 100 QA evals with human scoring.
  • Healthcare: Stanford Surgical Co-Management Navigator; applies institution-owned rubric from EHR notes; LLM outputs only yes/maybe/no; physicians override; benchmarked 5 LMs; overflag design led to 97% physician agreement on “no.”
  • Adoption: blue-collar field technicians; build with users, start with intentionally “wrong” prototypes, and use eval datasets + feedback flywheels.
  • Software teams: transmission-operator simulation prototype stalled due to token budget, domain knowledge, and verifiability scaling; other teams used shared knowledge to move toward higher autonomy.

Written by AI. May contain mistakes.

Chapters

0:36

Overview of the Midwest AI Summit

Discussion of the Midwest AI Summit and what listeners can expect.

“And I arranged with the conference organizers for us to be able to share a few snippets from speakers at the Midwest AI Summit.”
1:41

Introducing Heidi Porter

Introduction of Heidi Porter and her background in AI.

“We'll be sure to update our audience here when the details of that are shared from the conference organizers.”
1:54

Heidi Porter's Talk on AI Workflows

Heidi discusses how to develop durable AI-augmented workflows.

“She's a technology professional with decades of experience in networking, data analytics, machine learning, AI, and on-premise cloud deployments at large companies.”
6:04

Examples of AI Workflows

Real-world examples of AI workflows in manufacturing and healthcare.

“But Jeff talked about this a bit, being able to recover and replay and catch drift before an incident.”
10:00

AI in Agriculture and Healthcare

Discussion of AI applications in agriculture and healthcare, highlighting specific use cases.

“So this is a published peer-reviewed study.”
14:00

Evaluating AI Systems in Healthcare

Learn about the importance of evaluation in AI workflows, especially in healthcare settings.

“And, you know, if you look at the evaluation note here, They had 100 question and answer pairs from actual sales ref, scored it by human reviewers, used an LM as judge.”
20:23

Building AI Tools with Users

Explore the challenges and strategies for developing AI tools that users will trust and adopt.

“Most of this talk I am going to be speaking from is coming from one of my most recent clients at Focused.”
28:00

Building Trust and Feedback in AI Development

Learn the importance of involving blue-collar workers in AI development and the iterative feedback process.

“If you're working with white-collar workers, you might be able to get by doing most of this without truly interacting with them.”
36:48

Navigating AI Integration in Organizations

Explore the challenges enterprises face when integrating AI into their workflows.

“So this is a talk from two folks from SEP, Chris and Andrew.”
42:01

Identifying Constraints in AI Projects

Learn about the six key constraints teams face when implementing AI projects.

“What happened was there's new forces at play.”
Show all 12 chapters
44:51

Understanding Modes of Operation for AI Teams

Explore the different operational modes of AI teams and how constraints affect their functionality.

“So there's four modes that we commonly find our teams operating in.”
49:25

Five Principles for Effective AI Development

Discover five essential principles to guide AI development and improve team efficiency.

“So you might be asking like, what does this look like for me or my team?”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Practical AI, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:02Andrew Dunlap: Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show.

0:41Andrew Dunlap: I had the privilege of being the MC at this year's second annual Midwest AI Summit, and it was amazing to see everyone there at the summit just engage in a really useful, practical way around subjects like AI agents and decision models and AI security. And I arranged with the conference organizers for us to be able to share a few snippets from speakers at the Midwest AI Summit. So today's episode is going to be a very special one where you'll get to hear a selection of a few snippets from speakers at the Midwest AI Summit. We'll have Heidi Porter, who is a principal at Moyer Group AI. And then Chris Schenkel, Andrew Dunlap, who are with SEP.

1:27Andrew Dunlap: And then Kevin Solario, who is a staff software engineer at Focused. I hope you enjoy the content. I know I certainly did. And yeah, would love to see you at next year's Midwest AI Summit. We'll be sure to update our audience here when the details of that are shared from the conference organizers. So enjoy this selection of a few talks from the Midwest AI Summit. We have with us Heidi Porter. She's a technology professional with decades of experience in networking, data analytics, machine learning, AI, and on-premise cloud deployments at large companies. Her company, Moyer Group AI, provides education, strategy, and consulting on AI tools and where it belongs in your tasks and workflows.

2:16Andrew Dunlap: And it's those workflows that she's going to talk about how to make durable AI augmented workflows. Help me welcome Heidi Porter. Heidi Porter All right.

2:26Kevin Solorio: Thank you. Got the clicker here. Good morning. I'm Heidi Porter. Quick show of hands. Who has had an AI pilot that did not make it past the demo stage or stalled at the start of production? So there's a good amount. And congratulations to all the rest of you who just, you know, hit it on the first time, right? So a lot of times these workflows fail because it's not because the model was the problem. It was because the LM was asked to do too much. And Don and Jeff just gave great talks about when to say no to AI and when to say yes to AI. And so sometimes people ask it to hold the data, be the system of record, it's not a database, make the decision and check its own work.

3:14Kevin Solorio: The question I want you to come away with today is to look at the task in the workflow and decide what is the right mechanism for the task, okay? And then how to evaluate right answers and how to catch it when it's wrong. So evaluation, guardrails, et cetera. But before we begin to look at workflows, we need to say a word about data. So the data that goes into your workflow, I think everybody knows this a bit, but we have to talk about it. It sets the ceiling on how reliable your tasks and workflows can be. And the question isn't whether your data is good. It's whether, you know, good in general, it's whether it's good enough for the task you're doing.

3:54Kevin Solorio: So we check four things, you know, is it accurate, complete? There's a whole list here. Is it consistent? And as Jeff talked about, maybe your agent can help with some of this. But, you know, if the data is stale, it'll throw an error and just quietly give you the wrong answers. I don't know if an agent can help with that. So when there's an LLM in the loop, you need to ground it in your data at runtime. And we talk about systems of records. So you're grounding your LLM in your data. You're fetching the facts. You're using a query to query a database or some kind of knowledge graph. You're letting the evidence drive the answer.

4:30Kevin Solorio: And you have it cited sources so people can check if they need to. right? Grounding is only as good as the data behind it. So fix the data later first, because everything downstream inherits its quality, right? So now let's get into what a working system needs behind good, clean, integrated data. So four words we'll use for the rest of the talk. Mechanism, that's what we're going to call what actually performs this step, whether it's rules, classical ML, a decision model like Jev, and I'll talk about what that is, or an LLM. And an LLM has a large range of, you know, directly querying, you know, using an LLM to generate, using it to call tools, using it in an agentic architecture like Jev talked about, or using it with human review.

5:17Kevin Solorio: And you pick that step per task, not per project. And then there's three safeguards that surround it, right? And so you have guardrails that are runtime controls on each output. They can catch problems, escalate it to a person, constrain answers when something, you know, and then recover when something gets through anyways. And then evals are repeatable tests. Evals are really important and they're not talked about as much as they should be because they are determined whether the answer you're getting is right, whether you get a good outcome. And one of the things I want to call out is when you're looking at a workflow, don't just look at the averages because averages can hide a bad result.

6:00Kevin Solorio: And you need to know how bad your bad is so you can fix it. And then observability, we won't have too much time to get into today. But Jeff talked about this a bit, being able to recover and replay and catch drift before an incident. So build each step right and then safeguard every one. And what we're going to do today is show some examples of workflows across manufacturing, agriculture, and healthcare that actually worked and the lessons they learned. But first, let's look at building them. So for every task, there's a lot on this slide, I know. There are really three decisions. So the first one is what performs it.

6:41Kevin Solorio: And we just talked about that here. and we talked about the fact that the LLM mechanism has a wide range of how you can implement it. Second is how the work is coordinated. So if you know the steps, you probably just want to use a fixed workflow with programming, right? Or you can use an LLM or decision model to pick a branch at a decision point. Or you can have an agentic architecture where the agent works and is constrained and bounded by your cost or time, et cetera. And we'll show one example of that. And then there's third. This is where we start to get into guardrails. So what's allowed to execute?

7:21Kevin Solorio: And think of it as a dial, right? So if it's consequential on certain actions, a person approves it. If it's high volume with mixed confidence, let the system act above a confidence threshold and escalate the rest or vice versa. Time-sensitive work, act now and give a person a window to veto or undo it. and automate outright only what it's cheap to undo. So, you know, using like an agentic architecture. And whatever you pick first for these different questions, test it, treat it as a hypothesis. And on the output of each of these tasks, you need to think about the guardrails, right? That's what we said was very important.

8:00Kevin Solorio: To make sure that you get the outcomes you're looking for, you need guardrails around every mechanism. They raise the odds of a good result and you won't get a perfect one. So you need to measure quality against your own baselines, okay? And the first thing to really realize, and it's been talked about a bit, is that a rule written in a prompt is not deterministically enforced. A guardrail is actual machinery, a gate, a check, a human, a limit in code. And on this slide, you can see that the guardrails scale with the mechanism. And I want to call out a couple of these. One is, you know, you think, well, programming, we don't need guardrails.

8:41Kevin Solorio: Well, you should do traditional programming has always needed guardrails, you know, not to push a bug into the code, you know, to have some change control. And then another one is we think, well, you know, if we have a human review the LLMs, and Jeff talked about this, how there'll be thousands of things to review, the human can get tired, right? They can just have rubber stamping fatigue and, you know, that's not a good guardrail. So you have to be really careful about these kind of things. And we've listed some, you know, possible mitigations, et cetera. And then the column I really want to call out is recover because something will get through eventually.

9:21Kevin Solorio: And recover is how you contain it and get back to a safe state. Roll back, retract the output, undo the right, et cetera. And so you should make actions reversible where you can and where you can't require approval before they run and be careful of that human fatigue. And then the VELs sit over all of this. Like I said, this often isn't talked about enough. They tell you whether it's working and they gate everything towards more autonomy. So if the VELs aren't working, you don't want to promote to production because you will soon not have a working system. So I'm going to show you some examples in real deployments that are kind of fun.

10:02Kevin Solorio: So this is a published peer-reviewed study. So it's not a company, you know, it's not from a company, it's not marketed. It is a peer-reviewed study of a rubber hose production line. And so a camera watches the line and a modified, you only look once vision model finds the defects. And if you take a look at what they're looking at, you can appreciate the difficulty of this task, especially at speed. So it detects the defects and there's no LLM in this. And then a detected defect triggers the diagnosis. The system then retrieves evidence from a knowledge graft and it searches relevant SOPs and maintenance history.

10:39Kevin Solorio: The LLM then just gets the evidence from the systems of record, right? Not the whole knowledge base and it drafts a candidate explanation and corrective actions. But here's the thing. Now we have this additional guardrails, right, where they use rule-based checks, traditional programming, to check whether the draft stayed within the evidence that it pulled. So it's checking the LLM. And if it looks thin, if the evidence doesn't have enough support, it sends it to a human. So there's two different guardrails on this. The other thing is about this example is that operators can accept, override, and comment on any of these, not just the ones that are escalated.

11:20Kevin Solorio: So they have that control. So what did we learn? You know, notice what they didn't do. They didn't ask the LM to look at the hose and figure everything out. The LM explains, and there's additional checks and guardrails. And the results on this are good, but they're not perfect. So, you know, grounding still needs checking. And what they did as far as observability was they monitored this for 52 weeks to make sure it was working right. So let's look at an ag example. And I like this one a lot. So I think this, actually, I'm going to add this little note that I think this use case could be useful across a wide variety.

11:57Kevin Solorio: So think about it for some of your situations. So this is a seed company called Syngenta that has a big seed catalog. And sometimes farmers select the wrong seed for their field. So the salespeople or the grower bring this solution, CropWise AI, on site, and they do on-site seed selection based on local conditions. So the person asks the question in plain language, and then LLM interprets it. Good use of LLM. Before anything else, the user is authorized by the identity service in AWS, an application code, not in the LLM. Then the Bedrock agent routes the request to knowledge bases, APIs, Python actions, data agents, and it pools weather, soil, product, and ergonomic data.

12:42Kevin Solorio: And when the question needs a prediction, Sedensa's existing seed and agronomy models produce it. The LLM does not do agronomy, okay? Its second job is to explain the result in plain language, and the rep or grower makes the decision. So this agent, you know, is pulling from systems of record, getting the agents it needs, and it's sending out sub-Agent. We'll see a diagram in a second. So if you look at this, you can see blue is where the LLM or agent works, gray is rules and or ML, and yellow are the guardrails. So in this case, the LM is the interface, the router, and the explainer, not the agronomist.

13:22Kevin Solorio: And this is not as nice a picture as Jeff had of the knowledge system, but you can see that, you know, there's several knowledge bases, lots of documents is pulled in. And the reason I wanted to show this kind of diagram here is because of one of the big lessons learned in this. We talked about the first two already here, but the third is that the way they did not design that system, it doesn't just survive a model change, you know, swapping out a model. You can also swap out the knowledge bases or some of the other data sources. And so that makes that really, you know, you have to reevaluate afterwards.

14:02Kevin Solorio: And, you know, if you look at the evaluation note here, They had 100 question and answer pairs from actual sales ref, scored it by human reviewers, used an LM as judge. So they really, you know, did all these evals to make sure the workflow was successful. But it is a useful thing to know that you can swap out these different pieces pretty easily. So that's our takeaway here is design so the model and knowledge bases are the easiest part of the system to replace. Now, the next one I also really love is from healthcare. It's Stanford Healthcare built a tool called Surgical Co-Management Navigator, and it screens next-day elective surgeries to see which patients need surgical co-management.

14:45Kevin Solorio: So surgical co-management, I had to look it up, is when clinicians share the practice of patient care responsibilities during pre-op, inter-op, and post-op. And before launch, routine human triage missed half of eligible patients. And so, you know, if I'm having surgery and I'm eligible for co-management, I think everyone in here would want that oversight there. So this system is really a big deal for them, for the patients especially. So the new system pulls the pre-op anesthesia notes and the medication list from the electronic health record. And the criteria it uses are a rubric that the institution owns, written from their own literature, their own faculty expertise, and fixed in the prompt, okay?

15:32Kevin Solorio: And we talked about how the prompt is, you know, readable, not enforcement. But in this case, the ELM's only job is to apply the rubric and say yes, maybe, or no. So it can't hallucinate anything. Yes, maybe, or no, that's the answers. Again, just like rubber hose, every case shows up in the EHR list, flagged or not, and the physician can, you know, look at all of them and confirm decline or override. The clinician keeps the decision here. So before launch, the other thing they did was they benchmarked five different LMs against consensus chart review and picked O3 Mini because it performed competitively at a lower cost.

16:10Kevin Solorio: The model is again the replaceable part, not the center of the system. And what's really interesting about this case I like is in the analysis of false negative cases, this is a small sample, but there were a minimal amount of LLM errors. What happened is they found a lot of other errors in their process, and then they were able to work to fix some of those other errors. And of course, look at the LM errors as well. So three lessons from Stanford, and it's all about who owns the protocol. So the institution owns the criteria. They compared candidate models. And then third, they checked, this is the one I want to call out, is they checked why it disagreed, right?

16:51Kevin Solorio: And so what they typically did was overflag, you know, be more false positives. And when they looked at the no's, the physicians agreed 97 % of the time. So it overflags by design and physicians declined about four in 10 flagged patients, basically saying it wasn't complex enough. So these are three running workflows and their overview of using the LM for the right task or the orchestration piece. And so, you know, back to these four words we were talking about, the four parts we started with, we can map these to something you just saw in these examples, right? So does each step use the simplest thing that works?

17:33Kevin Solorio: Syngenta used the agronomy models, guardrails, the hose line escalated when evidence was weak, the system withholds the answer and escalates. And then third, the evals, You know, we just talked about Stanford benchmarked five models and just tested against 100 real questions. And fourth, observability. We talked about the rubber hose line. They observed it for 52 weeks. So if you can answer yes to all four of these questions, you have a system that most likely keeps working after the demo. Thank you very much.

18:08Andrew Dunlap: Thank you so much, Heidi. That was great. Great. I love how you dug into each of those workflows. Super practical. Whether you're an early stage startup or a Fortune 500, it's critical that you have an amazing website that looks handcrafted, but where you ship that website, updates to that website, landing pages, et cetera, in minutes, not days, to keep up with SEO and AEO and to make sure that your content is fresh. Framer is the AI website builder that helps creators, teams, and businesses ship production-ready sites faster than ever while getting every detail right. Prompt, inspect, edit, and publish in one place at a whole new pace.

19:00Andrew Dunlap: Learn how you can get more out of your site from a Framer specialist or get started building for free today at framer.com slash practical AI for 30 % off a Framer Pro annual plan. That's framer.com slash practical AI. Rules and restrictions apply. I'm really excited for our next speaker here, Kevin Solorio. He's a staff software engineer at Focus. They're one of our sponsors out here. I got to meet Kevin, yeah, probably a couple years ago. He was here last year at the Midwest AI Summit. Their team at Focus is an amazing just set of AI engineers that they build agents all day, every day. They engage with a variety of types of companies and actually get agents into production, evaluate those agents as we learned about this morning.

20:00Andrew Dunlap: Kevin is a self-taught engineer and AI consultant, again, building custom agents, mainly using the LangChain stack. Today, he's going to be sharing with us how to build AI with workers, not for them, which is part of that earning trust on the front line. So please join me in welcoming Kevin.

20:22Chris Shinkle: My name is Kevin Solorio. Most of this talk I am going to be speaking from is coming from one of my most recent clients at Focused. We started a new engagement and the CEO immediately told us adoption is key. No matter what we build, adoption has to be the number one thing that happens. And we're building an app for some blue-collar workers, which he up front said are notoriously slow to adopt new technology. They don't trust it, they don't like it, they don't see it, they don't want it. So in order to accomplish that, we wanted to take a few minutes and say, okay, we need to get this group of users to use a tool that they may or may not want to use.

21:05So let's dive in to discover why some of these tools

21:10Chris Shinkle: may have failed in the past. And so surprisingly, some of the first things that come up is it doesn't work. So when you talk to someone and they start using a tool and it doesn't meet their needs, they don't like it. They don't use it. But it doesn't work is very vague. So diving into that with them, you start uncovering, well, it's slow. It crashes. It might not meet their needs. It might do the thing that it's supposed to do, but that doesn't mean it's doing the thing that they need it to. So sometimes there's just a lack of connectivity there. The other thing that might happen is that the tool's not really helping them.

21:49Chris Shinkle: Sometimes technology is adopted from several layers up in the management stream, and it's there to solve a different problem or to track a thing that they have going on, and it has nothing to do with their day-to-day job, and if anything, it just adds more work for them to actually go into the system and update it with all the information they have to. The other thing is cumbersome. So if the tool is cumbersome to use, they're not going to. They don't want to. If they have to go into three or four different systems and extract information and then come back to this system and put it in there and then do that whole process, it's just not going to be something that catches on.

22:27Chris Shinkle: And so that's one of the other things that happens with tools, specifically when they are built for someone. So all of these things or symptoms when you have tools that are being built on behalf of someone else. So it didn't work means typically we didn't research the requirements enough. We didn't dive into the use cases. We didn't understand exactly what we needed to build in the first place. It was not built for them. If it's for someone else's, what is it, KPIs, OKRs, any of those other metrics, like if it's not really there for them, it's just in their way, they'll use it because they have to, but they might do it like at the end of the day.

23:08Chris Shinkle: Oh, I'll just go log everything at the end of the day, which means they've forgotten half of what's important to log. So it's not getting into your system anyway. And then the last one is the cumbersome. It's likely because it was designed by someone that wasn't doing the job. They didn't understand how much of a pain it would be to go to four different systems and then go into yet another one. So if we stop building for someone, and then we actually start, we'll start going back to that a little bit more. The big thing with building for is it's always through a proxy. Most of the time, when you have a user that you're working with, like a product owner, they're interpreting the results, they're having conversations with users, but they're adding their interpretation of it, which may oversimplify things, it may take things out, it may miss very important use cases that they just gloss over.

24:01Chris Shinkle: it's also again going back to when you build for someone you're kind of sending a signal of I know what's best for you this is how this work should be done which if you're anything like my partner she does not like it if I say hey do this thing but if I ask her nicely no problem so it's one of those situations where you come in and you say do this thing and this is how you're going to do it you're kind of coming back at saying I know how to do your job better than you So it just sends a weird signal with that. And then also this tends to codify the standard operating procedures that might just be like those in theory types of cases.

24:36Chris Shinkle: So in theory, all of our requests fall into one of these nice little buckets and you never go outside of that, which is rarely what happens. Contrasting that with building with someone. So when you start building with someone, you're going to start with a goal. Instead of saying exactly what you're going to do, you're going to start by addressing what problem it is that you're really trying to solve. And by having them in the room and starting to share information across both the silos, you start to build respect and trust with each other. So I know that they know their job well. In this particular case, we were dealing with field technicians that were repairing equipment from meat grinders to ATMs.

25:15Chris Shinkle: Like literally the same person being dispatched to fix your meat grinder might be the same person dispatched to fix your ATM. So they have a wide breadth of knowledge. That is stuff I do not have. And when I build a system, especially using an AI agent that's going to generate something, it will generate for sure. It will give me an answer. Is it right or not? I have no idea. So I really need to lean on them to help correct me where I'm wrong. And this is where those in-practice things come in. now historically when building with people in the day of web apps and pre-ai so like three years ago this would largely take the form of looking at paper prototypes you'd get in the room with them you might do some drawings on a whiteboard or some big markers and just start drawing things out what we're really trying to define here is what information needs to be surfaced at what time at what fidelity are we getting the right information to them are we giving them escape patches?

26:13Chris Shinkle: Is there a way for them to drill down into details that might not be there right now? Now, when you start going into agents, it's less about layout, because a lot of us, if you've seen anything, it's a lot of chat boxes. So there's not a lot of layout to discuss. You're going to work with an agent. So how do you make sure that it's doing the right thing? And with the agent, your goal is the right behaviors. It's calling the right tools at the right time. It's using the right guardrails. It's respecting the boundaries there. So it's not about the design as much as far as like the layouts and things like that.

26:49Chris Shinkle: It's really diving into is it doing the thing I want it to do. And evals really provide the perfect seam for that. So this might be a bit of a bold statement, but an agent without evals is a proof of concept. and this doesn't matter what infrastructure it's been deployed onto. If you do not have evaluations, you cannot confirm for sure that the agent is behaving as expected. And so in this case, I'll always say that's more of a proof of concept at this phase. And the nice thing is with evals, one of the nice, you can actually start testing your agent again and again and again. When I typically kick off a project like this, I will maybe synthesize a data set where I'll take what I know about it and then create a data set to just get something wired up and in their hands.

27:47Chris Shinkle: I want someone to be able to look at it, touch it, tell me if it's working right or not, and give me that feedback. The interesting thing working with blue-collar workers, they don't have traces as much. If you're working with white-collar workers, you might be able to get by doing most of this without truly interacting with them. though I do highly recommend having them involved. But with blue-collar workers, there's not that trail. There's not the list of emails. There's not the list of traces. You don't have everything that you would normally have if you were working with a white-collar worker.

28:22Chris Shinkle: And so to really start building for people, the first thing is to get everybody in the room. Start talking about it. And going back to what I was saying earlier, that synthesized data set, something for them to look at, I want to be wrong here. One of the things I found as a consultant is getting information from people can sometimes be very hard. But if you show them something that's wrong, they are very quick to tell you where it is incorrect. And so if you start with something knowing that, hey, this isn't going to be right and nothing's right on the first pass. So you go ahead and start with an idea.

28:53Chris Shinkle: This is what I have proposed that we work on here. Let it be wrong. It's OK. Get that feedback. Get that flowing in. one of my early mentors used to tell me marry the problem not the solution and that was because the problem is what's important if you get too hung up on a solution especially if it's one that you created when someone comes up with a new idea that might be better or more efficient you might feel a little hurt that your solution wasn't the one that was selected so in this process it's really important to realize that you're there to solve a problem and not just promote what it is that you came up with.

29:27Chris Shinkle: So as you present that idea, be wrong, take the feedback and begin to iterate on it. And that's where that comes together. And I feel like this is where you really start to build that trust is when they can see how wrong it was at first. And then through some level of interaction with them, they start giving you that feedback, you can then make it so that your agent is now performing the task that you're expecting. Or it's like starting to get there. The responses are making a lot more sense. With this, you are going to start to want to get that feedback from them regularly. So in this case, you might want to make it as easy as possible for them to give you that feedback.

30:05Chris Shinkle: So if you have a data set that you want them to label, give them a Google spreadsheet. They can go and label it on their own time, get it ready. Or if you have the time, vibe code up an app that's fun, make it gamify it, do something, get them a way to get that feedback for you. And then you can use that in your evaluations. And then this is the part where you're going to iterate a lot with them, but then you might broaden the circle. So initially, you might start with a handful of representatives that are going to showcase their knowledge and help you get on the right track, but they might not see every case.

30:39Chris Shinkle: So then you're going to widen that circle a little bit. So this is where you're going to want to start creating those data fly flywheels. If you've not used a data flywheel before, what that is, is you create an offline data set. And that's what you're going to use for your development practices. This is what you're going to use to develop your agent against. You're going to measure against that data set. And then when you deploy it, you're going to have online evaluators that are consistently going to look at a sample of traces and kind of score them. And that could be a code-based evaluators.

31:08Chris Shinkle: It's very deterministic. Or it could be an LLMS judge that can then help you find and flag certain cases that were not covered by your agent before. So those that get added to an annotation queue. And in that queue, you can review and discover what went wrong on that particular case. You can then add that back to your offline data set. And now you have a constantly flowing data set that is evolving with the use cases. So you get that flywheel set up and then you invite more people to the platform and you start seeing how they're using it. And you rely on that data flywheel to capture those additional cases and keep building.

31:45Chris Shinkle: And then when you're ready, you're confident in the product, you can go ahead and release it.

31:53Chris Shinkle: The big thing here, keep the loop running. You're always going to iterate. You're going to find cases that you didn't anticipate through the initial discovery. You're going to find cases that you didn't anticipate through the initial release, through the first couple rounds of beta users, through the widening of that. And so here is where it's really important to get that going. And then the people that participated in this, if you've ever heard of that IKEA effect, that is where you really add a lot of value to something that you help build, specifically if it went well. You know, that IKEA furniture that's your masterpiece in your living room, you actually prescribe a lot of value to that thing because you got your hands dirty, you built it, you were part of it.

32:36Chris Shinkle: So that's one of the things that can happen with this. And so then the people that you had helped build it become your champions. They're going to really be happy with what happened. They're going to be like, oh, yeah, see this tool? I helped build this. I contributed to it. I made this better. Now, one of the things with blue-collar workers specifically is that they're craftspeople. They want to see that you respected them and that you also have a craft. So with this, dive in. These evaluation data sets are typically very curated. You're not going to have millions of them. You're going to have enough that represents the use cases and make sure that you can cover all the things that you expect the agent to do.

33:19Chris Shinkle: You dove into the details. There's a lot of nuance, specifically in blue-collar work. It might be a thousand different things that come up, and you really need to dive in and understand what made this case different from another one. and taking the time to dive into their process and understand their flow and then be able to come back and show them that the system understands it because you took that time to say like, hey, let me hear you out. Share with me what you've learned. And also show it improving. Touching base on a frequent basis where you're able to show like, hey, I heard you. I made those changes.

33:51Chris Shinkle: Here's the feedback. Let's keep this going. That all ends up building a really nice, robust trust between the engineers and that team. And I've been fortunate enough, like right now we have a former designer, I would say former designer, he still does designing, but he's now one of our lead AI strategists and he comes in and he really listens and helps just like helps look for these nuances and helps lead that process. And it's been really great seeing how when you start working with people and really meeting where they are, that you can start building a system that everybody's really proud of and actually starts using.

34:27Chris Shinkle: that is all I have and I'm the last thing before you and lunch so I will not keep you here for

34:33Andrew Dunlap: much longer that was great thank you so much Kevin I love the how focused this was on on building four people AI agents are everywhere now and it's so easy to get access to one however if you're an enterprise and you're considering having this digital workforce of agents within your organization, there's a whole new level of things to consider, both around human identities and agent identities. Like, let's say Bob, an employee is onboarded into my AI platform. How do I scope his access to various models, MCP servers, tools within those MCP servers? And then if he creates an agent, Can he delegate access to that same scope of things?

35:25Andrew Dunlap: And what if he shares that agent with Alice, who has a greater level of scoping? Can she expand the scope of delegation to this agent? And what happens if Bob leaves the organization? Does that agent identity survive? These are a lot of the challenging things that are addressed if you have a powerful, sovereign, self-hosted control layer like Prediction Guard. I'm so proud of what my team that I'm leading at Prediction Guard has built and has deployed in production at various organizations that helps you manage your fleet of agents with complete confidence, knowing what they have access to, what they have accessed, how their runs have developed, how they've interacted with human identities, and how you have intervened through control mechanisms in their behavior.

36:20Andrew Dunlap: behavior and their various runs. I encourage you to check out what we're doing at predictionguard.com slash practical AI. Go ahead and schedule a demo with myself and the team to figure out how you can scale your digital workforce of agents without losing control. That's predictionguard.com slash practical AI. predictionguard.com slash practical AI. So this is a talk from two folks from SEP, Chris and Andrew. If you're not familiar with SEP, they've been building projects with clients across aerospace, medical, healthcare, finance, and other industries. So welcome, Chris and Andrew from SEP. I click here right here.

37:09Okay.

37:10Andrew Dunlap: All right, welcome. I'm sure a lot of you by now have seen the demos, the videos of people operating what they call a dark factory, right? Hands off the keyboard, agents writing all the code, agents reviewing the code, agents reviewing the agents, agents testing the code, agents shipping the code. And some of that's pretty impressive and makes sense in some situations. but I'm guessing somewhere somebody in your organization has seen that demo and has asked you like, hey, when are we going to start working like that? Like, hey, when do we turn off the lights and kick up the dark factory, right?

37:54Andrew Dunlap: And so that's what we're going to talk about today. And I know the picture looks like the lights are on the factory. They're not. It's the glow of the crazy robots in there.

38:02Heidi Porter: There's no humans.

38:04Andrew Dunlap: Yeah. So my name is Chris I'm the Director of Innovation at SEP, and this is Andrew Dunlap. He's one of our senior AI engineers. And today we really want to leave you with two things. One is sort of a sense-making framework, a mental model for how to make sense of how your teams should work with AI. We've heard talks already about how people are incorporating AI into the product. We're talking a little bit about how do you equip your teams to better make use of AI to write software. So we want to give you a sense-making framework. And then the second thing we really want to do is give you a set of principles to help you get started.

38:42Andrew Dunlap: So if you're going into a team and it's like, hey, we want to use more AI or we want to do something different, how do you get started? Where do you go? We're going to give you a set of principles to hopefully help with that. Now, if you've paid attention, there's lots of studies around AI. Here's three relatively recent ones. The 2025 door report is the oldest, but lots of studies about how people are using AI and making use of AI. And I think it's interesting that these three sort of have the same pattern or what comes out of it is the same. And they all essentially say, AI is not going to fix your team.

39:16Andrew Dunlap: It amplifies what's already there. That's really important for us. It's a question we have to ask when we start enabling teams with AI. See, we build software for companies where being wrong matters. We're going to write a lot of regulated space. We work healthcare, pharma, aerospace. And so we think long and hard around what it looks like to enable a team with AI. And that's where the title comes from. We don't believe that every team needs a dark factory. We don't think you need to be all the way there to have some significant productivity gains from AI. And in fact, I would argue if you're trying to move there and you're operating in the wrong mode, even the best of teams are going to struggle.

40:05Andrew Dunlap: It's going to feel a little clunky. And so what does it look like to struggle? What does it look like that you're operating in the wrong mode? So you may hear something like this. We're shipping AI slop and nobody knows what right looks like. Or we should be using more AI. Well, I don't know what more is, but like we should be doing more. Or the joy is all gone. All I do all day is review code. And if you've heard one of those things in the last week or been in a meeting, heard something like that, like you are in the right session. When the way you work doesn't fit your situation, like more AI is not going to fix that.

40:50Andrew Dunlap: More AI won't fix sort of the AI fit. And Andrew is going to walk through three examples from three different customers of ours, three different teams that are all using AI in a slightly different way or enabled with AI in different ways. And we're going to get into a little bit like why that is. And we think all three are operating where they should be.

41:19Heidi Porter: Thanks, Chris. So let's talk about a client we have, their transmission operator. And so they have these simulations. And the simulations have these models that convert from one form to another. This is a legacy system. It's been used by the business for a long time. It gets the job done. But they want to improve it. And so they had a clever engineer one day said, hey, I got these input models. I got these output models, and I got a whole bunch of agents. So I can just reverse engineer them and figure out the process in the middle. Makes sense. Two days later, lo and behold, he has a working prototype.

41:55Heidi Porter: So they want to get the prototype to primetime. Got to get it to production. That's when we get the call. We bring in a team. They get started. A week passes. We haven't delivered any software yet. What happened? What happened was there's new forces at play. There's some new constraints the team's facing.

42:16Andrew Dunlap: Yeah, so you're saying the constraints are what impacted the team's ability to move as fast as we wanted?

42:24Heidi Porter: Exactly. So there's six constraints that we found all our teams want to review when they're pinpointing the forces acting on them. So we have token budget. This is a really common one. Maybe an organization is all in on AI, but how comfortable are they spending$10 a day for tokens? $100,$1 ,000,$10 ,000? That changes the conversation for the team. Organizational readiness. Do you have a sponsor? Do you have someone that's willing to back this effort? Goal definition. Do we know where we're going with this? Are we trying something out? Are we experimenting? Domain knowledge. how much of the domain is written down, or how much is in people's heads.

43:06Heidi Porter: That'll affect the outcome when agents get into play a lot. Verifiability. Can you automate how, if you've gotten to write yet, do the agents know? Do you need subject matter experts? And then code oversight. This is a big one for regulated industries. If you need eyes on the code, that's going to change how you interact with the agents, right? So if we go back to our two-day prototype, What happened? At organizational level, they had some token budget constraints. When we moved to our team, they had a budget that only allowed a couple days of work before the devs ran out of tokens. At product level, there was too much domain knowledge and heads, so they weren't able to have the agents get to what the right answer was because you had to ask a person.

43:55Heidi Porter: Then at the product level, they found out, yeah, the prototype could be verified by one person, but when we scaled it to production, now we had millions of input parameters, and we didn't have a great way to scale that up to verify every output was good. So what these end up being are kind of knobs or dials that your team can hone in on. Now, some of the dials are going to be fixed. You may not have a say over token budget, but pinpointing that constraint brings up conversations that happened for our team, highlighting that these token budgets were an issue, highlighted to the organization that this is something that they need to weigh in on.

44:32Heidi Porter: For the domain knowledge, it changes who becomes interacting with the project. Okay, now we need subject matter experts here because we don't have enough information to the level of automation with the agents we want. So pinpointing these constraints is the first step of what leads into the next section, which is what mode the team operates in. So there's four modes that we commonly find our teams operating in. And now there may be more, there may be less, but this is a good delineation of very distinct different modes of operation. And they're spread out by a level of agent autonomy. So on the left, we have this individual mode where there's going to be a human in the loop.

45:13Heidi Porter: Quite often, they're going to be doing their own thing. As we move to the right, you can see we land on the dark factory. The humans have left, the lights turned off, the robots are running into place, and our product's coming out at the end, full autonomy. So for our two-day prototype, they're working in individual agentic coding mode, right? Because they were just getting a prototype out. They didn't have to worry about scaling too much. They could verify one output and say, yeah, we got a process working. Once our team scaled, that kind of broke down, right? That was bad fit. There was friction there now.

45:48Heidi Porter: So let's look at a different example for a different mode. So the team I'm working on, we work on some software that designs building components for a client that manufactures them. So building components, this room's actually fantastic. If you look around, we've got some concrete blocks and there's some steel beams. And if you look up, there's some joists and HVAC units. So building components. So high cost of failure, right? We don't want buildings falling down. How do you weave agents into that? because I bet you won't be too happy if I told you, yeah, the building will stand up because our agents verified it.

46:23Heidi Porter: So we started in an individual mode and we wanted to move to the shared mode. And to do that, we had to build a shared pile of information. So what that looks like is we have domain elements, architecture. We may have workflows, rules for how to implement, code, plan. But it's all in one body of knowledge that everyone has access to. Now, the developers day to day may still kind of work on their own. They may pair with the agent. But now we know what we're doing, right? We all have the same domain rules. So the point of this is now we're giving the agents a little bit more autonomy here. And the shape of that organization differs based on the constraints that our project allowed.

47:07Heidi Porter: We could have gone further, but we didn't want to because verifiability was really important for us, right? Buildings don't fall down. That's the goal. So we weren't comfortable going further. We made that choice. This was a good fit for us. Okay, what if we wanted to go further? We have a team that did that. So we have a team that worked on a mobile banking app for Forum. If you're not familiar, Forum's a local credit union here. So they had a banking app. It worked. It was in C Sharp, and they said, hey, we wanted to go to iOS and Android native. Yeah, great. Wonderful. What about tokens? Oh, we don't care about tokens.

47:43Heidi Porter: Spend all the money you want. Let's just see what we can do. Okay, so you have all the requirements set up here. Just want to go to another code base. There's no cost. We have backing by the organization. So we built something that looks like what you see on the screen now. Now you can see that we're losing individuality. We're talking about a team here. and a workflow of agents. The agents are starting to take on roles of their own. They're talking to our orchestrator. The team is providing intent, maybe some context. They're checking on the repo, but all the individually has been lost. And what we saw here was we had a greater level of autonomy and that was reflected in a 6X reduction of effort for the project.

48:26Heidi Porter: So you can kind of see there how we're moving up in the steps, right?

48:30Andrew Dunlap: Right. So the goal then, you went from individual to shared door. Like the goal is to get to Dark Factory. That's what we're trying to do, right?

48:38Heidi Porter: Maybe not necessarily because the goal here is fit, right? That's how we started. Constraints lead to modes. The mode is your fit. So what we do with our teams is we have our six constraints. We have them evaluate them. Depending on how the constraints evaluate, you may have a growing level of area that you can operate in. more constrained areas, you're going to work in a more individualistic mode. Maybe you're working on a prototype. Maybe you have a lot of constraints. Maybe you need to make sure every line of code is right. Maybe you just don't have access to the internet. Probably doing something more individual.

49:13Heidi Porter: As your constraints grow, we get that higher level autonomy, like we saw with the forum team. But it's about identifying those constraints and picking the mode that best operates for you, right?

49:25Andrew Dunlap: So you might be asking like, what does this look like for me or my team? What should I be considering if I understand these constraints? How does that actually play out? I spent many years coaching teams and clients on Lean and Lean software development. It turns out Lean has an answer to this problem. Lean, first off, would say there's no single blueprint. There's no one right answer for everybody. Again, the reason why we don't all need a dark factory. and so Toyota never told anyone to hang an andon cord and you don't know what that is it's it's a cord they would put into the factory next to a line and anybody could pull the the cord and stop the line if they saw a problem manifest it they could stop the line fix the problem and continue on and that sort of tied to Toyota's principle of build quality in like don't wait to the end to inspect it but like stop fix the problem right now and build quality into the process as we're going Now, interestingly enough, you may have heard of an anti-accord before, may not, but Toyota did not put those on every single line in every factory.

50:29Andrew Dunlap: It was very constraint-based. Based on the constraints of that situation or line, they might implement this. The principle build quality in, that applied everywhere. So principles, think about this, are true everywhere. The practices aren't. The practices will vary depending on, like Andrew said, your constraints. And so in 20 minutes, like I can't tell you exactly what your practices should look like. But what we can do, I think, is walk through a set of principles that's going to help you get started.

51:06Heidi Porter: Exactly. Okay. So we got five principles. Good news. They're easy to learn, hard to master. But we're going to go through them right here. So we're going to start with the foundation, shared infrastructure. Call back to my team. We went from individual to shared. We started pulling the knowledge into the code base so everyone had access to it. It's shared infrastructure. Next, we have deliberate iteration. So this is a key one I see a lot of teams fall through. They are doing things. They may even have shared infrastructure, but they're not talking about it. Maybe one person is having a terrible experience with a model.

51:43Heidi Porter: maybe one person thinks the security audit on the review is just noise. Until you schedule a block of time in the week for that and talk about it, it's not going to improve. Next is making it safe, which is verification before velocity. That's exactly what happened to our team in the prototype example, right? We slowed down. That's because the scale changed, the constraints changed, and we needed to make sure that we were getting the right answer before we went fast. Once we're ready to go fast, Next, we're looking for systematic bottlenecks. Review is going to be a sticking point. If review is a sticking point, you need to make it not a bottleneck.

52:21Heidi Porter: And once you do that, it's going to move somewhere else. You have to keep chasing them. Remember when we called back to the research? The team gets amplified in all their attributes. So the bottlenecks are just going to be worse than whatever they were before. You're going to feel the pain. You have to keep chasing those to make the team fast. Finally, remove the toil. Script the things that happen over and over again. Save the thinking for the humans. Save the analytical processes for the LLMs. It's going to save you tokens. It's going to be faster. It's going to just make life better for the team.

52:57Heidi Porter: Five principles. Three teams, same principles, different modes, different practices. Get it? So that's it.

53:05Andrew Dunlap: Thank you. Let's go. That was awesome. I think basically everyone here who manages a dev team should make that talk required listening. That was awesome. And yeah, those sorts of questions are really the questions that we're all asking as we're advancing our team and understanding how to work together again.

53:37Kevin Solorio: All right, that's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner, Prediction Guard, for providing operational support for the show. Check them out at predictionguard.com.

54:01Andrew Dunlap: Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now. but you'll hear from us again next week.

From the publisher

In this special episode, we bring you highlights from three talks at the Midwest AI Summit. Heidi Porter explores how to build reliable AI-augmented workflows using the right tools, evaluations, and guardrails. Kevin Solorio shares why building AI with frontline workers, not just for them, is key to earning trust and driving adoption, emphasizing collaboration and continuous feedback. And Chris Shinkle and Andrew Dunlap of SEP discuss how software teams can find the right level of AI autonomy based on their goals, constraints, and verification needs, sharing lessons from teams using AI to build software more effectively.

Featuring: 

Links:

Sponsors:

  • Hostinger: Create your website today. Go to hostinger.com/practical and use practical for 10 percent off now.

Resources and Events:

Chris’s new book The Art of Swarming will be published soon, written to be the seminal foundational book for the fledgling swarming industry, defining the principles and architecture that enable autonomous agentic UxS swarms with emergent unified embodied intelligence.

More from Practical AI

All episodes