210 | Your Next Hire Is an AI Agent: Building Agent Teams That Actually Work with Jake George

25:06This is the type of actions that you perform. This is why you're doing them. This is the outcome that we want. So very high level overview. Usually people stop at this and think that that's enough. And it just is not like AI can do a million things, but it only does exactly what it's told. Like it tries to do exactly what you say. It's like, if you're talking to someone and they take everything you say literally, and you're like, that's so annoying. Like, come on, use your skills and infer stuff. AI, it's getting better at that for sure, but it will still try to do what you say kind of literally.

25:36So you have to describe things like you have to be exact. And that's one of the things that we see people are not very good with, especially when we start with clients of like, oh, well, I just kind of, you know, I want it to like write me marketing emails. You're like, okay, but like, what's your process? Who do they go to? What should they, what should be the content? Where do you get the content? Who writes these? What's your strategy to write them? What, you know, there's a million little details that go into it, but a lot of times people are not so good at like breaking down and, you know, being specific on what they actually want.

26:04So from role and objectives, now we have a subsection under here about the core capabilities. And this is sort of like, this is how it does the task delegation. It leads the team of other agents. One of them is a CRM manager. One of them handles emails. It can also do cross-platform research to read emails and CRM data to make decisions. It goes into the decision making and then user alignment as well, just to kind of say like, hey, before you do like irreversible actions or like add or update leads or something, check with the user always. This is another thing that is very important to add. We never just let an agent loose into a company on the first go.

26:41We always add human in the loop and then a testing period. And we try to make it as transparent as possible, which is why we love using Slack. Because if they're already in Slack and they say, see that the agent is like, hey, I'm going to do this. Hey, I noticed this. So therefore, I'm going to do this. And they become more comfortable to the point where, you know, after a couple months of that, typically, then they'll be like, okay, put the, at least these parts on autopilot. Like I know it's going to do those. It always does them right. You know, I have confidence in it, but that's very important to let clients know like what is actually going on and yeah, just let them have their input on it as well.

27:16Cause it won't always get everything right on the first go. I want to pause you just for one second, because it's a very important point. The, when you're developing these things, the expectation should be, it is not going to work properly the first time or the second time or the fifth time. On the 20th time, it's going to work okay 90 % of the time, which for some of the tasks is fine. And for some of the tasks, it's unacceptable. It just depends what the task is. And the way to solve this is by testing it with live environments, meaning letting it run on actual data and then seeing what it actually does.

27:51Now to make sure, like Jake said, it's not creating havoc and trashing your best clients, is to put a stop point that lets a human check what they're about to do. And you could do this in Slack. I'm doing this as well with like tasks. So you can open a task in Notion or in Jira, Asana, ClickUp, Monday, whatever it is that you're using for task management, you can open a task. And until the user closes the task or confirms whatever the thing is that it needs to do, until that, the agent will move forward. and that's your way to increase your level of control and understanding of what the agent actually does.

28:30Then you go and fix it, you do it again, you go fix it, you do it again. Like Jake said, until you get to the point, like, okay, I don't need to check this thing. It's correct every single time. You're just wasting my time. The other thing that he does doing it in Slack or in your task management platform is that it's just another person on the team, right? That tells you what they're doing and updating you regularly. And what I tell people all the time is that building an agent or in general using these models is like having the best intern on the planet. It is going to do exactly what you tell them.

29:01However, if you don't give them all the information and you don't explain to them exactly how to do the thing you want them to do, don't expect them to do the task properly because they're still an intern. And so this is the approach, right? If you're going to break it down step-by-step and give them all the information just to give a person to do the task when they don't know anything about the company or you or the process or the project or the tools or whatever, give them the time, the resources that you would have given a person. They will do the task amazingly well. And they take feedback amazingly well.

29:33And so let's continue. So I, sorry, I broke you up, but this was a very important point. So you said, if you can scroll up, we started with a general kind of like task setup. So that was, you called it role and objectives. The second thing was your core capabilities. So these are the things that you can do with a bullet point that describes each and every one of the things. What is the third component? So then the third component is we have for this one goal. And these are not, I want to go into like, these are not, we don't always start with like, oh, you must have goal, you must have role and objective, so on and so forth.

30:09We really like every prompt is built differently and it should extend based on how complex the task is. And it should be a very iterative process. So it's like, sometimes I'll put goal and role and objective. Sometimes that doesn't need to be clarified. But when we're doing something like the manager agent, as you can imagine, has the most extensive prompt, and then the sub agents are a little bit less. And then the agents within workflows or tools and stuff are even less and more specific. So these are not like hard set things like you must have this or the prompt will not work. You really build the prompt around the task and doing it in sort of like an iterative testing phase.

30:44And this is what a lot of people get wrong they use ai for it and not that ai couldn't write a good prompt but there's one specific reason why ai is typically not as good at writing prompts as humans is because it doesn't go and test them it just says hey here's a whole bunch of fluff here's like 18 paragraphs going over everything that it could do and when you do something like that and you paste it in you don't really know what's working and what's not working and you also don't know how to change it because you don't know at what point it started to go off track so it's really best to start with like just the very basic things, like just give it like a one sentence role when you start and then like a one sentence goal.

31:20And then rules, I always just add as it does things wrong. Like that's the reason I add rules, unless I know like this is something that AI always gets wrong, or just some like rule of thumb best things to add in there. I just add these as it makes mistakes. I write all of my prompts by hand, like I write each word in them. Sometimes when I'm done with it, it might be like maybe a little messy. So I might send it to AI and just say, hey, format this better for me. But the most I ever put in from AI is maybe like one line. Like, let's say I try to explain something that is kind of a simple concept, but maybe when I write it out, it comes about this long because, you know, when people write, they're just like writing as they think.

31:57And I go, can you make this more concise and clear? And then if I read it back, I'm like, that's good. I might paste it in. But what I never do is be like, write this entire section because it's just guessing on what it should do, what might go wrong, what the rules should be. So I highly recommend that you don't use AI to prompt, especially if you're, you know, if you're like an absolute master at it and you built your own agent to write prompts, just how you'd write them and so on and so forth. Sure. I'm sure that there are people that are really that good, but it's like, I write all of it by hand and I recommend people do it too.

32:27It's a good thinking exercise and it's a good learning exercise. And if you always use AI to do it, you just won't get better at it. And that's why a lot of people struggle at these sorts of things because they never actually learned. They're just, oh, AI, do it for me. that I know what's good about it, bad about it. Anyways, the next step is rules. And then I break these down into subsections. So again, rules, bulleted list. And a lot of times these are, like I said, they're written because it messes up. And then I say, okay, you know, do this because it's screwed up on that. So that's the reasoning for rules.

32:57I break them down into subsections based on like, you know, the workflow, communication, emails, how to use tools. I think that there's one planning and reflection. reasoning strategy as well. So I guess this is under workflow processes. But this is another thing. So then when we have exact processes, because the cool thing, we were kind of talking about it before, when you have hard set workflows, sometimes you want things always done a certain way. When you use an agent, it's almost like an AI build your own workflow, because it has all these tools, depending on which order is using them. Every time you run it, it can build its own workflow.

33:31That's what's really cool and useful about it and makes it more similar to a human. However, there are some times that it's like you want to tell it, like always do these things in these order, like don't go and send an email before looking up data about the client because then you'll have no context on them. You would think that's obvious. It's obvious to a human. It's not obvious to an AI unless you specifically tell it exactly what you want to do. Also explaining reasoning strategy, things like plan extensively before each function or tool call, take a moment to outline the steps and then always like think between using tool steps, because sometimes another thing a lot of people don't think about, it will just run tool after tool and not really think about the returned inputs and try and do all the thinking at the end.

34:12You can actually change this by telling it, like think between the tool steps and then make the next decision. These are actually outlined in an OpenAI paper about the 4.1 models, because it will follow prompts a little bit more literally than the previous models, which would kind of try to infer more. And you can actually get a 20 % better result when you explain the reasoning steps and use this part here. And then there's another part at the end here, just telling it you're an agent, keep going until the user's query is completely resolved before ending your turn and yielding back to the user.

34:45By using this and then a reasoning step. And then there's also one other one, which I don't think was necessary in here about like reviewing data, or it may actually be in tool usage. But using those, you'll just naturally get about a 20 % better result from it just by reminding it of these sorts of things. And so it's really interesting because a lot of people don't, you know, they wouldn't think deep enough to think I need to explain a reasoning thinking process to get it to think like me and do things like me, but it really significantly improves the result. Next, we have steps. So this is still under workflow processes.

35:19So steps for researching a person, CRM and email. So again, we want them to always look in the CRM. We want them to want it to search for recent emails. And then we want it to review these and then see another thing that it does in the background is look for any missing data on that person. And if it can find that missing data, like their email or their address or their LinkedIn or something, if it can find it in an email from them, it will actually go in autonomously, add that to the CRM. Then it compiles a summary. So this is also important. A lot of things, if you, I'm sure that you've heard of it, maybe not a lot of other people have, depending on how much they build agents, but they sort of like context engineering of like, you can't just, there's usually two sides.

36:02People give it like a one sentence instruction and it's not very clear, or they just dump everything they can possibly imagine into it. And either of those is a great strategy. If you overload its context window, they just naturally like, yes, it might have a million token context window. That doesn't mean it will process all of that properly. AI tends to skip over a lot of parts. So you want to only give it really the necessary data. So when we do, especially in subagents, we kind of have it like summarize the data and then pass it onwards. It helps preserve its context window. Next, we go into the tools section.

36:36This is very important to explain. We give in here like a broad overview of what they do. and then it's best for structure to explain more in here, which I don't know if this one, yeah. So this one is pretty good. So this has like a more detailed explanation of how to use the tool, what the tool can do, sometimes when to use it, depending on - I want to pause just for one second for the people who are not watching the screen. Sure, yep. For each of the agents in every tool you use, it doesn't matter which, whether they're using NA10 or something else to build the agent, the agent had a section of the tools it has access to.

37:12You're literally connecting a line and giving it access to different tools. A tool could be a CRM. A tool could be something you build. So a process that you build, it's something that the agent can use to achieve the goal that it was given. And so what Jake is showing right now, he's saying that on the high level agent in the actual prompt of the agent, when it's describing to the agent what it can and cannot do, it's giving it a short description of the tool. Inside the tool itself, there's a much longer description of the tool, which makes perfect sense, right? Because you want the, it's, think about it just like Jake mentioned, think about the CEO.

37:51The CEO doesn't necessarily need to know exactly how to do each function in the company. It needs to know the function exists. It needs to know what the function does. It doesn't need to know how to do the thing. But the guy that creates the reports from the ERP to the inventory manager needs to know very, very well how to do this one very specific thing where you're going to define that tool in much more detail. In this particular case, this could be a Zoho CRM data query, which is what we're seeing right now. So that part has to be very, very detailed. But on the high level agent, the CEO agent, the orchestrator agent, you just tell it that it has that functionality.

38:36Exactly. Yeah, you summed it up very well. And kind of like a way to think about it in sort of a human level is think if you're given, you're an intern, you just joined the company, they give you this long list, here's all the things that you do, here's the processes, they have it all documented. You're not going to every time you go to do a task, read the entire thing and remember each step and whatever, you're going to look for that section. Oh, I got to update the CRM. Let me find that section. Okay, I need to use this tool. Let me go read how to use the tool. And that's kind of in a nutshell, AI doesn't think exactly like humans, but kind of like it pays attention to different parts depending on what it's doing.

39:09So that's why the structure of the prompt is extremely important. And, you know, it will pay attention to different parts of it depending on what it's doing. So you want to break it down granularly. And you yeah, you hit it right now right on the head that you know, you give it a very brief explanation of how the tool works. And then in the prompt, and it's also important to keep the naming the same. So don't call this one like Zoho tool. And then it's just the other one is like CRM managing workflow or something like that. Like I always will name them this sort of way and then reflect the name exactly within the prompt.

39:42So it knows, you know, be very specific and exact when you're speaking to an AI. So very high level overview when you go into the tool, it makes more sense. Give all the details there because it doesn't necessarily need all the steps to use it every time. If it's doing an email managing task, it doesn't need to know all the steps that the Zoho managing agent does in here. Lastly, we have vector stores. This is if you have external databases, vector stores, anything like that that it might need to reference, we just tell it, this is what's contained in here. A really cool thing that we've done, what we do for a lot of managing agents, but this one as well, is that it actually has a shared memory where all the agents will output to a database in Supabase.

40:26And they'll kind of like write down like, here's what the input was. Sometimes it's reasoning steps, and then the output. So this makes it like a whole unified team, because if they each have their own separate memory, they won't always know like, what was actually done. They just know user input output, that's it. And that's a lot of times not enough context, especially in multi agent systems. So by giving them like a shared memory, it's like having a group chat. Imagine like if you're trying to coordinate something with your team at work, if it's all one-on-one conversations, no one knows what's going on.

40:55If you do it all in the group chat, everyone kind of stays on board and they're like, oh, and if they don't know what's going on, just go and check. So we do that here. You can also use this as more of a structured query database rather than like RAG, which depends on what the use case is, but there's different ways to query this, but that is essentially the gist behind why we do that. I'm going to pause just One second for the people who get diarrhea when they hear the word database or Superbase. So you don't have to worry about it. Like it sounds very technical, but the reality is the tools today became really, really simple.

41:32Like literally, if you go to Superbase, which is one of the most commonly used tools in these worlds, it's a database tool. And you can set up a new database and give it a name. And that's more or less everything you need to know. And from that moment on, the agent, or in this particular case, all the agents can call that database, which is just a storage. It's a container of information where they can all save information to. And you tell them what information to save. So when you take an action, announce it to the database. When you post something to the CRM, put it in the database. Like you tell them what to save and it's called a vector store because these AI databases are built on vector information.

42:15So basically it's like a length and a direction. But again, you don't need to know all of that. You just need to know there's a container that saves information and saves data that all the agents and all the tools have access to. And that's their way to collaborate and have a unified knowledge of what's going on across the different processes. Absolutely. And there's in a nutshell, there's two ways that databases are commonly used for agents. And one is just their like chat memory. So like maybe the past 10 things that were said. And then the other one would be the vector database. And usually that's when people say I trained an AI to do something.

42:54Usually what they mean is they uploaded all this information to a vector database. And it's just like, imagine having like a user manual for something. You can go in there and look up any data when you want to. So there's like chat memory. And then here's all the data that you can look up if you need it. And that's how those two will typically operate and why they're a little bit different there. Then at the end here, this is super effective and important. The AI models will pay attention to the things at the top and bottom the most. And I believe bottom a little bit more than top. So when you have a very long prompt that it might kind of like skip over things on, it's really good to give it a recap.

43:35And again, I do this iteratively of like the things that it might forget or is not doing well that you're like, it's in the prompt. It's just not doing it. Just restate it at the end, because the more times that it appears in the prompt, the more that it will adhere to that instruction. And also, if it's at the end, it pays more attention to the end and it just makes it that much more effective. For this one, it follows the instructions very well, largely because they're well-written, well-structured and everything. The thing I put at the end is from the OpenAI paper about how it's an agent and keep going until the user's task is complete.

44:07The AI models, they just understand what agents and agentic frameworks are. And so then it just thinks more like an agent rather than like a, you know, you send me a question, I send you an answer sort of thing. It just makes it perform better. So that is how this main agent works here. I can definitely go into maybe a sub-agent now, or I can even show like sort of how the outputs look like in a user experience sort of way from Slack. So we're short on time. I want us to do two things very quickly. One, let's open just one of the sub-agents to just people see what it looks like. And basically, if you think about it, a sub-agent is kind of like a tool for the main agent.

44:46and then you can go as many layers deep as you want. Or a process could be a tool for an agent and so on. So you can see now that Jake opened a subagent, it has a lot less stuff. So let's just go over the components that this is connected to so people understand. And then maybe we'll look at one run and see how the output looks like. And I think that's going to be awesome. Sure, yeah, for this one. So yeah, it is much simpler. And one of the reasons for that is because its input is just directly, it's just a natural language string from the manager agent. We don't have to worry about processing the data from Slack, having it reply, so on and so forth.

45:24So what this one does, we actually use some different MCP servers here. These ones are more of like workflow ones that we had to build different steps into. These ones are more like direct actions of like getting deals, leads, many deals, many leads, accounts and contacts from their CRM. And then it also has, so the Postgres chat memory here is the database, like I was saying, where it's just the chat memory, like it has, you know, this many messages that happened, helps keep it on track a lot more. And then as you can see for the prompt, it's structured very much similarly to the other one. We have like more step-by-step processes in here, explain all the different tools, so on and so forth, how to do different processes in here.

46:06But yeah, this one is mainly focused because it doesn't need to know all the instructions of everything. It's really just focused on like, here's how you complete the queries that will go to you. Here's the tools that you have available. Here's how you would use them. Also using the think tool is quite useful as well. It's, you know, allows it to kind of have like a sort of like a scratch pad where it can just like put down thoughts and kind of like reason through them. And it just, it helps it think better through a longer, more complex tasks that might take many different steps. So that is one small thing.

46:35First of all, great recap of what the subagents too, you touched on MCP servers. MCP servers are the ultimate AI connectors, and they basically are built either by the companies themselves or by third parties, and they're exposing the functionality of a tool in this particular case, Zoho, but this could have been Salesforce, ERP, email platforms, knowledge graphs, social media, whatever has an API that connects to it, it exposes it to an AI in a very, very simple way. So instead of having to develop APIs for weeks with a team of 10 people, you write four to 10 lines of code and you have access to everything that the tool has to offer.

47:21So when Jake is saying I connected MCP servers to provide me information or take actions in Zoho, what he means is that they've connected specific functionality from that into this agent, which again, sounds really, really complicated, but it's a lot easier than you think because that's what MCP does. It makes it very easily accessible. And then in simple English, you can tell the agent, hey, I need you to go and do this. And it knows how to go and grab that function from the MCP. And then it knows how to call, in this particular case, Zoho and say, hey, I need to know all the new leads that came in in the last 24 hours.

47:57And then it just works without you having to figure out what's actually happening in the backend. Yeah, it makes these things very quick and easy to develop too. You just simply go, you make your MCP server, you throw all the tools in it, and it makes it very easy to spin these things up very quickly. One more, I can show sort of like a demonstration. Let's see how this looks like on the Slack channel. So people understand what the output looks like after all this detailed work. Let me go into here. Okay, Slack.

48:39Okay, do you see Slack now? Yes. So what we have here, this is kind of like the, this is another one that we're working on, but this is kind of like a manager agent here that has a whole bunch of different sub agents. Joe named it Jarvis for this one. But these are sort of like the functions of how the different agents would interact with the user. For example, he says, hey, can you look for information on himself? So he's going to go ahead and look through the CRM. It says, hey, here's the CRM summary. I also look for all the emails from him as well, and then ask the user what they would like it to do next.

49:14So sometimes we use structured. Another really cool thing that you can do is use Slack blocks, which I'll give a visual demonstration of in a second. A lot of times some agents we use like unstructured text like this because it can be so dynamic. but other ones that are focused on a very specific task. This one, for example. So this one is the email managing agent, which is under the sub agent. So again, like I was saying, sometimes if it's just like, hey, I know that I should write an email to this person. We don't need Jarvis to tell us and then the email agent to tell us. It's just duplicate.

49:44So then Jarvis would choose, hey, email agent has it. I don't need to respond. Then the email agent would send this. So it's just like, hey, this is the email that came in. And then here's the new one that I'm going to send based on that. So then we have it referencing past emails. So it knows exactly how to write those, knows what's going on. And then to make it very easy, because it's like you can have sometimes to keep it better on track, especially for clients, because it's like in your mind, you might know exactly all the 8000 things that this agent does, but a client does not. So sometimes it just makes it easier to just like these are your four options and just give them some buttons because it's like they either want to send it, draft it to be rewritten or just decline it in general.

50:24So it's very useful to kind of just give them those direct options. Yeah, for people to understand the way you do this, you build, and again, now we're getting way too technical, but you build a Slack tool that has these functions that then connects in the backend to the functions in the AI agent. So that's how these two universes interact. Jake, this was fantastic, very detailed. And I want to do a very quick recap. And then I want to let people know where they can find you. So agents are awesome, especially when you combine them with structured processes and the right data sources and the right tools.

51:05And that's the way to actually make them work. And what Jake showed you literally openly, which is incredible because it's work he actually does for his clients, is how to think about it. So how to create the architecture, how to build the layers of the different things, what kind of tools you need to think about, how to connect them, and what are the actual problem structures of a master agent and then the sub-smaller tools level or specific operations level agents. And then we showed you a very quick example on how that can work while connecting it into Slack as the way to communicate as the, if you want the front end, the user interface for these agents.

51:46So the users don't need to know any of this. The end user just sees a Slack channel and they talk to the Slack, just like it's a person and they can communicate back and forth. So again, incredible, incredible work. This is extremely helpful for people. By the way, I don't expect anybody after listening to this podcast to be an agent ninja, but it gives you a very good idea of what it takes to actually develop an agent. And then you can call somebody like Jake to actually build it for you, but you will have a much easier work of explaining what you actually need, which is a big, important part.

52:22So, Jake, if people want to learn more about your work, work with you, follow you, see your content, what are the best ways to do that? Yeah, so definitely connect with me on my LinkedIn. Hopefully you can just drop a link in the description or you can just search for me by name. And then also at my website, which is just agenticbrain.com, which is spelled right here. Awesome. Really, really great. I really appreciate you. I appreciate what you're doing. I appreciate the content you're sharing and keep on doing the hard work. And for everybody listening, if you want to learn more about agents, we have several other episodes on several other platforms that are not NA10 so they can broaden your horizons, but they all follow the same exact concept that Jake beautifully laid in front of us.

53:05So thanks again, Jake. I'll see you around. Thanks everybody for listening. Thanks for having me.

53:13you