Tool calling and agents

0:03Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, productive, and accessible to all. If you like this show, you will love The Change Log. It's news on Mondays, deep technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at Fly.io. Launch your AI apps in five minutes or less. Learn how at Fly.io.

0:44Welcome to another fully connected episode of the Practical AI podcast. In these fully connected episodes, Chris and I keep you updated with everything that's happening in the AI world. If we can, there's a lot. And we try to give you some learning resources to level up your machine learning and AI game. I'm Daniel Whitenack. I'm CEO at Prediction Guard. And I am joined as always by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris? I'm doing great. I just don't know what we're going to talk about because just nothing ever happens in AI.

1:23There's nothing happening. There's never anything going on. Elon has done nothing. Ignore it. Oh my gosh. Elon is throwing spitwads at people again. Let's say he's been suing OpenAI and now He's put his bid out in the last few days for open AI. Yeah, we are. The article that I saw was, we are not for sale, chat GPT boss says. I know. Good old Sam. Sam said, we're not for sale. Because the two of them really love each other. Oh, yeah, definitely. Elon Musk and Sam Altman, they are best friends. Best friends. That's how we'll report it here. That's how we're reporting here because we always look for the upside in the AI world here.

2:11Yeah. Who knows the motivations behind billionaires? It's an interesting thing to watch. It certainly spices up conversations in the workday and is a nice point of discussion with friends. So there it is. That's about how I'm taking it. Yes, I would agree with that. The spats between billionaires just doesn't quite make it to my, not being a billionaire myself, it doesn't quite make it onto my list of concerns. Yeah. What do you think is kind of the trajectory with players like OpenAI? You know, there is someone someone was well, the other thing that happened in the US was this this fits into maybe the other thing I was going to ask as well, although I got sidetracked in my mind because I remember that there was a Super Bowl and there was a Super Bowl commercial that OpenAI had, which, you know, it was a it was a cool commercial.

3:15I think I didn't know what it was going to be at first because it's just the artistic dots around, you know, forming scenes. And then I think I gradually realized that this is those little dots on the OpenAI ChatGPT app that, you know, expand and cool ad. And someone commented to me, they spent like$14 million or whatever a Super Bowl ad costs. I forget how much it was, but I was like, that's really nothing compared to what OpenAI is losing generally on hosting models and infrastructure. So yeah, that's what circles back to my other question, which was, yeah, what are your thoughts on, I mean, if Elon doesn't buy OpenAI, what's the future?

4:08I honestly don't know. And I got to be honest with you. I'm not sure that I care a whole lot. I was thinking about that as we were leading into this is that, you know, I mean, there's not a protagonist here from my standpoint. There's not a side that I'm for or against so much. um uh you know you have elon with all the you know the the the adventure around elon musk and i say that word kind of tongue-in-cheek and then open ai and you know you know there is a kernel of truth to what elon says when he talks about it being you know going from being the non-profit with the grand vision that it started out with in the early days and then it has increasingly gone commercial and become for profit.

4:56So it's another big giant AI company like the others and stuff. I'm watching it with half an eye like everybody else in the world, but not sure. I just don't know and I'm not terribly sure I care. Yeah, yeah. Is there somebody out there in our audience that is deeply concerned about this? I would love to hear somebody who is not Elon or Sam Altman tell me why this is a big deal. Yeah, maybe we'll leave it at that. It's a good point. I am interested in, you know, some of the dynamics like OpenAI released their deep research product. So if you kind of look at the trajectory of what they're releasing, what they're doing, There's this deep research product, which is really geared towards this, you know, multi-step online information research type of task.

6:00So, you know, going and looking at various, you know, trends across various sites with various data, reasoning, certain information, consolidating that, you know, contributing to some sort of research project. And I find it interesting that OpenAI introduced this. One of the dynamics I love watching is OpenAI releases the application level product, so like deep research. And then, so I see the blog post by Hugging Face, it was like the day after. So they say yesterday OpenAI released deep research. So this is a blog post that I'll link in the show notes from Hugging Face. And basically, they just decided to make sure that they could reproduce the functionality with open source code, maybe some recently released models like DeepSeq models or others in 24 hours.

7:03And then they wrote the blog post and released it. I don't know how long of a 24 hours it was, but you see that dynamic happening. So you see that with deep research and then you have, you know, the open deep research thing. You see kind of the operator stuff where it's operating your screen, your browser window. Now, earlier today I was running Hugging Face Small Agents. They have a web agent, which is essentially that it spins up a browser window. It does certain tasks for you in the browser window. So you can type a prompt like, hey, find the most recent episode of Practical AI, summarize the topic, and then find seven other articles of a related topic, list them out in markdown format, and output that.

8:01Something like that where it requires this sort of agent operating over the internet. Super slick, super fun. I would definitely recommend people, if they want to try that sort of thing, try the small agent's web agent. But yeah, you see this kind of trend where at the application level, some of this is just, it seems like you can't develop a moat generally there. Now, you might be able to develop a kind of moat as a company in a specific domain or a vertical or with certain knowledge or proprietary data, right? But it's very hard at that kind of general application level, I would say. I think, you know, I keep wondering as, you know, OpenAI had had a substantial lead and it was taking quite a period of time for a while for open source options and application, you know, level things to come about.

8:58And we've seen that the, you know, that time interval shrink tremendously here. So, you know, and ironically, at the same time that Elon makes his$97 billion effort to buy open AI, but you can't help but wonder a little bit about what the future business model looks like, you know, to your point there about, you know, if it takes so, you never have time to create a moat, you know, if you're one of the main players. Now, there's certainly business models for other players to come in their industry, as you just mentioned, and create capability because that's their thing and it's not something that the big boys are going to go after.

9:41But as we've seen this interval between the commercial players and open source shrink to almost nothing, what do you think that means for the business models going forward for the Googles and the open AIs and the anthropics of the world? I mean, I think part of it is maybe this sort of integration in the kind of enterprise stack. And what I mean by that is the kind of bundle effect that you get from something like offerings from Microsoft. So, you know, absolutely no one in the world wants to use Teams because it's absolutely terrible. And I will go on record as saying that. Sorry for those that work on it.

10:26I have to use it. I have no choice. I guess you have a podcast, you have an opinion. But that's my opinion. But I'm also not going to pay hundreds of thousands of dollars to Slack if I can just flip on Teams in my Microsoft tenant and they already have all my data and all this stuff. So the fact that they're tying in Copilot and those licenses around Copilot in an ecosystem that's already so embedded in the enterprise world. There is a very strong bundle effect there. And yeah, it's very real, right? And it doesn't mean that it's necessarily the best solution, but it is a solution depending on what you're looking for, right?

11:11At that kind of generic co-pilot level. In a case where you need kind of single tenant, meaning in theory, the terms and service are my data is not being used in certain ways. That kind of gets that generic case. But again, like the real business value that a company has, the way I see it is you've kind of got these generic cases where someone random is going to want to find a Word document or paste in an email. Then you've got like the core business value, right? So a pharma company that has their most sensitive tiers of data that are the lifeblood of their company, or a healthcare company or a finance company that has certain classifications or regulatory burdens around certain tiers of data.

12:02It's a whole other thing to think about integration of those tiers of data into a generic system like that, because they're a generic co-pilot system for those kind of less sensitive tiers of data, there's still something that needs to be solved at those other layers, which is where I think vertical AI players, but also tooling and infrastructure players can still make a lot of progress. Do you think that the bundling that you're describing that's occurring between the vertical capabilities where they're producing these and open AI going and doing deep research or Google integrating Gemini into the Google suite, which they've been doing and trying to drive a premium from users for that.

12:53Is that bundling going to be critical to them going forward? Or do you think that the open AIs of the world, and we've seen this historically with Google, maybe not in an AI context always, but driving into specialties where they open up a new vertical underneath the umbrella and stuff. Is OpenAI going to have to do that to survive since it's going to have open source chomping in its heels coming down the general path? Yeah, I don't know. It could be by vertical. It could be, I mean, you look at Palantir, for example, stock price soaring. Most regular people aren't using a Palantir co-pilot in their day-to-day.

13:39But they have that a certain market, particularly around, you know, DOD or defense or other areas. They have really put a lot into serving that well with the less generic, but still fairly generic across different use cases set of functionalities. And that, you know, has served them well, at least from an outsider's perspective, if I'm looking at that. So it may be a specialization in terms of tools or vertical. It might also just be a segment of the market that you choose to focus on and is kind of the bread and butter. It's interesting because you've got all of these really end users direct to consumer traffic on OpenAI and these things now where a lot of what we had talked about before with data science and AI and machine learning was really enterprise focus, not direct to consumer.

14:35So allow me to throw one other layer onto this conversation. As we circle back around to AGI ideas with, you know, kind of having artificial generalized intelligence being bantered about, Sam Altman was just saying that he was expecting GPT-5 to be smarter than he was. And so as we look at that, you know, I think GPT-3 is smarter than I am. me too i agree with you but but with that you know with us with that the agi chase continuing at this point and you know we've heard you know with deep seek and all all these others going in and talking about business models and bundling and such and and exploring new verticals how do you think that the agi race fits into that yeah maybe that's the piece that in my mind i'm not isn't really entering into my mind much in the same way that you don't think about Elon so much, which is probably good.

15:37Yeah, I think it's an interesting question and there's implications. The questions that come into my mind at a more general level, which is what you could talk about it as AGI or not, I don't know. But the questions that come into my mind are more the downstream effects of some of these things. Are we building systems that enhance human agency rather than replace it? Are we building systems that allow us to trust more in human institutions or fear and distrust them more? Are we building systems that actually drive us more into isolation as individuals or into community together? I think those are interesting directions that are on my mind as I think about the more general side of this.

16:33Well, there's no shortage of AI tools out there, but I'm loving Notion and I'm loving Notion AI. I use Notion every day. I love Notion. It helps me organize so much for myself and for others. I can make my own operating systems, my own processes and flows and things like that to just make it easy to do. checklists, flows, et cetera, that are very complex and share those with my team and others externally from our organization. And Notion AI on top of it is just, wow, it's so cool. I can search all of my stuff in Notion, all of my docs, all of my things, all of my workflows, my projects, my workspaces.

17:17It's really astounding what they've done with Notion AI. And if you're new to Notion, Notion is your one place to connect your teams, your tools, your knowledge, so that you're all empowered to do your most meaningful work. And unlike other specialized tools or legacy suites that have you bouncing from six different apps, Notion seamlessly integrates. It's infinitely flexible, and it's also very beautiful and easy to use. Mobile, desktop web shareable it's just all there and the fully integrated notion ai helps me and will help you to work faster write better think bigger and do tasks that normally take you hours to do it minutes or even seconds you can save time by writing faster by letting notion ai handle that first draft and give you some ideas to jumpstart a brainstorm or to turn your messy notes, I know my notes are sometimes messy, into something polished.

18:18You can even automate tedious tasks like summarizing meeting notes or finding your next steps to do. Notion AI does all this and more, and it frees you up to do the deep work you want to do. The work that really matters, the work that is really profitable for you and your company. And of course, Notion is used by over half of Fortune 500 companies and teams that use Notion send less email, they cancel more meetings. They save time searching for their work and reduce spending on tools, which kind of helps everyone be on the same page. Try Notion today for free when you go to notion.com slash practical AI.

18:55That's all lowercase letters, notion.com slash practical AI to try the powerful, easy to use Notion AI today. And when you use our link, of course, you are supporting this show. And we love that notion.com slash practical AI.

19:14Well, Chris, we talked a little bit about tools and agents. Well, agents generally, the web agents, the deep research things. And we've kind of talked about tool calling and the connection to agents at certain points on the show. But I don't think we've really dug into the detail in a way that maybe will make things clear for people. I still see a lot of confusion around this. Even in my day-to-day as I'm talking to customers, the question of, well, how do I make an LLM talk to this system? Or how do I, that deep research tool, how do I make an LLM go and do a thing? That's often how the question comes.

20:09And what I think I realize when I'm hearing those questions is there's kind of a fundamental misunderstanding of what the LLM does and how it's tied into a framework, which you might call tool calling, you might call agentic. The names kind of get mushed around a lot these days, unfortunately. They do. I was thinking that as you were saying all that, and then you got, that's literally what was in my head in terms of the misuse of different names of these technologies and what's doing what. Yeah, yeah, exactly. So in my mind, so this is, I'm feeling very opinionated today. I don't know why. Go for it.

20:53Excellent. In my mind, how I kind of draw the lines here, there's, of course, models, large language models. They predict probable text. They generate text or images or whatever you want them to generate. Then there's other systems kind of over on the other side. So you could think of your email or your bank account or an external system like an Airbnb where I might want to make a reservation or my company's database, right, which contains transactional data or another system that I use like HubSpot or all of these types of things. There's all of these other things. And to ask a question, well, how could an LLM go and create a new deal for me in HubSpot?

21:46It just hurts me when you phrase it like that. It causes pain in my head. But that's how people phrase it. To be clear, these questions are the questions that come up every day. So the question is often phrased, how do I make the LLM create a new deal for me in HubSpot? So right in that phrasing, to your point, I don't know, what makes you cringe about that? It's just that's a fingernails on the chalkboard kind of moment for me is, you know, to answer that question, in the six and a half years that we've been doing the show, and we have evolved through a number of technologies, you know, that at each point in time, we're the hot thing.

22:34And inevitably, people focus in on just that for a while. But right now, we're at a point where generative and LLM the last few years have been the hot thing. And we forget that they don't necessarily do everything out there. It's not, you know, people will say LLM. In fact, they only do one thing. That's exactly right. And not only that, but there might be an AI architecture that could do the thing that they want to talk about. But it's not necessarily the thing that they're talking about. And they're mislabeling. It's not the model. It's not the model. And so that's the fingers on the chalkboard of we there's we've kind of talked about this over the last year, the tunnel vision of the generative AI era, you know, in terms of everyone focusing on that.

23:18But it's to the point that there are other technologies in the mix and there is a technology that will do the thing they want to do. They're just not picking the right one in the way that they're verbalizing it. Yeah, yeah. So let's let's maybe break this down into components. So let's say there's the LLM. You know, we'll just talk about text now. Certainly there's multimodal and all that stuff. But just think about text. There's the LLM, which all it does is complete probable text. So I could, you know, ask it to autocomplete. I could ask it to write something for me. I could ask it to generate something for me.

23:56That's what it does, right? Then there's the system. Let's say we'll take the HubSpot example since I use that. HubSpot, for those that aren't familiar, it's a popular CRM solution for those that maybe don't want to mess with Salesforce and all of that world. So HubSpot, I can create a deal associated with maybe a sales lead I have. That is its own software system that's hosted by HubSpot. And I actually, I don't know this, but I assume HubSpot has an API, a REST API, meaning you could programmatically interact with HubSpot. This is how apps on HubSpot work. An app on HubSpot is regular good old-fashioned code that maybe allows you to add these fields to these records or retrieve this data or report on this data.

24:51That's just good old-fashioned code. It uses the API. So this is a separate system. And so there's really no connection between, there can be no connection directly between the LLM, which generates text and this other system out there that's a CRM that does certain things. There's no connection between the two. Except in the middle of that, there can be this process, which I would generally say I would categorize as tool calling generally or function calling, which let's say that you wrote a good old fashioned software function that creates a deal in HubSpot via the REST API of HubSpot. That has nothing to do with AI.

25:38It's just a software function where you tell me the email of the person, the name, the company, I'm going to go in and create the deal in HubSpot via the API. So there's a function, you give me these arguments, I'm going to create the deal in HubSpot. Okay, still no connection to the LLM. But if I then ask the LLM to say, hey, I have this customer information, email name, etc. Generate the arguments for me to call this function, which takes these specific arguments. then the LLM could generate the necessary arguments to call that function. And if you create a link between the function and the output of the LLM, so the LLM is still not really doing anything other than generating text, but in your code, you literally take the output of the LLM and you put it into the input of that function.

26:36Now, you could put something on the front end into the LLM and have the result be a flow of data out of the LLM into the function and then into the HubSpot API. So that's sort of how this tool calling, function calling thing works. Which makes perfect sense. And that's standard software development. You know, that's the only thing that is different there is the fact that the function parameters that you're using have been generated by the LLM, which is a generative model. Perfect. That's what it does. And there are some special things related to this in the sense that, you know, if you look back in time at LLMs, first we had kind of really good autocomplete models because that was a meta task for people, you know, training language models.

27:27Then people figured out, oh, I kind of want to use these as general instruction following models. models, right? And so they develop specific prompt formats and prompt data sets to fine-tune LLMs for specifically instruction following, right? So here's your system message. Here's the message, you know, the message I'm providing you. Give me the assistant response. And they trained it on a bunch of general instruction following things. Well, they've done the same thing now because they've realized, oh, a lot of people want to do this tool or function calling mechanism. So certain people, including OpenAI in a closed sense, but others in an open sense, like Noose Research, who we had on the show, they have a data set called Hermes.

28:16This includes a set of prompts that are related to function calling specifically. So they've given a huge number of examples of function calling prompts to a model that they would train like a llama model. And now you have Hermes, llama 3.1.70b. It's been fine-tuned to follow that Hermes style prompt format for function calling, which means it kind of has an advantage if you like, or certain models that have been trained with these examples have an advantage specifically for that function calling task, right? So there is an AI element in the sense that some models are better at this than others because of the way that they've been trained.

29:00And there's certain prompt formats that are special and you'll get better performance if you use those prompt formats or if you use a model server like VLLM that supports or has the inbuilt translation to those prompt formats, etc. So there is an AI element of it, but it's only in the sense that you're preparing the model for this type of use case rather than, you know, connecting, there's some inbuilt connection of the model to something external. So I'm curious, can you tie in the tool calling into what would be, you know, might be considered a full, you know, agentic implementation? What's the leap there, if any?

29:43Yeah, interesting question, because people use the term agent very loosely. So some people would say what I just described, even just that chain of processing. So I put something in the front end of the LM, deal is created in HubSpot, that might be considered an agent, my HubSpot deal creation agent. I would say that's really just a tool calling example of how to use an LLM. In my mind, what separates out the agentic side of things is where you have some sort of orchestration performed by the LLM. So what I mean by that is you have a set of tools. So let's say I have access to Airbnb's API and Kayak's API and United Airlines API or like whatever other travel things I need to do, maybe my Gmail for various things.

30:51and I say, hey, I need to book a car next week for my trip to wherever, right? That input could then be processed through the LLM not to call a single tool, but first as an objective to determine what tools to call and in what sequence with what dependencies. Try to do a first step of that and then reevaluate and then do the next step until you reach the objective, right? So first, in order to book my thing, I need to know when my flight is. So I go to my Gmail and I look for the confirmation. Or second, I use that date in the Kayak API to look for choices. And then I evaluate those choices and then I use it to book the reservation.

31:38So there's a series of steps that might call different tools or systems. It could be data sources, unstructured or structured data sources like a database or a RAG system. And so that thing that I talked about, like that HubSpot deal creation tool, might be one of those tools in an agentic system where an agent could choose to use it at certain points. And I'm being I'm anthropomorphizing here. It's not choosing anything. Right. But it's useful to talk about it sometimes in that way. So forgive me. It's choosing to use that tool in one case and maybe other tools and other sequences in other cases.

32:16In my mind, that's what really distinguishes the agentic side from just the tool calling side.