#277 Alex Salazar: Arcade's Vision to Make AI Agents Secure and Scalable

0:00In this one prompt, I've asked it to do three different things. What's also important about this prompt that we're about to execute is that all of these things I'm asking it to do are ultimately on other third-party services, right? We're going to talk to Gmail. We're going to talk to Slack. We're going to talk to GitHub. And that itself is pretty difficult for agents. But most importantly, all of these services are secured. You can't access someone's Gmail unless you've authenticated as them. And similarly for Slack. So all of this is happening on behalf of me. People are learning the hard way that APIs don't work with large language model very well.

0:37You can't just hand a large language model an API definition and have it go figure it out. It might, but the error rates, hallucinations, they won't work with enough confidence to make it valuable. Our sponsor for this episode is Xtreme Networks, the company radically improving customer experiences with AI-powered automation for networking. Xtreme is driving the convergence of AI networking and security to transform the way businesses connect and protect their networks to deliver faster performance, stronger security, and a seamless user experience. Visit extremenetworks.com to learn more. That's extremenetworks, all run together, dot com to learn more.

1:29So, hi, I'm Alex Salazar. I'm the founder CEO of Arcade.dev. And at a high level, we make it possible for AI agents and large language model based applications to perform actions and connect to other systems and other services securely. And to put it in a very relatable way, most of us have used ChatTPT to draft an email, but you can't send the email from ChatTPT. And that's because ChatTPT's agent can't properly connect to Gmail and authenticate as you in order to send the email or read the emails and vice person. So we helped solve that. Prior to starting the company, I did a brief stint in venture capital at a deep tech venture firm called NeoTribe.

2:23That's where I ran into the problem. I was mostly focused on AI infrastructure and started to see where the challenges were in AI. And as a former founder sitting in venture capital, I couldn't help but get the itch and go start the company that I wanted instead of funding it. Prior to that, I was VP of product for Okta for their developer products and manager for all their new products. And prior to that, I was a CEO, co-founder of a company called Stormpath, which was an API for developers to implement authentication, login, password reset in their applications. And Okta acquired that business.

3:05That's part of its developer-facing customer identity business. So my background is heavily in authentication, authorization, developer products, and helping organizations change how they do business with their customers through software. And then a computer science degree from Georgia Tech and MBA from Stanford University. Okay. And the arcade.dev is, how would you describe it? It's a platform for building agents? Yeah. Correct. So our customers are developers, people trying to build agents themselves. So typically organizations or all the way down to hobbyists who are trying to build something fun for themselves.

3:56But our customers are building the agents and then we're giving them the ability to connect those agents securely to other services like gmail and slack and twitter and salesforce.com and anything that might be custom and internal to their organization and you i think we began talking at some point about uh and this may have been the way that we met i don't remember about a model context protocol developed by Anthropic. And I know you guys have done some work with Anthropic on developing that protocol. And I was just at a conference with IBM where they were introducing their age and communication protocol, ACP.

4:48and it's all of that sounds like it would fit in in the arcade.dev platform so yeah can you can you talk through how a developer would use arcade uh and whether ibm's acp and anthropics mcp are are the dominant protocols or whether there are a bunch of them yeah yeah it's man it's a great question um so i'll start with the protocols uh there are there is an exploding number of protocols right now but they're all in their in their infancy it is it is uh it is like the very beginning of first inning. And so I would say MCP is the one that's most talked about. And it is getting tremendous popularity because some of the coding agents that developers use to write software, the most popular one being Cursor, adopted MCP and that really exploded its popularity with developers.

5:58when people are building agents there's a lot of promise with with these protocols uh there's a lot of promise mcp in fact we're we're pretty heavily involved with anthropic and helping to evolve mcp properly specifically around off and authentication authorization but it's still very early um but the protocols are really about helping everybody agree on how an agent and another service are going to talk to each other. Just like that language between the two. So we, as a platform that helps developers build this and automate these connections and properly handle the authentication and authorization and the retry logic and all that.

6:48We're ultimately protocol agnostic. We believe in MCP and we're investing heavily in MCP. But I don't I don't think that developers ultimately care at that level. What they really care about is they want to make sure it all works. And so that's where we put all of our energy is just making sure developers are able to check the box, deliver the product they want to deliver. and we'll give it to them in the best way we can. And yeah, so let's go to the platform and you can introduce the platform and what it does. And I'm guessing with these protocols, is there a dropdown menu and you pick the protocol?

7:35I mean, how does that work? Yeah, it's a great question. So I'll give a quick demo. And for people listening, I'll walk you through it and we can all experience it in the theater of the mind. All right. So what I'm showing here is not actually our product. It's what we refer to as a sample application. And this is an example of what a developer could build using our product and the kinds of features that they could deliver. And so for the people who can't see the screen, this looks very much like a chat TPT type interface. It was like a chatbot. And so one of the big differences is at the very bottom, you're going to see this little dropdown that says toolkits.

8:23This is our terminology for all the connectors and plugins to other services that you might have in your agent. And so we've set up five toolkits here. We've got GitHub, Google Workspace, so things like Gmail, LinkedIn, Notion, and Slack.

8:46And each of those toolkits themselves are going to have a set of what people in AI refer to as tools. So each of them will have, let's call it, on average, about 10 tools. The tools themselves are the actions. And so if it said, hey, read my email, reading an email is an action. Reply to the email, that's another action. Send the email is another action. And so each of these tools have at least 10, the Google one's actually pretty beefy. That one has a lot more than 10. And so that's gonna dictate what the agent can do. So let me show you a use case. So we're gonna say,

9:27read the general channel on Slack, pull the latest activity on the Arcade AI GitHub repo and send a summary of it all to alex at arcade.dev. So what I've written but not yet submitted is what's called a multi-turn prompt. And so in ChatTPT, you'll typically say, hey, write me a poem. And that's one turn. But if you say, hey, write me an email and then summarize it. And then this other thing, the LLM, the agent, the user will have to take multiple turns to achieve what you want. And so in this one prompt, I've asked it to do three different things. What's also important about this prompt that we're about to execute is that all of these things I'm asking it to do are ultimately on other third-party services.

10:38We're going to talk to Gmail. We're going to talk to Slack. We're going to talk to GitHub. And that itself is pretty difficult for agents. But most importantly, all of these services are secured. You can't access someone's Gmail unless you've authenticated as them. And similarly for Slack. So all of this is happening on behalf of me. This is end user-based authentication and authorization, which I haven't seen many examples of in the world today. So for many people, this is the first time I'll ever see something like that. So we're going to hit enter and we're going to pray to the demo gods that it works.

11:16Live demos are always tricky. And so what's happening is we sent the prompt to the large language model. And this is an off-the-shelf large language model. We're using GPT-4.0. And along with the prompt, under the hood, we sent it a collection of like a multiple choice question of all these different tools available to it. And we sent about 50, which is a very large number, most engineering teams can't get past 10 without the LLM hallucinating, we're able to get to about 80 or 100. So we sent about 50 in this scenario. And think of it like a calculator. If somebody asks you to multiply two really large numbers, you're probably not going to get it right.

12:02But if I hand you a calculator, you're going to get it right. And so what we've done here is all of these tools are like calculator buttons. Instead of the large language model trying to figure out what it even means to look up something in Slack, it knows to hit the button that it hit here, which is the Slack get message in channel by name tool. And then it also went and hit the GitHub list repository activities tool. And that's all it had to do. And then it passes us the arguments. So it needs to know, hey, channel by name, which channel based on the prompt, it's guessing the general channel.

12:43Hey, on GitHub, which repository based on the prompt, it's hitting the arcade dash AI repo. And then we execute all of that logic for the agent. Now, when we go receive that request to go execute those tools, the very first thing that happens in Arcade is we check to see, does this tool require authentication and authorization? And in both cases, the answer is yes. And so then we say, okay, great. Is that user for whom the agent's acting on behalf of, has it already authenticated and authorized the agent to perform these actions? And if the answer is yes, we proceed. If the answer is no, then we pop up a login screen and the developer goes through what's called an OAuth flow, which almost all of us have experienced on the Internet where you go to the Google's website and there's a screen telling you, hey, do you want to allow this other application all these rights to your system?

13:42yes or no and you hit approve we handle all of that for the developer and then once developed once the user does that it sends the alert back to the agent and the agent proceeds with the requests which is what's happening here and then so that's how the service works yeah let me just go back a little bit so when you make the initial request uh the the platform is like an orchestrator It knows what tools it has access to. And rather than, as I've seen with many platforms, you deciding which tool you need to use for this workflow, it'll automatically make that decision for you. And then it's just a question of whether it has authorization, whether the platform has already gone through authorization, for example, to access your Gmail.

14:40Correct. One thing, a lot of these tools or services require an API key. is is that what you're referring to on the authorization step or is that a different step yeah the people have to go through no it's it's a great question it actually speaks to the complexity of what has to get built so um if you think of if you think of how security works at a really high level, there's a resource. Let's think your email is the resource. And then there's the subject, the person trying to access the email. And so in most scenarios as humans, it's a very simple process. We show up, there's a login screen, we type in our information, and then we're in, and then there's permissions inside the system saying, what do we have access to?

15:38Which is in our Gmail, we have access to all of it. But if we're trying to access the finance system at our organization, we probably have very limited access. Very simple use case. There's also the concept of machine identity, which is not new. This has been around forever. And so you might give an API key to another service, another piece of software. And then that service can go authenticate in its way, very similar way, to another service like a finance system. and might be able to pull information. And once it authenticates, it itself has all kinds of permissions that it accesses. But this is different.

16:21And so the number of use cases where an AI agent can and should access something as itself and purely as itself, what we call a service account, are pretty limited. You know, it's the Google search API. It's a weather API. It's typically things that aren't terribly secure. You know, you don't worry a ton about them getting broken into. But the vast majority of where end users get value from an agent require the agent to go access something that is tied to the user. You want to access your email. It's tied to you. You want to access, you know, financial information at your organization. it's going to be tied to who's reading it.

17:08The CEO is going to have very different level of access than the intern. And so when you do that, the naive approach is to have the agent simply get the user credentials and just authenticate as the end user to those services. That's wildly insecure. And more importantly, it's unsafe from an AI perspective. So let's use an email example. Your inbox, you can do whatever you want. you can delete all the emails if you wanted to. Do you want to give the agent the ability to delete email? No, because it might hallucinate and accidentally delete your email. There's a very famous coding agent called Cursor that I use actively.

17:51It once hallucinated and tried to delete my root directory. Oh, wow. Right. For those people who aren't engineers, you know, deleting your root directory is the equivalent of deleting your entire computer. Now, thankfully, you know, my Mac operating system has a bunch of authorization already built in. And so even if I had said yes, it wouldn't have been able to do it. Thank God. But you don't want to, even though you, Craig, have full access to your email, you don't want to give the agent that same level of access. And so the right approach is to give the agent an intersection of permissions, an intersection of what you, Craig, have access to, but also an intersection of what the developer and Craig had granted the agent access to.

18:40And thankfully, there's protocols for that. There's a very popular protocol called OAuth, but it's very difficult for developers to implement. And it's, until recently, almost impossible for them to implement in the context of agents. And that's what we've sold. I see. And in that case, the, well, let's go back to the API issue because LLMs don't handle APIs very well. What happens when you need to hit a service or? Yeah, that's a great question. So I think this is the thing that a lot of people are learning. I mean, the AI industry is very new, right? You know, ChatTPT and GPT 3.5, you know, had their second year anniversary this last November, right?

19:35And so it's mind boggling to really like internalize how fast this all happened. And so people are, as people are trying to build agents that can take actions, that can do things and not just generate content, people are learning the hard way that APIs, don't work with large language model very well. You can't just hand a large language model an API definition and have it go figure it out. It might, but the error rates, hallucinations, won't work with enough confidence to make it valuable. And so what people do instead, the right way to give an agent access to other services is to build what's called a tool.

20:24Terrible name, SEO is terribly, but it's the terminology everyone in AI uses. And a tool is like an action. It's an intent-based action that might wrap one, zero, one, or multiple APIs to perform that action. So I'll give you a really simple example. Reply to an email. if you're in an agent that's email based and you say, Hey, reply to my email from Craig, there is no API endpoint on the Gmail API to respond to an email. There is lookup email and there is send email. And so a reply to Craig, you know, when I say reply to Craig's email, the tool it needs is going to be reply to email tool. That tool is going to do multiple things.

21:19The first thing it's going to do is it's going to go find the email I'm referencing that I want it to reply to. So that's already one API call. Once it retrieves that email, it then has to unpack the email structure. It's called MIME. You have to unpack this email structure. It's actually a fairly complex piece of software. There's attachments, there's HTML versions and full text versions, et cetera. Unpacks all of that. then has to go generate the response, insert the response, then repackage it all up again in that email structure called MIME, and then hit the send email endpoint. Altogether, it's about 100 or 200 lines of code, depending on how diligent you're being.

22:04And that's not even including all the error handling and checks to make sure it's all working and safe. That's a lot of work. that's a distinction between an API and a tool. Now you go something much more complicated, like, hey, DoorDash, order me a pizza. The order me food tool for DoorDash would be like 10 API calls. So there's just a different level of complexity, but most importantly, it's presented to the large language model very differently. If you've ever looked at an API, it looks like gibberish. It looks like a URL. out. AI systems don't understand that. So a large language model, when a tool is presented to it, it's presented in natural language English, is a tool for replying to an email by a sender.

22:56It takes in the name of the sender, the subject line, the body of the new response, and it's all explained in English. And that's what makes a large language model really good at selecting the right tool and then selecting the right input arguments for it to go execute okay and and the um this all takes place internally i mean is that the the chief uh i mean because there are a lot of platforms out there now to build agents uh differentiator with arcade that it handles this authorization in the background so you don't have to yeah so when we yeah so the the way to describe where we fit so we are not a large language model company yeah they're very good large-time model companies out there anthropic and open ai being the most notable and so the large language models the one that selects the tools we we've designed Arcade to be the system that receives the selection and then executes the tool.

24:06And so what we're really good at, at a really high level, is helping developers get that tool execution to be so good that they can get their agent from a demo to production. And so anything that requires that, we've put a lot of IP into. The very biggest problem everybody runs into is that end user authentication and authorization. And so that is the backbone of our feature set. But there are other things that are also critical, but they're a bit in the weeds, like parallel tool calling, retrieval tool logic. So that demo I gave you, it works remarkably consistently, which in AI, that is the performance metric in AI.

24:54It's not how fast it is, it's how often is the error out. That demo I gave you, I've given that demo like a hundred times. I don't think I've ever seen it error out. And it's not because the large language model is not hallucinating. It is hallucinating just like every other system out there. But part of our IP is that we can catch those errors and then have the large language model retry with additional context. And when you do that, the consistency rates go through the roof. And so all of that nitty gritty detail that helps a developer take an agent from demo to production, that's where we focus our efforts.

25:36Yeah. And this is a developer's platform. Why not make it a B2C product for general users? Yeah, it's a great question. um you know to know thyself right um my team and i we're just passionate about infrastructure um yes uh you know i always give a house analogy um i'm trying to give you the best pipes in the industry i want you to build a beautiful house right um i i just happen to be really passionate about pipes.

26:16And yeah, but that doesn't explain why not let, you know, I think from a consumer side, oh, why not let consumers use the service to build their own agents? Yeah. Oh, okay. That's different. Yeah. So the answer I was giving you, why didn't we just build an application? like why didn't we compete with Jack's right right oh I understand that so there we want to be that we want to you know be picks and shovels why do we not expose our picks and shovels to the average consumer um there you know people call this like you know no code or low code agent builders um that's not a space one we don't know that space particularly well so there is a bit of like a domain mismatch.

27:04But second, if I'm being perfect honest, I'm just not a big believer in those because the building an agent is extraordinarily complicated. And we really want to have a big impact on what everybody in the world, what the world of AI and agents looks like and and i think that the biggest impact we could have would be giving it to the developers because they're the ones who are think are going to we believe they're the ones who are going to build the the most powerful agents of tomorrow and so that's what that's where we focused yeah and so on the platform once you built an agent uh how is it deployed or how is it moved into a production product?

27:54Yeah. So the kind of the lifecycle of an agent developer is they start to build something and they get working in demo and everybody gets wowed. You know, I'm sure, you know, you've seen a lot of agents wow you in a demo and then, you know, they never quite make it to production. It's actually one of the biggest problems right now in agents is I think like the The stat I heard somewhere is less than 30 % of agents ever make their way to production because it's so hard. The demos are easy. And so what happens next is they show it to everybody, they get approval to go really invest in it. And then they start to build and start running into a bunch of problems.

28:32The biggest problem they run into is the first one they run into is authentication and authorization because they simply can't deliver a feature. Let's say they get past that. The next problem they run into is consistency and accuracy, right? that notoriously agents and AI hallucinate. And if those hallucination rates are below 80%, it's not going to be a valuable agent. People aren't going to use it. And so a lot of things fail over that. Now, if you clear that hurdle, the next thing you're going to run into is a cost issue. OpenAI and Anthropic, they cost money. Every time you call them, you're paying for each token and that stuff can really add up at scale.

29:14So a lot of teams we talked to who don't go to production yet, somebody did math and realized, oh, if we implement this, it's going to bankrupt us. And so then there's some re-engineering of how do we implement this with fewer large language models. Now, if you're using us, we unblock the first two. The third one is, unfortunately, out of our hands. And so to deploy us, it's very easy. The developer gets an API key from us, drops it into their software. And when they're implementing their logic, they're just calling out to us for what's called the tool calls. And then we handle everything else behind the scenes.

29:50So we've offloaded a lot of the logic so that they used to drop an API key. And that's typically it for them in most use cases. If a developer wants to build their own integration and they want to author their own tool, we have a whole SDK for them to do that and leverage all the same value and benefits. And how they deploy that custom tool really depends on what they want to do. They can deploy it in their own infrastructure, their own servers, or they can deploy it to us and then we'll put it on a cloud server and run it for them. But that's really all that's needed. Yeah. And then does it create

30:35a web app for the tool that then the developer can disseminate within their organization or how so i'm a developer at a large organization i i create this agent to help people manage their emails read and send and and all of that uh that the the actual um logic sits in in the cloud with uh arcade i hit it with an api but what's the interface that that's a great question so this is an important distinction so the agent itself let's call it let's call it the app because an agent is is is just an application the difference is that it's leveraging large language models for the workflows um so instead of a developer somewhere saying okay step one step two step three in this order the there's a large language model in there and the large language model is deciding what the next step in the workflows are you know whether it's a consumer facing you know a writing app or a CRM or a finance app.

31:51That's a distinction. That app, we don't touch. That app's deployed and written in whatever way the developers want. So maybe it's deploying to AWS or to Microsoft Azure. But when it goes to take an action that talks to another service, like an integration, at that point, that app calls out to us. Right. And for most of our customers, we are used as a cloud service ourselves. But for our larger customers, they can deploy us in their own private, you know, private clouds, their own, you know, private Azure and AWS environments. Yeah. And this platform, you say you're not a model company. then you can choose the model based on whatever, you know, whether it's price or latency or whatever.