Your AI demo is a lie (and how to make it real) | Arcade’s Alex Salazar

30:33The second one ends up being security and safety. And it's not even really just like a, it's not even really just like a, oh, you know, tinfoil hat, you know, CISO, large enterprise thing. Why can't JTPT send an email? It's 2025. They still can't send an email. It's a security thing, safety thing. And so, yeah, in a demo, you can give it like a Google app key, or you can hard code your credentials into some other service. There's all these hacks you can do in demo to make it work but that's never what you would do in production it might work for one single user but it would never work for a multi-user environment and so a lot of agents are blocked i mean why why isn't there a personal assistant agent today we've been talking about personal assistant agents for years and there still isn't one a lot of it is because how is the agent going to go talk to your gmail calendar the airlines how's it going to purchase something for you.

31:28Those problems are largely solved by what we built, but many of them are still aren't solved yet either. And the last one ends up being the last two are total costs. When you're doing a demo, you're not looking at the bill. You spent 50 bucks, no big deal. It was a demo. 50 bucks is nothing. But then when you extrapolate to like, you know, a hundred thousand users, suddenly like that's, you know, every day, the finance of the agent doesn't make any sense anymore. You'll lose money. You'll be negative margin. And then latency. People really underestimate the impact of latency. As you try and get the large lines of the more accurate and smarter and do things better, you start stuffing in more things in the context of my job.

32:05Well, that's going to drive the cost up, but it also drives up latency. And the more you do sequential thinking or any other kind of complex workflows, things really start to expand. And the smarter models are slower. And so all of a sudden, you have this really cool operation. Maybe it is really accurate. But it takes a minute or it takes, you know, 30 seconds or even 10 seconds, depending on what it is. Like, depending on the use case, the user may or may not find that viable. And so just, you know, so those are things we typically see chill projects. Yeah. And these constraints you've kind of defined are, they're interesting because they make for a very unique space where if you have like those three or four strong constraints around like why we don't have AI systems today, why they don't send emails, all of these things.

32:50like$50 for a demo versus, you know, millions of dollars for all your daily active users. Like when you have all of those kinds of constraints, you kind of get tempted to sacrifice one. And so that's where you get this demo phenomenon. You sacrifice security for the sake of putting on the veneer for the other kinds of functionality, the show like, look what this cool demo does, or look what we're building. But then when you actually go to build it, it becomes something totally different. And that's also because obviously the space is moving so quickly. It was a few months ago on this podcast, we had Kabir Bhatti from AWS, and they built an AI video generator tool.

33:25And he talked about that at length, about what they started with and then what they built over the course of the last year, and then what they ultimately brought to the market for mom-and-pop business owners. It probably had three different forms, ultimately. So that's where you get this rapid evolution. and a lot of times when we talk about this on Dev Interrupted, it seems like a team's expectation problem more than it is like, you know, all of those constraints, they're certainly solvable if you have the right expectations and you work about it in the right way. So, you know, what are some things that you see for teams that get it right, get it wrong about how they move past like the demo gloss and actually start solving?

34:08Yeah, so I'll tell you our story, which is ultimately how we pivoted into what is now the product, which is a tool execution and tool authorization system. So we started off as a SESSRE agent. And we were getting compounding error rates, which meant like every, you know, if you think of like a diagnostic flow, like why is service A slow? You're going to set the databases, you're going to set the servers, you're going to check to see if somebody committed code last night. There's all these checks. But like, it's almost an infinite space of things they might check. And the diagnostic like workflows, is the chains themselves can get really long.

34:43But every time you ask a large language model any question, you're rolling the dice. The power of a large language model is also its weakness, is that it's approximating this. And so, because it's approximation, it's non-deterministic. And so, it can get things wrong. It can hallucinate, you have errors. Every time you call it, you're rolling the dice, and if you roll a one, it fails. Well, if you've got really complex flows where you're calling it multiple times, you're rolling the dice more. And if you roll a one once, the whole chain fails. Right. And so we're running into these problems. That was problem one.

35:20They're called compounding error rates. And the second thing we were running into was, okay, well, we've given this agent super user access to Datadog and the servers and the databases. And it works. Right. Because it's great for a demo. But now, as we're going to go try and make this thing actually sellable, it's got to not do that. We have to really scope down resources and leverage existing permissing systems. And there was no way to do that. And so we ultimately had to invent all of this stuff for ourselves to make it work. And when we got it working and we started to show people a working demo, people were blown away.

35:54People who knew AI were blown away. They couldn't believe that we had really consistent data plots or that we were authenticating as on behalf of me into a service and properly scope permissions from within side of the agent. And then we realized, oh, this is a bigger product than the agent. So we ultimately pivoted and started showing and selling people that, which is what our key is now. yeah so my answer of like if i were building an agent today what would i do to get around all these things and to to really increase the odds of me succeeding obviously the issue is arcade but let's put that to the side so one i would this general software engineering best practices like you should be using stuff off the shelf to the degree that you can because there's so much to figure out and and build and learn that if you try and custom build everything from scratch you're going to run into a really big problem.

36:49Yeah. And so the easiest thing to start with is go grab an agent, go grab an agent framework and avoid trying to build your own. And yes, the learning curves are really steep, but they're really steep for a reason because when you custom build it yourself, you can fool yourself into thinking that you've got it because of the demo phenomenon. But all the frameworks are really complicated because they're building for production and they're not necessarily building for demo. And so by volume, we largely see LandGraph. And then behind it, we see OpenAI's Agent SDK. And there's a bunch of other ones.

37:23There's Pidantic, there's Mastra, there's ScrewAI. But pick one that you think fits your requirements in production and go and run. As you're picking those, I strongly recommend really thinking through what production is going to look like, not the demo. Optimize for the final state, not the demo because of this phenomenon, which I see a lot of people get stuck on. they'll optimize oh well look I deployed this one because it was so easy but they're optimistic for the demo and so they'll optimize for prod I think the other thing I would say is you've got to start with evals like you've got to build a muscle of evaluations right out of the gate because there's really no way you're going to get a production grade agent that's consistent enough if you're not really good with evaluations and there's a lot of products out there that can help you Planksmith there's Brain Trust we have a framework very specific for tool use and tool calling so the tools are out there you just need to set them up and use them and for people who are not native to aiml like traditional web developers this is very alien but it's really critical because we're all used to deterministic tests you know right integration tests unit tests and like how do you do a test on a function where there's an infinite number of correct answers an infinite number of incorrect correct answers and an infinite number of in the middle gray.

38:43Right. And you can't build those. Like there is no CI system that can support that. You need that. That's what evals are. And there's a different animal. The last thing I'll say on this topic, because I talk a lot about it to people is de-scope the living daylights out of your project. Even teams that have picked the right stacks and have built evals and all this stuff. But like the other thing that like where they hang themselves is they just, they try and do too many different things. and this stuff is complicated enough and new enough where you just increase your up significantly if you're trying to automate like one particular workflow first get the win and go and a bonus one is picking the project that you're going to go do like let's call it project selection like what is it you're going to work on what is it you're going to what's that what's that workflow you're going to automate man like that's half the battle and i would urge people to focus on places where there are modern APIs already available because if they don't already have modern APIs to access, the complexity goes through the roof.

39:50Like you try to build an agent that's browser-based, you can make it work. There are really good products out there like browser use and browser-based and a few others, but... Introducing a lot more complexity. A lot more complexity. And so if you're going to build an agent and you have the privilege of picking where you start, I would try and focus where there are modern APIs. Yeah. Okay. I love how you framed it about the eval situation, being so alien from writing normal tests. I think that's something that rings really true. That's something that I've had to adjust to as an engineer, moving from writing deterministic code to evaluating how my prompts or my workflows are doing.

40:27The idea of setting up the side-by-side benchmark is definitely new to a lot of folks, but it's a really critical part of building those tools. And it's just as important as the demo. I find whenever I work on those is starting from the eval, just like how you would start from tests or start from that more deterministic way is actually going to help frame what you're looking for a lot better. You know what success is earlier and you know failure immediately. I like too how you talk about your product moving because of opportunities, because of problems, because of constraints in the space, moving from like an SRE kind of agent, which an SRE is already solving a very complex, highly secure, very specific, like architectural infrastructural, you know, problem for teams, right?

41:11So then when you shift into like, we're just going to focus on solving the security, it's like, sounds like you taking your own advice, like what you just said about descoping. Because that security element that Arcade now is, was probably, or, you know, initially a very critical component of SRE Agent that was of the days of yore. And so you descoped yourself and you found what you're building now at Arcade. And just to bridge it into MCP a bit, because our listeners have, we talked about this a bit. We've had some guests even talk recently about MCP. We had CTO, co-founder of a first-of-its-kind MCP agency, who's like, they're out there making MCP tools for companies that are using these with agents right now.

41:50So there's a lot of interesting kind of conversations in this. We always are kind of going around security a bit. So I'm kind of curious to know, you know, we've talked a bit about the growing pains and about building with the tools. But, you know, what's your advice for folks that are experimenting with tool calls as part of their workflows that they're building with their teams, especially like in their engineering world that they live in? Yeah. So right now, tool use, tool calling is all the rage, right? It is the biggest bottleneck and biggest opportunity. in agents, which is crazy because we were deep in it before in school.

42:31Oh, so you're saying I was at those problem spaces. We were the OG. We all showed up. We were the OG. But so it's been really awesome to see the whole community kind of come to the same realizations that we had come to when we made the pivot and started to build the for-profit. That part's awesome. So let me contextualize it on why it's so important. Like why is it so exciting for everybody? But if you go back to what we just talked about, compounding error rates and consistency and accuracy, and if a model asks too many questions, you're going to hallucinate and have an error and all this eval stuff.

43:05The big insight we had in our agent, the reason we got it working was we ultimately, agents are non-deterministic, right? Lars Lang with Lars are non-deterministic. That is their power. That's also the risk. And so the big insight that we came to, like led us to Arcade was we dialed up determinism. There's, you know, prior, at least for us, prior to this big insight, we were all in on the large language model, figuring everything out. And that wasn't working because of error rates. And so when we, when we just had this big epiphany one day and we just started to like dial determinism back up, the analogy I give people is instead of letting a large language model just figure stuff out and go through its planning phases.

43:47and then, you know, in multiple series of planning phases, we were like, we're going to give it a discrete set of multiple choice questions. We're going to do, we're going to end at the TI-85 calculator and we're going to get all the questions. And it can only pick the butts during it. And when we, when we constrain the large language model, like to that degree, that's when things started working really well. And so the core insight that led us to Arcade, which is also the core insight that everyone's now having around tool use and MCD and all these different technologies is that if you dial up determinism and give the large language model these buttons and these buttons represent things that can connect to the outside world, you can build all the things that we all want to build but couldn't before.

44:36I see. So if I understand what you're saying correctly, it's instead of being like you are a free agent that can consider these tools as part of your thought and knowledge working process. Instead, it's like, okay, sit down. Here's your multiple choice exam. Each of these questions, there's five options, and you need to pick the best one from each. And that's actually a really fascinating kind of comparison to even how humans who are non-deterministic are evaluated. You could pick or use any kind of tool, but you want to know who's the best at it. You sit them all down in an exam, and you give them the same exact questions, and you see who picks them all correctly.

45:14So that's actually a really interesting way of kind of hijacking the same system that works for us as non-deterministic things. I mean, it's how all of us do math, right? You know, if I gave you, if I asked you to multiply two five-digit numbers together, like you'd probably get it wrong without a calculator or without a lot of time with pencil and paper, right? But if you're me, yeah. But even pencil and paper are tools. So if I just literally had you like do it in your head, you would probably get it wrong. So will a large language model. But if I give you a calculator, you're going to ace the exam.

45:49So will a large language model. And the impact's pretty dramatic. Like if you, if you give, you know, GPT 3.5 turbo, the oldest, worst tool calling model out there, and you hand it up like math tools, it will outperform a thinking model, like a brand new state of the art thinking model. and it'll do it much faster and much cheaper. And so the impacts are huge. And those are really simple operations. Now you think of something complicated where authentication and authorization is involved. Like, hey, go read my email. Go email Bob letting him know I'm five minutes late. Well, now what the large language we're going to have to figure out is actually almost impossible.

46:33Take hallucination rates out of complex workflows out of it. How's it going to handle an authorization flow, like an OAuth 2 flow, if you can't trust a model to hold keys, how are you going to, a security flow without you holding a secret is really difficult. We didn't know how to do it. And so, so how do you allow a large language model, an agent wrapping a large language model to go execute secure operations like reading your email? I mean, most things that matter require some degree of security. How are you going to do that? And the answer ended up being, you're going to do it inside the tool call, inside the button.

47:10okay so what does that mean for folks that are building now do they need to look at how they're building things differently so when you know let's let's pretend we're building a personal assistant agent and you say hey go check my email well the agent asking the agent to go figure out ola asking the agent to go infer like all the things that we need to go into that doesn't make any sense. Don't do that. That should be encapsulated in a tool. Whether it's MCP or RCA or the protocols don't matter as much. It's encapsulated in a tool. And that tool can be as deterministic as you want it to be. In many cases, they're completely deterministic.

47:50And so checking your email doesn't need a model to do anything. There are Google APIs and there's all kinds of stuff. And so you would, so when the model says, oh, I want to check email button, And after that, your function definition and your deterministic code is going to go through a standard OAuth flow. It's going to do all this different stuff. And it's going to go hit the right endpoints. It might be one endpoint. It might be multiple endpoints to go do the thing that you asked it to do. And now, here's the rub. This is where a lot of developers are getting stuck right now. They think, oh, well, Alex, what you're describing is just APIs.

48:27I'm just going to take the APIs that exist and put some natural language around them and I'm done, right? Unfortunately, that's not the way it works. So I'll give you a really good example. Let's say you say you want to reply to an email. Well, you're going to give the agent a reply to email button. So it's going to reply to email, right? Well, like, there is no reply to email, but there is no reply to email endpoint in Gmail. There's actually a series of operations you have to go do. A lot of it is in your own code. So it all has to still exist as a set block of API calls and you still need that orchestration, you know?

49:00Yeah, and they don't, and they're completely orthogonal to RESTful APIs. MCP and tools are ultimately APIs too. They're just different user, right? Like all the APIs that we know and love in the last 25 years, they're all, the user is really the developer who is wiring them into the code and they know exactly where they're going to use them and they know exactly how they're going to use them and it works every single time the same way. That's not how agents work. And so REST APIs typically are resource-based CRUD operations, create, read, update, delete. So almost every API you can think of is typically defined as a resource.

49:35And then most of the operations are create, read, update, and delete. An agent is different. They think in intention-based workflows. I'm going to reply to an email. I'm going to send an email. I'm going to go check to see if so-and-so replied. And those are orthogonal. And they might include zero API calls. They might include 100 API goals. They're just workflows versus resource-based front operations. And worse, RESTful APIs, because as we typically know them today, they're really what I refer to as inside out. Like a Google Drive API is going to describe Google Drive. It's going to explain Google Drive in a lot of detail.

50:14It has APIs to manipulate everything in Google Drive. your agent, let's say we're having a sales agent and it has an operation to brief you before a sales call. It's going to go check the CRM for communication. It's going to check Drive for brochures. It's going to go check your email for something else. It doesn't give a shit about Drive. What it cares about is getting the brochures. And so if the button is get brochures, that's all it cares about. All it cares about. But if the button instead is 37 different Google Drive operations, you're asking it to hallucinate. Because now you're asking it to not only hit the right button, but to hit the right sequence of buttons and figure out at inference in real time, like what the right sequence of buttons is to get to the right brochure.

51:01It can do it, but it's going to be slower, it's going to be more expensive, and you increase the odds that it's going to get it wrong when you don't need to. And so part of the problem, part of the paradigm shift for a lot of developers is it doesn't matter who gives me an MTP server. Like Google might tomorrow have the most beautiful SPS server that ever existed for Google Drive. You're still not the customer to build your own because you got to ground it in the domain and the use case of your agent because your agent's intention-based workloads. You're still going to need something between you and that or you're going to have to modify that to fit what you're looking for.

51:37Yeah. And I think right now, you know, that's something that people are only now starting to realize as they're starting to build tools. Because, yeah, it's super early. I mean, tool MCPs aren't going to have enough. But outside of very general purpose agents, which is the minority of the population of agents, you have to custom build your attention-based workflows. I want to shift the conversation a bit to learning a bit more about how your team is kind of like orienting itself around these problems and tackling these problems. Because like you've described in our conversation, it's very much a mindset shift in how you approach the problem, how you solve the problem with the constraints that are around you, but then also then how you test it.

52:16And ultimately, you build things by taking parts of other things and combining them in a secure way that ultimately specializes in on that intent. You talked about LLMs, they operate in intent versus APIs, they operate in actions or endpoints. right so when you are forming a team and you've and you brought people together to build arcade what does it look like on the inside of an ai native company and what are some things that you do there that you think that you do differently from maybe like more other companies yeah well i mean it's a great question so the first thing is who you hire it all starts with hiring and team makeup now because our product is ultimately like a service bus yeah we're executing just the get brochures button is is on the platform so we're like a service bus but a major feature in that service bus is that we're going to handle authentication and authorization on behalf of the agent right off probably their biggest feature so because we do that for our company in particular it's critical that we have the intersection of three really important skill sets for us you have be agent native and that's really hard to find there aren't a lot of people that don't build agents let alone they don't work at open ai and dropic and so my co-founder sam is one of those sam has implemented more than 100 agents or devras and so super lucky that we started this company together so somebody needs to be asian native and really understand how these things are go and go to prod have to really and this is this is the hard part not demo native but prod native which is a different thing.

53:56Yeah, that's definitely one of the big takeaways, I think, from this is moving the conversation from hype and from demo into prod. Yeah. But then once you get past agent nativeness, then it's really about what is it you're actually doing? And you got to be expert at that too. And so for us, as an example, we have to be expert at distributed systems because we're running all of these services, all these tools, and we have to be expert at all. And so, you know, half the team is out of Opta. we had to invent new ways of doing off to make all this stuff work right i think that's an important piece because you can't just like you can't just pattern match you can't just say oh well it worked like this last time we're gonna do the same over like it will rhyme but it won't be the same and so that intersection of expertise is what leads you to the right innovative solutions versus the old players just adding the word agents to something and then saying well now it's agent natives.

54:52Yeah, yeah. I mean, I think a lot of teams and folks are seeing that as well. But I want to double click on that first one you said being AI native or being like an agent native kind of worker. I think that's really interesting. So what do you think about that? What does that mean to you? I mean, that means you've put agents in production before. Okay, so I mean, like you've just made agents, you've taken the production, not necessarily that you are operating in an agentic way. If we're talking about an engine, like an engineering team. Yeah, like the right kind of like the mindsets or the skill shifts that you're seeing in your own team of like, oh, wow, like they stand out.

55:25They're a high performer and they're a high performer because of this. And a lot of times when I have these conversations with leaders like yourself, when we kind of peel that away a bit, it's because like, oh, they built this AI agent that does this thing for them or, oh, they captured their workflow in this way, which is one of those victories. So you have like. Yeah, yeah, for sure. I mean, like, I think right now everybody can, I'm going to sound like a hot tape, but people can fool themselves into thinking that they are, at least in an engineering capacity, that they're agent native because they use Kershaw or they use Windsor.

55:57That's an important skill set. Any engineer who's not deep in that stuff is going to be out of a job relatively soon. Because it's such an enormous lift. But it's not the same thing as having built an agent that went to production. Prod, not demo. Prod, not demo. Taking an agent to prod is a very different experience than having a conversation with Cloud Code. And that is what sets people apart right now in the engineering teams that are trying to build agents. Okay, that makes sense. So I'd be interested in knowing how that engineering group grows over time, the people who have built agents and taken them to prod right now.

56:36That's a small pool of people. It is a small pool of people, but you don't need a huge group because you still need everything else, right? And so, but what's great, if you have a really good engineering culture and everybody's in the same office, like us, I don't know how you do this in distributed teams. So I'll pass that to somebody else. But in a local team, the cross-pollination happens really fast. Yeah, it does. The person who is the agent expert is sitting next to the person who is the auth expert is sitting next to the person who is distributed systems expert. And they all start cross-pollinating.

57:06And all of a sudden, the distributed systems person is now starting to become a little bit more agent native every single day. and then all of a sudden they have an epiphany one day that even the agent builder didn't have. And vice versa. Like the auth person is cross-pollinating with the agent builder. And the operator's like, oh, well, maybe we did this way or that way. And so that's where the magic happens. For sure. We've talked about this with some guests before too. We had JJ Tang, CEO of Rootly, and we talked quite a bit about their knowledge sharing, working in the open, sharing team practices.

57:34This cross-pollinating effect that you're talking about is really critical for being a customer-oriented engineer because all of them have different aspects of what the customer is looking for or what the product needs to be. So the more you put them together, the better. I will say, in a distributed team myself, I will speak in defense of the distributed teams that we're also quite good at the cross-pollination. It's just that it happens a little differently. But I totally agree with you in terms of that being really important. And I know we're kind of starting to wrap up our conversation and come near the end of some of the stuff we've talked about.

58:07But you've captured a lot of really interesting thoughts about how teams are building tools, but also how they're orienting themselves around solving those problems. You know, is there any kind of lasting advice or that you'd like to give for if there may be that there may be there more that traditional engineering team who doesn't have that agent native engineer somewhere within them? You know, you can't cross pollinate from nothing. So how do you really start that culture? And what opportunities do you see for that kind of team? Yeah, I think the answer is both easy and extremely difficult. The answer is you have to start building with agents.

58:45And maybe the team's not ready to start building production-grade agents. Maybe it starts with hackathons. Maybe it starts with the leadership of the organization carving off time to letting the team play and learn. This stuff is really difficult. This stuff is, you're not going to pick it up in a day. it's going to take you months of tinkering and playing and building to run up against enough of the walls and learn because it's ultimately a new paradigm probably most listeners here don't remember the pre you know internet dates but a web developer was a alien creature to the traditional client server developer yeah right was an alien creature to like you know the mainframe developers, right?

59:30And so, you know, we take for granted how native most of us were to web, but it was a huge paradigm shift. And similarly right now, it's a huge paradigm shift. It's a whole new set of technologies, a whole new set of vendors, a whole new ecosystem, a whole new patterns. Testing's different. Workflows are different. Like, so the only way to learn it is to start doing it. And the books, you know, there might be books out there. They're great, but nothing, nothing is a replacement for hands-on experience. Yeah. You got to get out there. If you haven't built the demo, you got to build the demo. If you built the demo, maybe it's time to challenge yourself and think about how would that demo go to prod.

1:00:05I think that's a really powerful takeaway from this conversation. And when you build that demo, you should start by using Arcade to make it. So that when it goes to prod, it's easy. Yes. So, Alec, where can folks go to learn a little more about Arcade and how you're solving this problem that we've talked about? Yeah, thank you. So, yes, our website is arcade.dev. And so they should go there. We also have a YouTube channel where we have a ton of examples and walkthroughs on how to build pretty breakthrough agents that can go interact with the outside world. Things like Google and databases and your own APIs.

1:00:38We show you how to custom build your own tools in addition to using our out-of-the-box tools. We've done everything we can to make it as user-friendly for traditional developers as we possibly can. And I think we've done that. So I hope you come to our Q2 tips. Yeah, definitely. We'll put the links in the show notes so our listeners can go check it out, learn a little bit more about the problem space that you're working in. I think it's cool that y 'all also have examples up on YouTube. I'm definitely going to go check out the tool. Maybe there's a chance for us to collaborate on some of that stuff.

1:01:07I really appreciate you sitting down with us and walking us through some of the things that you've been building that's been top of mind. It's been really interesting for us. And for those that have been listening, if you've been listening to the conversation, you are one of those engineers who happens to have taken an AI agent to prod. We would love to hear about it. what you thought about today's conversation with Alex, but also just about your own experiences in doing that. So please drop a comment somewhere, anywhere that you're listening to this or find us on LinkedIn. We'd love to continue that conversation.

1:01:35And if you're not, you're still experimenting. Well, we want to hear from you too because we're all on the journey together. And the only way that we're going to get there safely and securely is by building it together. So thanks for joining us on Dev Interrupted and we'll see you next time.

1:01:54Be safe. Bye. Bye. Bye.