More Agents Than Employees: How Zapier Disrupted Itself Before AI Could

16:31Matt Paige:Yeah, that's a great point. It is funny. There is this element of a model FOMO where it's, yeah, maybe the lower tier model would be okay, but maybe I may get that extra bit of something from use. I'm guilty of that all the time. But it comes down to, especially when you're doing things in a more structured way, the evals and evaluations. So how do you think about that? And is there a method for actually creating evals to determine, okay, this model is perfectly fine for XYZ task? Yeah.

17:04Wade Foster:What I've learned about what makes a good eval is a couple of things. I think, one, you really do want to have your eval be tough. it should be really hard for these models to achieve these tasks. And the best evals are typically evals that are actually kind of easy for humans to do, but really hard for these models to do. Because now you're testing something really interesting where you're like, huh, there is a true capability gap there. So I think that's one that's really important. The second thing I think that's really important is it's really valuable to have the test data entirely private.

17:39Wade Foster:this really this prevents basically the models ending up like benchmark maxing on these by accident i think one of the things we've often seen is that as these benchmarks get leaked all of that stuff gets leaked into the training data and then all of a sudden the models like hyper focus on fixing it and so the model gets good at the benchmark but it doesn't necessarily get better at just the things you might actually be exposed to in real life because ultimately a benchmark is still just a proxy for the type of work you do in real life. It's not actually real life. And I think this is what can happen a lot of the times when folks get frustrated with AI in the real world.

18:17Wade Foster:You'll see these benchmarks where it's 99 % or whatever. It's, ah, this is a crazy model. And then you go use it yourself on a task and you're like, dang, it kind of sucked at that. How is it so good at this? But it kind of stinks at this other thing. And it's because the benchmark is still just a small representation of all like the knowledge that sort of humans might ask the model to go achieve.

18:39Matt Paige:Yeah, it totally makes a lot of sense. I want to get into the topic of agents, right? You obviously are leveraging agents within the business within Zapier, but the term gets thrown around like crazy. How do you define what an agent is for our audience? Because I'm sure there's a lot of folks that have differing opinions or have no clue how to actually define it. Sure.

19:03Wade Foster:I think colloquially agents have just become basically like things that automatically do work. But I do think it's important for folks to understand the differences between a deterministic workflow and a purely agentic system. A deterministic workflow is something that's been around for ages, pre-AI. It's a program. It's Zapier in some ways is build agents before agents exist, where it's like, hey, a trigger happens over here and we're going to automate adding this customer to a CRM and then we're going to alert somebody inside of Slack and all that sort of stuff. These are just the workflows that sort of exist inside of a company.

19:39Wade Foster:And these things have only grown in popularity in the age of AI. Now, an agent is slightly different. The way an agent actually performs is you give it a goal. You say, hey, I want you to go complete this task. And then the agent gets to go decide how it wants to go execute on that thing. So based on its training data, based on how the model works, it will say, maybe I'll do this task and then I'll do this next. And then with this combination of data, I can go complete my goal. And every time it gets a similar task, it may not execute it the same way every time. It might choose to follow a different path and it still might get to the same outcome.

20:12Wade Foster:It might get to a slightly different outcome, but it behaves more like a human does where you give it a task and it just goes about its day. It might mostly do it the same way, but it doesn't necessarily follow strict reels. Now there's pros and cons to both of these models. A deterministic workflow, what you get is reliability, consistency, and cost advantages. It does the exact task the same way every single time. So that's the real benefit that you get from these deterministic workflows. The failure case is it's rigid. What happens if something goes wrong? What happens if something unexpected comes up?

20:45Wade Foster:What if you have to work with like messy sets of data, like big unstructured tasks or images or things like that? All of a sudden it gets really, it gets a lot harder to do the task inside of these deterministic workflows. So you run into those problems. Agents are almost the exact opposite because they have agency to go complete the task. Reliability isn't quite so good. Cost is a lot higher. As a result, it can be a little bit slower to complete these things. However, they can be a lot better at handling edge cases because an edge case pops up and it starts to reason through it. What should I do about this?

21:18Wade Foster:How could I go solve this one? And it might find its way through the problem. It can handle a lot more randomness inside the situation. And so that is like the blurring of these two systems. Now, oftentimes, the magic that I find is inside of a business setting, you often don't have just purely deterministic setups and purely agentic ones. So the magic is like when you can actually blend these worlds together where you might say the first couple steps of these always just work the same way every single time. And so we're going to just have that be purely deterministic. Right here, we have one of these ambiguous situations that right now a human is making all these judgment calls on because we can't really do anything with it.

21:55Wade Foster:Okay, let's put an agent right in the middle of that. But then once it spits out an output, we're going to feed it back into a deterministic system to complete the loop. And I think the most sophisticated customers we see, the folks on the frontier, are often getting really smart about how they do these things. They're doing the stuff like Brian and Coinbase is doing, where they're mixing models or thinking about what is a workflow versus what is an agent. They're really starting to optimize these setups. Most people aren't there yet, but I do think that is the world we will find ourselves in as more and more tokens get spent in these organizations.

22:24Yeah.

22:24Matt Paige:And that's the beautiful thing you mentioned is the marrying of both, because to your point, when it was purely deterministic, you had to figure out every edge case or you had failure patterns all over the place, but now with probabilistic, it's a whole new world. How do you think of iterating on a workflow and optimizing it over time? Because I feel like there is that element of being able to improve something because back to the deterministic nature is it did the thing or it didn't. But now that you have AI in there, there's this using the same term again, probabilistic nature of the workflow where it can improve over time.

23:00Matt Paige:Any thoughts on like people that are actually doing these things in their business? How do you like methodically think about starting small and improving a workflow that you're trying to automate over time? Yeah.

23:14Wade Foster:Yeah. I think what the best folks are doing is they have these, they're building their own mini evals for these setups. And as more and more scenarios come through, anytime the agent fails to complete it, they're analyzing why did it seem to go wrong? and based on how it went wrong, let me see if I can give it another test case or another scenario that it can compare against. And then they'll add that to the eval suite and then they'll run more through. And then each time that happens, they'll just keep adding more examples to it. Now look, not always, more examples doesn't always make it better.

23:46Wade Foster:So sometimes they are having to subtract from it, but by and large, like the best way I can describe it is these little test scenarios act as like a guide to the model where you're trying to steer it in the direction that you want more over time. And the more you do this, you can usually eke out higher and higher percentage of accuracy with these models.

24:05Matt Paige:And so, Zapier, you were effectively built on the API explosion as the basis of being able to provide this product. And MCP is the new thing. Connectors, being able to connect all your tools together. What similarities do you see between those two? And APIs are still a massively important thing in this new age, too. So how do you think of MCP versus AI and this ability to connect things together?

24:29Wade Foster:Well, they're really just the extension of the same thing. At the end of the day, it really is just about connecting to the tools that you use. And so APIs, and for most of our history, have just been running a call in a very specific way to sort of extract data from a tool or perform an action in another tool, so on and so forth. MCP just allows that for an agent to do it. And the nice thing about what MCP gives agents is the ability to kind of search and discover and kind of work in a sort of fuzzier way. And so you don't have to be as like specific about like the way to perform the task. The MCP sort of gives the agent the ability to kind of go figure it out on its own.

25:04Wade Foster:But under the hood, it's still effectively doing the same thing.

25:07Matt Paige:So in terms of the nature of jobs and work, there's a lot of debate on is AI taking jobs, is it creating new jobs? What is your take there? And I guess what are you seeing in terms of your own people? How would you describe the ideal type of worker right now? what type of qualities do they have? So you're hiring somebody new at Xavier. What are you looking for? Yeah.

25:32Wade Foster:The most general thing I think I can give is that the people who are very curious, who are doers, who have a high level of caring about their work, are just absolutely thriving right now. AI is giving them a jetpack to go figure out all sorts of things. And so whether you're just entering the workforce or you're on the cusp of retirement, these characteristics seem to accelerate people who really orient themselves that way.

26:01Matt Paige:Yeah, totally. And so let's shift to this. I'm curious, how are you using AI in your day-to-day life? What is the most unique, interesting use case of AI that may be not a common one that everybody's playing with today? Anything weird off the wall?

26:19Wade Foster:Weird off the wall stuff. I do a lot of -

26:22Matt Paige:We're just super productive. It's something that just - Yeah.

26:24Wade Foster:I mean, my favorite thing is, I mean, one of my favorite workflows is a little bit mundane. So a lot of folks like to talk about their daily brief. Candidly, I think the daily brief is like just okay. It's like fine for most people. But the flip of that is the daily recap. I think the daily recap is one a lot of people are sleeping on. So what does the daily recap do? It basically reviews everything that happened for you in that day. So you can have it go loop over all of the digital exhaust that you have from the day. So this might be emails, Slack, meeting recordings, everything, your calendar, and it can help summarize everything that's happened.

27:00It can help track down the key action items.

27:04Wade Foster:Even better than that, it can actually start to take action items. So if you were in a meeting and you said, hey, I'll make sure to make an introduction to this person or I'll make sure I'll follow up with this. Great. It'll already have those emails drafted for you. It already have like sample comms ready to go for you. And so you can do a lot of this stuff. The other nice thing I have it do is I often have it prompt me to say, how'd the day go? And so it's a very lightweight journal for me. And so usually I'll just do voice to text. Yeah, I'll just do voice to text real quick. And I'll say, these three things were awesome about my day.

27:36Wade Foster:These three things sucked about my day. And in the moment, that's not hugely valuable to me. But now I've been doing this for probably about six months now. And so I've got six months of data about what gets me pumped up during the day and what sort of makes me frustrated during the day. And so now I can do a whole bunch of interesting analysis on that and say there's some commonalities here that makes Zapier more successful, makes me more successful. And then there's some anti-patterns here where I find myself not being at my best or find the company not being at my best. And so how do I try and architect my day and architect my work to avoid some of those scenarios?

28:09Wade Foster:And so I actually find the daily recap to be just under discussed. And I think it's way more valuable than the daily prep.

28:17Matt Paige:No, I love that. And there's this memory component as well. Like when you're doing that reflection, it's learning over time. That's one thing for the, for lack of a better word, daily brief that I do is it's actually learning about the previous day. If I've slipped on something three times in a row, it knows, right? So there's this element of not repeating the same thing again, essentially over time. And so in terms of work, so we wrap it here, but the companies are obviously, they know the impact of AI. They know it's important. It's the top of everybody's strategy. But from somebody that literally called the code red at the beginning of GPT-4, what recommendations would you give to companies to actually drive this adoption internally?

29:00Matt Paige:Because it's not just about rolling out a bunch of licenses saying, hey, go do this thing. What advice would you give to companies other than start using Zapier to automate some things throughout your business? Of course, use Zapier for everything.

29:13Wade Foster:Yeah. No, I think there's, I would probably boil it down to a handful of things. And I would bucket it into two categories. I would say one I would call floor razors and the other category I'd call ceiling razors. Now the floor razors, this one's really important. This is about building widespread AI and fluency inside your whole organization. It's building a comfort level and excitement and energy for what is possible here. Now, the best floor raiser activities are generally things like hackathons, workshops, lunch and learns, any place where you're asking people to put their hands on the keyboard and actually learn and play around with this stuff.

29:52Wade Foster:What can they do that's practical in their day to day? We just talked about the daily recap and the daily brief. Great, go build one. Everyone should have something like that they can go feel, figure out. And whether you're an engineer or a marketer or a sales rep or an accountant or an HR, there's probably something in your day that you can use to build something that would just make your life better. And then you can start to build stuff that helps your team and all that sort of stuff. So these types of hackathons are just so powerful for doing this. Now, the other thing you get from this is you build a culture of an experimentation and you build a culture of excitement around what's possible with AI.

30:26Wade Foster:this becomes really important because you're going to have to go do some more interesting and harder things down the line. You might want to rethink like how the company works top to bottom. You might want to rethink job families and job structures. You might want to rethink just a lot of core concepts. And it's really important if you've built some of these, this floor raising culture internally, because now people are going to be a little more supportive of that because they're going to have a much more practical understanding of what's possible with these tools? What's not possible with these tools?

30:56Wade Foster:And so any other changes you're making in the organization will be less scary because it comes grounded in actually information and knowledge versus just people responding to the external headlines. I find the external headlines and what is actually happening in companies to be honestly pretty different in a lot of cases. So that to me is like the first category. Then the second category is the ceiling razors. Here, this one's important. It's how do you go from individual AI to institutional AI? And this one, I think a lot of companies are struggling with. Right now, they have figured out how to accelerate a lot of individuals inside their company.

31:33Wade Foster:We can all point to that engineer, that marketer, whoever, who's doing like crazy stuff with AI and is massively more productive. But it's a lot harder to go find companies where you can say, wow, this company is growing so much faster. They produce their cost basis. All this, the company is totally transformed with AI. Those are just a lot harder to find those examples. And I think a lot of this boils down to just because you accelerate one individual inside of a company, it doesn't mean that you've actually accelerated the whole company as a whole. You've just moved the bottleneck around. And so these institutional AI use cases are a lot more challenging.

32:07Wade Foster:You have to look around and try and break down what are the bottlenecks that prevent your company from growing? What is the thing that prevents you from getting another customer or making that customer happier or making that customer grow faster? And oftentimes those are things like, how do we actually truly accelerate time to market for a key product? How do we actually improve conversion rates in our sales and marketing funnel? How do we actually take waste and cost out of this business that's unnecessarily there? And so those, you really do have to rethink some of the core ways in which you go to market or you build your product or you operate internally.

32:41Wade Foster:And that requires a lot more tops down leadership and identification of the big levers inside of an organization. Much harder to do. much harder to do than the individual AI setup, but the rewards are much, much larger if you do accomplish this. And I would say there's very few companies who have really pushed the envelope on this. Even the companies you read about, I would say are still not as good as the things you've heard.

33:03Matt Paige:Yeah, it's so spot on. The individual productivity, but it's not spanning across the org. How do you think of the org of the future? You have Jack Dorsey talking about this hierarchy to intelligence. There's a lot of hypotheses of how the nature of an organization is going to evolve. Any thoughts or perspectives?

33:22Wade Foster:Yeah, I think Jack has got a lot of interesting ideas. The shift from individual to institutional AI is one of the things I'm really excited about. And I think the big bottleneck is you want to figure out how to increase the amount of time that your organization spends on those activities that are more likely to create a customer. It's how do you ship a product faster? It's how do you do better sales and marketing? And so you want to increase the time, energy, people that are spent on that. The challenge is as companies grow, you often have a lot of work that goes into stuff that's not that. You have a bunch of managers inside of a company.

33:58Wade Foster:You have a bunch of telephone games that sort of have to transport information from one team to another. If you've ever seen that movie Office Space, that's kind of what happens to companies as they grow. It's the TPS report that gets handed from one department to another. and a lot of that stuff doesn't really actually add value to the customer. It doesn't make them happier at the end of the day. It's just something that kind of has to get done because that was the best way to solve the problem inside that company today. So what I get really excited is how do you actually put AI at the center of how your company operates?

34:29Wade Foster:How do you build a brain so that the company actually has all of its knowledge institutionalized around it? How do you put automation and tools on top of that brain So now the AI can take action on those efforts. How do you put a governance layer around that so that it's really clear what humans can and cannot do in that system? And also it's clear what agents can and cannot do inside of that system. And then how do you put a harness around that so that there is an interface between the human layer and the AI layer inside the company to take action on all of that stuff? I think if you do those things really well, you can start to build a system inside your organization, that does take out a lot of that operational overhead where you can start to run closed loop meetings, where you can start to have the AI acting on your behalf in certain places.

35:18Wade Foster:And if you do that well, you can push more of your energy and effort to the things that humans are uniquely good at, which is this product development, which is the sales and marketing side of the house. Those things are stuff that distinctly bring value to customers. And that, I think, is when organizations get a lot more fun, but I think it's also when they get a lot more leverage from AI.

35:36Matt Paige:Yeah, it's so akin to system design, system architecture in a way. And last question for you, how do you think about your product roadmap? Things are evolving so quickly. And do you leverage AI on the strategy front, thinking through the product roadmap, what you're going to build next and how Zapier evolves? Has that changed at all for you over the years? Well, yeah.

35:58Wade Foster:You know, I think the thing that we have noticed is that the like six month roadmap is basically dead. What I see internally, what I hear from other companies is that you're having to be a lot more nimble on your toes about where you're heading and what is possible with these models. And so, you know, you're often thinking two, three weeks ahead, you know, at most. Now that said, I do think there are certain things that you're always trying to figure out, which are like, what are the foundational things that we do care about? And how do we sort of keep those at the center of where we're heading?

36:29Wade Foster:We know that this is the mission. We know that these capabilities are where we're heading. And so we're going to go hill climb against this and we'll climb against this for six months, 12 months, for however long as we think is important. But the way in which we get there, this sort of scoped out epics and stories and things like that, that stuff is kind of falling to the wayside in favor of these much more agile two-week roadmaps, or it's weird to even call them a roadmap. It's sort of just like, we just know we need to go achieve these things. Let's go figure it out.

36:58Matt Paige:Yeah. It's like there's tent poles in your strategy. To wrap us up, who are you following? Other CEOs, other leaders, other people in the space? Are there a few folks that you consistently look to for leaders in the world of either your world, AI in general?

37:14Wade Foster:Yeah, there's a lot of companies I think right now that are trying to be on the frontier. Obviously, you pay attention to the labs themselves. Those folks are really interesting to pay attention to. On the application layer, people who are maybe more closer to my seat, You look at the Brian at Coinbase and the Ramp crew, the folks over at DoorDash are doing really interesting things. So there's a whole bunch of folks that are trying to figure out, like, how do we operate differently with these things? What do we build that's different with these things that I think are really useful to pay attention to these companies and CEOs and whatnot?

37:45Matt Paige:Awesome. Wade, well, thank you for being on. Thank you for talking AI. Where can people go learn more about Zapier? And honestly, if you've used Zapier in the past and haven't used it in a while, like the product has evolved so much. I was just playing around with it recently. But where can people, you know, find you and learn more?

38:03Wade Foster:Yes. I mean, you can follow me on LinkedIn or Axe. I'm Wade Foster there. But, you know, if there is one thing I'd recommend that's different about Zapier is I would try and install it into your agent hardest of choice. So you might be using Claude or Chachukiti. You might be using Codex or Claude Code or whatnot. Now, to me, this is the best new interesting capabilities that we've added to Zapier. And so I think you'll have a really fun time playing around with all of the connections that Zapier has inside of these new tools. I think that is the one really exciting thing that you should check out with Zapier.

38:35Matt Paige:No, such a great call. It's like a new modality and way of working in a sense. Wade, thank you for being on.

38:40Wade Foster:Thanks for having me, Matt.

38:42Matt Paige:Thanks for listening to the Talking AI Podcast. If you enjoyed the show, give us a follow or subscribe on your favorite podcast podcast. And don't forget to leave us a review. We love those. For more info on Talking AI, visit TalkingAiPodcast.com. Quick break in the pod. If you're listening to this podcast, chances are you've been thinking about how to actually use AI inside your business. And that's exactly why we built the AI Opportunity Finder. It's a free tool that helps you uncover high-impact, tailored AI use cases based on your business, your goals, your pain points, and your industry. No fluff, no generic use cases, just real ideas that fit your business and they're ranked by ROI potential.

39:22Matt Paige:It takes about three minutes to run and it's like having your own personal AI strategist for free. If you want to try it for free, check out the link in the show notes or go to hatchworks.com backslash AI dash opportunity dash finder.