How Unify cut its AI agent costs 95% in two weeks

18:29Connor Heggie:You mentioned prompt caching. That's definitely a topic that we see a lot of people talking about. How did you guys implement it and what were the effects of it? So all of our agents right now run on OpenAI models through the OpenAI API. We also have some capacity on Azure so that if there is an incident on OpenAI, you can kind of like, you know, gracefully fall back. It's like a bunch of things around like, because we use the responses API, there's a bunch of stickiness between it that you have to like worry about. And so like, what is prompt caching? You know, I know that you know, but just like for anybody who doesn't, basically the outputs to us is when I send you the next step in a, you know, a model thread.

19:10So, you know, we're still doing reflect tool call, but it's just trained into the model now. So, you know, the model runs, it does a tool call. And then when I send the result of that tool call back to request the model to run more, you want to get a prompt cache, which basically says, read the KV cache, read the activations that you already had that you were just running for me, but just add a little bit more. And it's 90 % cheaper through the OpenAI API than it would be to send the whole thing. So it's like binary makes your product work or not work, basically, as well as speed it up. So it's really important that you get these cache hits, or otherwise you're just, you're cooked.

19:48By default, what OpenAI does is it takes the first, I think it's like 70 characters or 60 characters, some number of characters and it hashes it and then it routes based on that. That's really great if you have low volume. If you have high volume, that prompt cache key, I think it's 15 requests per second. So if you are above 15 requests per second, they make no guarantees on any of this, by the way. It's all best effort. So you can be mathematically perfect, which I'm not saying we are, but like we're, we did the math and we're pretty close and we're at like a 95 % cash hit rate or something, which is fantastic.

20:22Connor Heggie:So I didn't actually realize this. You're saying basically like if I send a hundred requests, even if all of them like match the prompt prefix, only like 15 of them roughly will like hit it and the others will not hit that prompt cash. Yeah, so let's take this case, right? So let's say you send one request with, you know, 100 ,000 tokens or something and that goes down. and then the next second you send 100, you will get 15-ish, right? Interesting. Yeah, prompt cache hits. But those other 85 will route to different machines on OpenAI's backend on the cloud that will then get the cache themselves.

21:01So the next time you send it, you will be more likely to get a cache hit because you have a bunch of machines warm, right? A bunch of the GPU racks warm, essentially.

21:11Connor Heggie:actually realize that's how it worked. I'd assume that if you sent it, you would always get a cache hit. No, that's on the first message. And then after that, they use the response ID if you're using the responses API, which everybody should use the responses API. If you are not using the responses API, you are missing out on 20 % or 30 % of the quality because you don't retain the thinking trace between different calls. So the response ID is actually used to route then to the correct machine. You can still get prompt cache misses there. But again, if you, for example, let's say forked a chat with subagents.

21:43So we have two types of subagents. We have child subagents, which is a full fresh context window. And we have forked subagents, which takes the current chat and forks it, obviously. And you pass the previous response ID and you spin up 10 or 15, you're probably okay. But if you spin up 20, you get a full cash miss on your entire context, which could be huge. And if you're using 5.5 is really expensive, right? It's like 80 cents or something. Again, on a$20 a month plan, that's like a big, big, big deal. And so you want to keep basically, there's like this game between keeping enough of them warm versus not.

22:16Another interesting point. So there's the user message and then the agent, you know, assistant message will have many model calls inside of it with the tool calls. And then the next turn, right? So then it gives its final answer and you're like, cool. You know, I ask it, how much money has Unify raised? and it says this much money. And I go, cool. Okay, thank you. What about this other company, right? The next message down, by default, right, the prompt cache key is hashed, right? You can pass in your own prompt cache key to basically say, hey, nudge, use this, right? Even if you get that perfectly right, on the next message down, they remove from the previous turn all of the old thinking tokens from between the messages.

22:58Does that cause a prompt cache miss? It causes a prompt cache miss. Why do they do that? Because it ends up being fewer tokens overall. And they think that on average, people want that. Do you want that? We don't want that. So if you do the math for us, the next message down, getting a cache miss. So let's say I sent a request and something really complicated, right? It took 15 minutes and 100 tool calls or something, right? Maybe not 100 because you'd have compaction and that messes it up. Let's say it's like 20 tool calls. in our case that you know maybe there's a couple thousand thinking tokens between that you know maybe it's a few tens of thousands of thinking tokens right that would get removed out that causes that whole thing to be missed right the issue with that is our output from tool calls is much much bigger than that 10 000 tokens of thinking state so we're paying for all that when we could have gotten you know the 10x reduction and so instead they they i think last month released an API param that lets you retain those immediately increases your prompt cache hit hit rate or it's not that the hit rate it's that the amount that you get in your prompt cache hit

24:08Connor Heggie:like the amount of cache tokens is much higher did you find that that also helped performance because now you're seeing these thinking tokens whereas previously they were removed i know earlier you said that it really really mattered does it matter in between turns or is it largely just for that for that single turn we found it mattered some to be honest so hard to measure that like you know does it like they say it doesn't really help like you know i look at 100 examples 200 examples i'm like it kind of helps like who knows the thing that's really important though is it's much faster for our users it's much cheaper as well because you don't have to reprocess all this tool call input at full api price input and so that that was one of these big improvements.

24:47But now it's really important because you have this game of, let's take the sub-agent case, for example, which is kind of the most illustrative of this prompt caching issue, this 15 requests per second issue. So let's say I'm going to spin up a thousand sub-agents, right? Which, by the way, value of our product, right? You can use Claude if you want to spin up three sub-agents that can do that. If you want to spin up a thousand, they can't do that. You can't do it on your local machine. You really need this durable cloud-hosted thing. Cool. So I'm going to spin up a thousand, a thousand sub agents.

25:17Um, let's play the, the like prompt cash game, right? So what do I do? I send my first one, I get a full cash miss. Okay. Dang. So my next 15, I get full cash hit. Awesome. So do I just process 15 at a time? Like that kind of sucks. Okay. So what if I process 15 in different, uh, you know, prompt cash keys, and then I can send 15 times 15-ish, right? So you can kind of like fan out. Well, that's great, okay, for a single user. I can maybe get, you know, you can do the math for like what the right amount of fan out is to start off with to get, you know, the optimal sort of like gradual warmup. But we have this great benefit, which is most of, for these subagents, most of the cost for us on input is this huge upfront context of system prompt and tools and things like that that are retained across users.

26:07So how do I play the game of, you know, I have a thousand users that are sending messages in a day. How do I distribute my cache, my warm cache across all of them? And to make a very long story short, the way that we kind of ended up tuning the knobs, you take the user ID and you hash it. So we have 16 hashes of user IDs. And so if you're user X and user Y and you hash the same thing, you'll share this first part of the prompt cache key. and then we just have a random number between one and 30, right? So for us, you know, we average at our peak, whatever that is, right? It's like 30 times 16 or something.

26:49You know, we average that about at our peak throughput on subagents. And so we want to have that maximal distributed across all of these different prompt cache keys so that they're warmed across a day to then do it, right? There's like, you know, go ask Fable to do the math for your like specific numbers, does it perfectly.

27:05Connor Heggie:Do you think that a lot of this prompt caching logic will just be built into the providers over time? Or do you imagine maintaining this? It can't be built into the providers, which is the interesting thing about it, because they don't know the distribution that's coming in. We have a bias, right? You know, like back before it was cool to do AI, we were doing ML, right? You know, traditional machine learning. And the thing that we talked, one of the things that we talked a lot about was inductive bias. And so what is the like, you know, inductive bias that we have or the bias that we have in our prediction mechanism?

27:33Well, we know that this is a specific user. We know this is part of a specific subagent batch of runs. We know that users on average have similar, you know, prefixes of their prompts. We know that users in the same tenant on average have the same prefix up to some point. And the model provider doesn't know that until they get that and get a bunch of the data. maybe over time you could imagine sending enough metadata to them that they then you know predictively look at all the different attributes that you're sending and then and then do it which would be an interesting product but for them they don't care right they charge you either way right like you know and like yes it's like more efficient because they'll push more tokens through the system but it really like kind of falls on you as a developer as it always does right like you know ai is amazing and all these like tools and stuff but like you still have to do the like hard engineering work.

28:25You mentioned fork sub agents. What's the use case for that? So let's say I'm researching, you know, you, I'm researching Harrison and, or let's say I'm researching five founders, right? Cause it needs to make sense to fork it. Doing like a ton of research there in chat. And I'm like finding really interesting stuff. I'm finding these interesting similarities between them. And then I want to go write each of them an email, right? okay like the you know easy way to do it would be to write the email out one at a time and so write the first email for the first person then write the second email for the second person the third email for the third person that's just really slow you know and now let's say you want to get with 20 people like that's really slow and then your context window gets really big and then you know like it actually bails out like midway through because these models are like kind of lazy you know they just go etc um although that's not not really a problem with the latest gen of models and with a fork subagent, you give them all one clean task and you say, hey, we've been talking about all these people.

29:21Just go write an email for Harrison.

29:23Connor Heggie:Why can't you use a regular subagent and prompt it with all the knowledge previously? I have my take on it, which is mostly from intuition. And there's two different pieces to it. One of them is you have to summarize all this stuff, right? And it's going to be a really lossy summary no matter what. Or it's not a lossy summary and you put a ton of information in this prompt. Okay, like I was saying, we've optimized prompt cache hits a bunch. Okay, now you're getting a full prompt cache miss. Like I could just be getting a cache hit and then 500 tokens out for the email. But now I have to go summarize all of the stuff so I get a big output and then pass it into the input of a subagent and then have that parse through all the input and then output, which is really messy.

30:07But on more of a like thesis-y level point, the distribution, you know, like imagine, okay so these what are these llms like these llms are like functionally modeling language right they model the distribution of language over like some trace or some trajectory like some chat right and i've gotten it into just the right spot with all the things i'm saying to it right because i'm i got i know what i want and i got it right to the right spot that i want like it's so obvious what the next the next thing is the just the perfect email for me and so it's in the right sort of vector space and it's like, you know, language model.

30:40And I just wanted to like output that. Whereas, you know, with the subagent, it's like, got to come up in some random fucking thing.

30:47Connor Heggie:You mentioned like sending emails. How do you think about human in the loop? Like I imagine I would want some form of human in the loop if I had an agent sending emails for me. How do you guys handle it? Both like maybe from like a UX point of view, but then also like, yeah, anything interesting technically under the hood? I'll do this maybe in reverse order. um technically under the hood we do a bunch of managed deliverability for our customers so um one of the great things you know um like i feel very fortunate i feel very lucky that we started the company three and a half years ago sort of pre ai wave because we had a bunch of this really strong like standard architecture engineering that you like you know to build on top of right you know we had this amazing deliverability system and durable you know durable email sending, as it turns out, like guaranteeing an email sends exactly once is really hard to do.

31:35Like, you know, you can get it to be like on average once pretty easily, but like not sending it, double sending it and always sending it is not trivial. And so we had all of that built already, which was awesome. On the UI side, the UX side, totally don't want emails ripping out of this agent sight unseen. What we ended up on was this sort of proposal and then approval system. So inside of our agent, we have an artifact, you know, UI, UX, very similar to what you have in Claude, where the right side is sort of a full screen or almost full screen view. We'll put a table in there most of the time when you work with a table or for email sequences for emails.

32:17We have this sort of quick approval flow where you can scroll down and read it. And then we, you know, and that's kind of the easy version, right? So that's like V1. And you go in, you can edit the email, you can look at it. But then one of the things that we found was that all of the sort of second order interactions end up being really important. So the highlight a sentence and then ask it to change it, not just on this email, but across 100 enrollments.

32:40Connor Heggie:Yeah, I was going to ask, like, I get how you might show one email on this right hand. And I want to talk more about UX because I think UX is pretty under invested in and under talked about. And so I love that you already went there. So like, I imagine what that could look like for one email, but you're talking about like maybe a thousand emails. Like how do I, like, how do I insert myself in the right position there? For us, um, a couple of things that were, that were really important. Um, one, you had to be able to see the full email and you had to be in context, right? So you needed to see who I'm reaching out to, the job title, what company they work at, a bunch of information.

33:11Connor Heggie:For all like thousand rows in the table. For all, for all thousand rows, right? So we played a bunch with, do we show, like you're saying in a table, the full email, and then, you know, columns off to the left. We tried it. Ended up being really noisy. People I found, myself certainly, and seeing other people work with products, you're much more effective going one by one really quickly than you are looking at a bunch in a row. We had this internal tool. I used to work at Scale, which is a data labeling company. We used to have this tool internally there called Speed Audit, which basically let you queue up a bunch of work and then you would just hot key central, look at it, move on to the next one, look at it, move on to the next one.

33:50And it was really effective because you could zone in, you could work through it one by one, right? Sort of like, you know, the modern day, like Cal Newport deep work of just like focus on the thing and like do the thing really effectively. So instead what we opted for was show one email with all the contacts you need very quickly. But then if you want to modify things, give you tools to modify it across the whole batch. Although I might be looking at one email at a time over five or six or seven or eight, by the time that I've gotten to the 15th one or the 20th one, I can give them modifications and then I'm probably ready to send all of them.

34:25Connor Heggie:So if there's like a thousand emails, you generally see people maybe like reviewing the first 15 or so, giving enough feedback where it applies to all of them where they're comfortable setting, but they don't go through all the thousand. It really depends. Some people, this is, if humans were going to spend time in one part of the system, this is the point to spend time in, which is kind of the, you know, funny thing. You know, SDRs, the salespeople that we sell to, this is their secret sauce. You know, it's not like Google searching until I find the right company. So like we totally automated a bunch of that work, right?

35:00Gave them the scalable things on that, but they really want to make sure that they send the perfect message, right? There's not an m-bash in there, right? They're talking about how they surf and they saw that this person surfs and like they were just, you know, surfing at, you know, blah, blah, blah. You know, I just went to Hawaii and it was so nice, you know. That like human connection point is actually their alpha. And so, yeah, we'll make it really easy for them to do that part of their work. And then if they want to like ship all of them, like, you know, command A ship, like that's fine.

35:27Connor Heggie:On that note, how do you guys think about personalization? Because I imagine every seller probably has a different tone or different things they like to mention. And great, so they can go through this flow and correct it. Do you remember that over time? do you let them set preferences like upfront? Like what does that personalization story look like? Yeah, I'm so glad you asked that. Great, great question. So we have two flavors of this. One of them is memory, which is built into the agent. The other one is sort of more of this Unify specific thing that you're not going to get in Cloud or ChatGPT because we specifically are building for sales reps.

36:02And that's when you integrate your Gmail mailbox. We look through your whole inbox. we come up with the ways that you like to write emails and not just like the generics of your tone and the generic you know like the things that all sorts of products you know might do when you integrate your inbox to try to replicate you but we learn how do you talk about your product and why and what types of people do you say what types of messages and is that is that different than memory it's a different process to get it in but then we push it into our memory system so that it can be recalled at any time.

36:35It can be pulled into the email, you know, content writing pieces. But the system to get it in isn't a, hey, remember that I like to write XYZ kind of email. Or even, you know, we do have the, okay, now edit this email to say X, which is great. And there is some signal there that we push into our memory system. But maybe I just like this one email wanted to say something different, you know, when I say, oh, hey, like make this kind of funny. Maybe it's because I'm sending, you know, my friend an email, not because I want all of the emails I send as a sales rep to be funny.

37:10Connor Heggie:So I want to dive into this because I think this is like pretty similar to some of the other things we see in memory where there's maybe this like upfront kind of like just knowledge extraction. Sometimes like we wrote a bunch about this this past week around like this like wiki creation. I saw that. It looks sick. Yeah. I mean, we see this in like coding agents, you have like deep wiki from cognition and things like that, where they do a bunch of upfront work, they take this raw material, in their case, the code base, in your guys's case, it sounds like their email history, and they produce some like condensed version of it that's like, useful for agents kind of like going forward.

37:42Connor Heggie:And then there's also this other like, as you're interacting with the agent, how do you how do you take learnings there and update? So I actually want to talk about both of them. Question, like, what is the underlying data structure? Is it a set of files? Is it a wiki? Is it a knowledge graph vector store? Like what is this data structure for you guys? So it lives, lives in our Postgres database. It's essentially, let me maybe like explain the overall extraction and it'll end up like with the right data store. There's the email, there's the bootstrap step, which is kind of what you're saying, which is this initial knowledge extraction.

38:12And then there's the ongoing system, which both proposes, right? I'm really big on this, this like thesis or concept of generative and then discriminative, right? You're convergent, divergent sort of thinking where we propose, propose over propose a bunch of things of, hey, like in this message, I think it's every third message or something you send right now, we will take, you know, proposed memories out of it and say, hey, this user might have said blank, this user might have said, you know, might like like this might like that. In this kind of scenario, do this kind of thing. They're all structured.

38:44So they're, you know, we have email voice, we have a general user preference, we have hard user preference.

38:50Connor Heggie:So you basically set like a schema of this data structure that you're trying to ingest things into. Yeah. And so it's, it's arbitrary strings. So it's, you know, natural language for the content of it, but then we give it some structure above that to give it like different classifications so that at recall time, we always recall at least some of the right ones. And then we can do the, you know, semantic search over the rest of them. Okay. A few questions. Yeah. So, so like the, the keys, not the values, the values, arbitrary string, the keys, the structure you're imposing, is that a fixed structure?

39:25Connor Heggie:Can it like add more keys? Can it add more structure? No, it can't. So we gave it, I think there's like six or seven of these. And they're, like I said, they're, again, we sell the sales rep. So we have a good structure on top of it. And we have, you know, email voice, right? Like rep voice, attributes about the rep, attributes about the company that they work at, user preferences, you know, soft user preferences, hard user preferences, a few things like that. And then we have an other category, which is kind of our bucket all, you know, for that. We decided not to do the arbitrary keys and have the agent sort of self-manage, largely because every added dimension of complexity is multiplicative instead of, you know, additive.

40:10And so having it manage both the keys by which it'll classify as well as the classifications, you know, felt scary to me. Maybe we end up there, you know.

40:19Connor Heggie:I'm curious whether you'd agree with this. I think we generally see that the hardest part of this is like getting, I'm assuming you're using an LLM to reflect on both the raw data and the conversations. The hardest part is usually getting the LLM to decide what the right thing to remember is. The data structure is not that hard. As you said, it's a key value store. It's not that hard. The hardest part is getting the LLM to remember. And so, yeah, like enforcing the structure and like saying, hey, we only care about remembering these seven things, I think makes a ton of sense if you if you can do that reliably it's all about that like inductive like what is our inductive bias like what is the thing that makes unify special and it's that we care a lot about sales reps and what they need from a chat product and we know what they should you know what the agent should remember about them totally so so the ingestion of like the raw emails i'm assuming there's like some agent that's doing that basically and deciding to write to those keys and then for the conversations you said like every three messages you basically run this this process Lightweight proposal.

41:10And so what'll run is it'll overgenerate proposals. And then we have a system that'll run on a cron in the background, which takes all of the proposed memories, as well as all of the historical active memories. and then we do a sort of justification step or a processing step where we either promote a draft memory to an active memory, we merge memories, we deprecate memories, you know, and then we actually give it a pretty structured set of lineages to process then over time where it can't, it doesn't just decide what to do, you know, here's my current memories and here's my new memories. It has to say, okay, memory A and memory B, the op I'm going to run is merge these two and then here's the new memory c and then output that or drop memory d completely right and then it's marked as dropped or this memory actually supersedes this one and so you know this one's superseded and then you end up with a new set of active memories why does it matter that it like

42:11Connor Heggie:marks something as dropped like do you use that like history in any way or is it just yeah why does it matter observability right for us we can go in and a lot of a lot of the LLM engineering is just black box right you say you know and we have evals that are great right we love evals you can you can learn things over time but so much of it is a black box and just like imposing structure in the right way to make it observable to you so that when we as engineers go and look at the system in three months and we say well that was pretty dumb like what was it thinking there, at the very least, there's like some structure to say, oh, well, you know, it merged those two memories.

42:53And I guess I could see why it merged those two memories.

42:57Connor Heggie:Speaking of that, like how much do you expose to the end user? Do you let them see their memory? Do you give them control to say like, don't remember this, remember other things? Do you let them approve changes to their memory? We don't show them in app today. If you ask it, what are your memories about me? It'll tell you. We don't have a structured UI. We've gone back and forth on that. We might add it. I don't, as a user, want to see it. Like, I'll ask you what you remember about me. You know, like, I don't want to look at a bunch of bullet points about me. I also get like a little bit of the, you know, like, I get like embarrassed about what the LM knows about me or what the agent knows about me.

43:32So how much do we let the user see? You know, not that much. The thing that we do take, though, is one of these classifications that we have of types of memories is strong user preference, direct user preference. And so anything that the user says, hey, remember blank, will always be inserted as this top level kind of like first class concept. The funniest use case for this is there's a bunch of people that are say, you know, remember to always talk to me like a pirate. And it remembers it every time, which is so funny. Or talk to me like a dude, bro. And it talks, you know, and like so many of our customers are, you know, their sales reps, they're like usually oftentimes is their first job out of college.

44:10You know, they're like pretty new. They're like, you got to have some delight and fun in the product. And this is totally one of those ones that ends up being a delight.

44:18Connor Heggie:How advanced do you think AI penetration is into kind of like the sales world? Like, it's probably not as far along as coding. So I imagine you get people who are maybe earlier on and trying things, but like, where is it? Is it super early on, middling? Yeah, I would say everyone's using it somehow. Teams aren't standardized or structured today. And so one of the things that we hear all the time from sales managers, sales leaders, even C-suite execs, is how do I figure out what the most effective AI usage for my sales reps are and get them all to do that? In fact, it doesn't even need to be the most effective use.

44:59It just needs to be like a pretty good use. And then how do I get them all using it? Then how do I hold them accountable to using it? How do I, you know, get them using it in similar ways? How do I track them with AI? How do I make sure that they're not spending a million dollars or a thousand dollars a month or whatever your budget is per person?

45:17Connor Heggie:A few questions there. I imagine one version of this could be like, hey, someone's got a really good skill. Let's make sure everyone uses that skill. Do you guys have a concept of skills in your platform or are you guys basically like, hey, the workflows we have, these are basically souped up skills that are way better and it's just used this? Yeah, it's 100 % that today. We've done so much. We have dozens and dozens and dozens of skills and skill files and guidance and things like that across different data sets and data providers and the way that you would access these data providers and whatnot.

45:47So today it's all sort of like Unify curates all of it. We have all these best practices. You know, we work with some of the best go-to-market teams and we take, you know, all the things that we know and, you know, like have worked with them on and make it really accessible for everyone. Will we add skills? I think we will because different companies do have different theses. A great example is, you know, when you're accessing one company's Salesforce instance versus another company's Salesforce instance. I know the most riveting thing on the planet. All engineers love talking about Salesforce.

46:14It's very important. Really different on how to do it. And so those, you know, we would want. But whether or not we make them skill files or not, I'm a little bit torn on. Engineers love writing things like skill files. You know, it's like writing a runbook. But most other people don't. Why not just have it be memory about your company, right? Like my company, our Salesforce operates like this. Okay, well, that's like just a memory.

46:38Connor Heggie:You mentioned people caring about costs and how much people were spending on this. I saw one take on Twitter recently that I liked, which was like, every AI product will have some form of like cost controls built in for their end users to control that. Have you guys started doing that already? Or are you still in the phase where it's just so early on you're trying to get people? because like I'm thinking about coding up until six months ago, we were just like, yeah, use whatever you want. And, and, and recently it's become where we need something. And so it switched somewhere recently. And so I'm curious, yeah, like is, is that built into the product yet or is it on the roadmap?

47:13Yeah. When you sign up, uh, there's per seat limits. And then, uh, what we've seen is that's largely good for everyone. You sign up, you, you spend 20 bucks a month, 60 bucks a month per user and you get some allocation and then that kind of helps like guardrail you directly. And then you can pay for sort of pooled credits for people who go over very similar to like a, you know, Claude or ChatGPT model. We don't have the per user guardrails where, you know, within this then shared pool, let people, you know, guard within that. I think we'll build it really soon. We need something like that. It is really interesting though, because the important thing will be that it's per user guardrails.

47:52Because what we see is some people are really trusted. Like if we look internally at, you know, at Unify even, I think I spent like$7 ,000,$8 ,000 on LLMs last month. And like, I trust myself. I don't know. I trust that spent. Like I was crushing PRs last month. And then the next highest on PR or on, you know, token spend on, you know, dollar spent was like 5 ,500 bucks. And I also really trust him. He was cranking like that was well worth it. But there's other people on the team that if I would not trust to spend$5 ,000 a month, I would trust him to spend$500 a month. if that goes well like totally like let's rip open a thousand dollars a month maybe but there's kind of this like um you know like you kind of earn the right you earn the trust to be able to do that

48:37Connor Heggie:we also see i mean we see this in coding that like some of it's just like accidental spent yeah and i imagine if you're spinning up like thousands of sub-agents that accidental spend could also kind of like creep up on you but i think like you know again we didn't really care about costs until maybe six months ago and it sounds like sales is like earlier on so it's I think, as you said, like people want just more adoption and more standard adoption of best practices. And I think that makes perfect sense. One of the things you mentioned earlier, evals. How do you guys do evals? We use Langsmith to run our evals.

49:07Yeah, how do we do evals? We do a couple of different kinds of evals. So a lot of my thesis on evals comes from my time working in self-driving. So before, right before I started Unify, worked at scale on the ML team, the mapping team before that. And then before that, I worked in a small self-driving company. It was like 15 people, 11 of them were math PhDs. Like it was run like kind of a research lab. At a self-driving company, the product is the car that drives. And like the thing that makes it drive is the vision models, right? So how did they run it? How did, you know, how did we run it? Of course we had evals.

49:42Of course there were metrics that we looked at. And that was very important. But the gate to get a model on the car was this big grid. it was this DQA video this like video this like hour and a half or two hour video that had you know seven second clips it was this table of you know six rows down five rows wide or five columns wide and each row was a different model run different like training run and each column was a different checkpoint along that training run and we predicted the same video on all of those different checkpoints of all those different models with the current best at the bottom you know semantics It was a semantic segmentation at the time.

50:21And we watched it. We sat down as a team and we had a projector in the living room equivalent of the office. And we sat and we watched like 30 minutes, an hour of video. And we took notes and talked about it and said, you know, oh, like it actually gets this right there and this wrong there. And there's just no better eval than looking at 100 examples or 1000 examples. So we do a ton of that, a ton, a ton, a ton of that, oftentimes filtered down to specific distributions of things and issues. And so we'll classify every message as a certain user intent. So we have, you know, this user is trying to find people, this user is trying to send emails.

51:00And we might look through a bunch of examples for that to see how the model's doing or how the agent harness overall is doing. That's a bunch of it. And that's really helpful to get a current feel of like what's good, what's bad, you know, where can we improve? But then you need regression testing basically to make iterative improvements and so we'll pull traces from that into a data set which you then run an eval over we have different basically buckets of it we call them dqa sets dedicated qa sets or or like you know they're basically just groupings of these examples that are meant to cover you know some specific distribution of usage right whether it's our hero use cases that we just know should be rockstar, you know, rip out, or kind of adversarial use cases of people trying to prompt inject, or, you know, weird use cases where somebody is speaking Spanish midway through and we didn't expect that, or things where they, you know, etc, etc, right?

51:57And we'll iterate on one of those at a time, usually. So let's take, you know, today, for example, we have this specific data vendor that the agent is just hilariously over-calling, right? Like you show up and you ask for 10 companies and it calls it like 500 times.

52:13Connor Heggie:They did some good agent engine optimization. They crushed it. They did. And we spent a lot of money on that API. And we should not be because most of those results don't get to our customer. And so we have a DQA set, which is 40 examples where it went off the rails and called this a bunch of times, pulled those into a data set. And then we iterate on it with a bunch of metrics, a huge number of metrics of number of tool calls, the tool call efficiency, the trace efficiency, the credit cost that we would give to our customers, the LLM cost, kind of all these metrics that you would look at. How do you test services like that where it calls, where it costs money to call it?

52:51Connor Heggie:And then maybe also like emails, like you're not going to send emails as part of, well, maybe you will send emails as part of some tests, but like, yeah, how do you think about calling these services during tests that are either cost money or like take actions? So when sending the emails, the actual end result of the email sending doesn't matter to the agent. That one's easy. You mock it out. It like fake sends. Okay. Easy peasy. A harder one is ask user questions. So you ask the user some questions like, what do you do in the eval harness? So we like have another LLM play the user. But now you're starting to stack distributions.

53:23One thing that I always like, a drumbeat that I always hit with the team is when you have an LLM as a judge, it is, or it's interacting in this user interview point, it can never be the same distribution as the original model, right? So what does that mean? If we are running GPT 5.4 in our agent, we have to be using an anthropic model to do the judge, to do, you know, things like that. Probably for a bunch of obvious reasons, but just to say it out loud, right? You have this, like, you know, mode collapse of, like, agents talking to each other, this, like, you know, overlapping distributions where it's, you know, the equivalent of groupthink in humans.

54:00And you just, you badly don't want that. you want it almost adversarial, right? You want it in a completely different distribution, you know, like language modeling distribution. And so we do that for a bunch of them. On the, you know, APIs that we might call, we have two modes. One of them mocks it out and will replay sort of old ones. That's not a perfect test though, because let's say this new one, it actually just calls the API once, but with really good params to get the single result. Well, that's not going to get replayed. So it's not a perfect fit. so oftentimes we'll just actually take the cost hit and run it for real we are already spending a lot of money on the llm costs to go run these so the data cost is kind of marginal on top of that

54:43Connor Heggie:so i want to ask about that we've talked about cost a few times it sounds like you've got cost of llms cost of some of these search providers cost of sandboxes i presume in some form what do do for sandboxes? It sounds like you maybe have more kind of like stateless, quick ephemeral sandboxes, or is it incorrect and you actually have stateful, long-running ones? So they are stateful, but they're not long-running. This is so great. There's like the, you know, anthropic blog that was, what was it, like split the hand in the heads or something, the brain in the hands or something. I know what you're talking about.

55:14Connor Heggie:I think it's, yeah, brain in something. Brain in the hands, whatever it is. Basically run the agent outside, have a sandbox that executes things, but it's not running inside there. But it's not running inside of. Very important. Strongly agree with that. Especially important with this cloud execution environment that you might want because you need, again, durability. If it crashes, you need to be able to handle that really gracefully and things like that. Then this other point around how do you ensure tenancy to make sure that the data is always well-scoped to a single user, a single customer's data set.

55:50And so where we netted out, we don't have a full VM. So one of the really common options is you create, you spin up one of these, you know, like Daytona VMs, LinkSmith or LinkChain has a VM. Do you guys have just the, do you guys have the VMs and repels or just VMs?

56:04Connor Heggie:No, just repels. No, we have VMs. Yeah, you have full VMs. And so you would spin one up, spin a full VM and then, you know, inject commands into it. Yep. Which is awesome because you get statefulness, you get a file system, you know, you get a bunch of really great stuff. The place that we struggled with that was you then have to write a bunch of CLIs. So maybe you wrap a bunch of your code in CLIs, but then the CLIs you need to like, you know, ensure tenancy over them, which like then it can actually write arbitrary code inside, which was really scary to me. Then you have to deal with all the networking, which is also really scary.

56:39How does it access if it needs to access other services? Like, do you have it then pop back out and hit your API and then your API hits your internal? Like it's, you know, really, really complicated really fast. And so we stumbled upon this open source framework called Monty. you've seen. It's in Python, which is great. And essentially what it does is it gives a fake Python interpreter or real Python interpreter that the agent can spin up and run and actually execute real code inside of. Except you bind functions into it that once you hit that, you suspend the whole REPL and the whole code execution environment.

57:13And then you break and you say that your host process, your backend code gets basically a function name and some args and says, hey, it tried to run this function. You get to run that function however you want. Then we inject a tendency at that level. We say, cool, we're going to run this in the scope of this specific user, in the scope of this specific thread. We'll add a bunch of billing stuff inside of it, et cetera, et cetera. And then when you're done, you resume that REPL backup and inject the outcome, the output of that function call basically into it. Like I was saying earlier, we really want it to be writing TypeScript because we're fully TypeScript shop.

57:50So we rebuilt that internally to do TypeScript. And then that runs in our actual backend process. So we're running workers that are actually calling the LLM, calling the agent, and then spinning up these really ephemeral repls that add to over time. So even if it's calling it, you know, once, then, you know, again, similar to a Python notebook where it can access variables that it had previously, but, you know, now at the end.

58:14Connor Heggie:We just launched something in Deep Agents using QuickJS. Amazing. Which is, I was about to make a funny comment about how you guys are a TypeScript shop using kind of like using a Python REPL. But that sounds like it's not the case. What did you guys do with QuickJS? It's the same thing with like the ability to like host, like inject in host functions. Yeah, yeah. It's super useful. Super useful for in particular kind of like programmatic calling of subagents. Yeah. So that's the big thing we launched. basically write a script that has, we have like a task function, which is basically the same as just our subagent function.

58:50Connor Heggie:Just kicks it off and we have some nice prompting to like tell it. It's like a combination of like Anthropics, like dynamic workflows thing. Yeah. And like the RLM paper. I'm obsessed with RLMs. We haven't come up with a way, it's way too expensive to run for us, for customers, but it is brilliant. It gives you like basically merge sort, but over with LLMs. It's all just programmatic calling of subagents. But yes, to your point, like, yeah, we did some benchmarks and yeah, it gets better results. It's like, it is a lot more expensive. It's so expensive, but it's so sick. Yeah. Cause like, think about this, right?

59:20Today in Unify, you can go run this query and you can say, pull my book of business, pull the hundred accounts that I am assigned to, and then score them. Tell me the best one for me to reach out to today. Okay. How do you do that? You either have the main agent iterate over each one and remember, yuck, or you have it right, which is what we'll do right now inside of Unify is it'll do some research on all of them, create a scoring function, then order by the scoring function, which is very, very lossy, very noisy, right? How do you do that? So really what you want is basically like semantic merge sort.

59:50You could do that with RLMs, which is crazy, which is so sick, yeah.

59:53Connor Heggie:So it sounds like because of this, your sandbox code interpreter costs, zero. Yeah, great. Okay, so you've got LLMs, you've got search. How do those compare? Depending on the day, different. We actually got probably a 90 or 95 % cost optimization from two weeks before we launched to the day that we launched. It was a very big effort internally. We were very concerned because what that would have meant is you show up to Unify and you send one message on your$20 a month plan and you're done. And that's insane. What'd you do? So many things. Well, so we were running so many sub-agents and that's way too expensive.

1:00:33So how do you move to doing this main agent that's smarter, that can write code that it maps on top of the table? That was like a big set of optimizations. A lot of the optimizations that we found, like, you know, you go and you just like, really the answer is we dig into the traces. And you bucket things and you find ways that it's going off the rails where it's being dumb or not doing the right things. And then you optimize the prompts for it. you look through all the, you know, one thing that we did that is very obvious, but of course, like, you know, made a huge difference. We looked through all the skill files, we looked through the system prompt, and we made sure that there were no contradictions.

1:01:10Because every little contradiction meant that it messed up a tool call, it messed up like the ordering of something and did it. You know, I think we need this, like, we don't have this today, somebody should build it as an open source project, or maybe you guys should build it as a product. We need a set of like semantic linters over our skill files. Similar to how we do with evals that do things like self-consistency. Is within a skill file, is it consistent? Is every combination of skill files, if loaded, consistent with instructions? Please tell me you guys are building this.

1:01:38Connor Heggie:No, but I've thought about it because we have Context Hub, which basically stores kind of like skills and agent.md files. I hadn't thought of it for, I thought of it more generally because I think another use case is if you're updating these automatically or even not automatically, you don't want PII in them. Like that's just generally bad. and so having some like lint rule for that but like yeah i think like ensuring consistency or just like is it too long i don't know totally yeah like you want both like program at like programmatic versions of it which are like the hard you know like are there m dashes and like we don't want it to write m dashes so we shouldn't have m dashes in our you know instructions so boom right like easy one or like we try to enforce a rule which is we don't say do this don't do this never do this, because the more that you add that, models have just gotten so smart and smart enough that it should, if you give it the why behind that, they're really good at figuring out what to do and coming up with the right trajectory.

1:02:32But they also will follow your instructions no matter what you say. So if you say always blank or do not blank in the 3 % of cases where that's wrong, it'll never do it.

1:02:45Connor Heggie:So going back, you got costs down. We got costs down a ton. What percent of that is model versus kind of like the search? It's hard to tease out how much of it was model versus not. I would say probably of those reductions, a grand majority of it was fewer token costs. So actually bringing token costs down very, very dramatically. A large part of that also, we were using 5.5 instead of 5.4. We brought five, you know, we like brought that down a bunch. A lot of it also was just inefficient tool calling. So it was doing a bunch of tool calls to vendors it didn't need. So we did a lot of optimizations of how do you make sure that you are always calling functions that you will actually use the results of.

1:03:27That's like a big part of our evals, wasted tool calls, something to drive down. It's kind of the number one thing to drive down for us. Sub-agents, a ton, a ton, a ton there. But actually, like I would say on a maybe like a practical takeaway thing that really like moved the needle for us was adding in a really robust planning step at the beginning. You know, going back to our three years ago, adding a one, right? Or two years ago, adding a one to our, as a planning step to our four O agent, just tell the model to pause, whatever the user asks you, pause, think about it for a little bit, come up with the ways that you might go solve it, and then ask the user a couple of questions to clarify it and then go and do it.

1:04:12And then as part of that planning step, it'll scout out a couple of trajectories. So let's say you say, hey, find me 30 companies that aren't in my CRM that are hiring for AI engineers and have raised a series B in the last two years, right? It's actually like a pretty hard set of criteria to like find all of in one go. And so it might scout out and say, okay, cool. I'm going to try and call this data vendor and get some companies that are series B. I'll try to call this data vendor to find some people hiring for AI engineers. And then I'll look and say, oh, well, this one's higher recall, but this one's really high precision.

1:04:49So let me, I'll actually start with that one. And then from there, you know, go run the full suite of finding 20 or 200 or whatever it is.

1:04:56Connor Heggie:It sounds like you guys are mostly on open AI models. How do you think about model choice and model selection? Yeah. Yeah, we are. Honestly, it was a pretty easy decision for us to start off with. OpenAI was also our first investor. So I have to say that they like are great partners. They're amazing to work with. And the models are really good. They're really cost effective. I'm like clearly talking a lot about costs. And, you know, somebody I remember me five years ago would be listening to this and being like, oh yeah, you know, haha, costs are so important. Like, I just want this thing to be sick.

1:05:30And like the sick part is that our users can log in and send 30 messages or 40 messages, 50 messages, get a ton of value out of it instead of using Fable and it killing it on one message and being like, okay, well, here's another 50 bucks. For the price performance, it was really, really easy for us to make that call pretty early on. We evaluated both of them. We've been using both of them for a long time. Open source models at all? So we evaluated open source models. We did this big, big, big cost reduction in Q4 of last year where we took our old agent costs, that research agent, brought it down to sub one cent, which was awesome.

1:06:10So, you know, like we passed all of that value, by the way, onto our customers. We 10x reduced our credit costs to our customers. And actually now on that old agent, tool calls actually dominate the cost there. And this one in this new agent that we have, we try to think about it a lot as the value that we're giving to our customers is in large part, not entirely, but in large part, the data that we're bringing to them and helping them define and craft and filter and curate. So how do we keep the token cost to be a percentage, a small percentage of that data cost that we're bringing to them to then output?

1:06:48And so open source models would be a fantastic fit for that. The issue that we ran into when we evaluated them in Q4 of last year is they're so much less tool efficient that even if they are 10x cheaper, which they're not, but even if they were 10x cheaper on the token costs, the tool efficiency is actually not worth it, net net. That being said, I've been hearing great things about GLM 5.2. I've been, you know, trying it out on the side and it seems great. So maybe that is like the saving grace, right? And we will certainly fine tune and model ourselves very soon. You know, all of that good stuff.

1:07:24fable came out you know maybe not to date this this you know podcast but like fable just came out again yesterday and people are rerunning all their evals on it and it's actually a pretty cost-effective model because it uses way fewer tokens than certainly sonnet 5 which was using way too many tokens and so that trade-off is really non-obvious and again to go back to the evals point the only way that you can do that you know rigorously is with great evals all the evals that we run, we have a pass case, so we'll run like, you know, five runs of each, you know, trace to evaluate to see how well they perform, you know, rather than, you know, one at a time.

1:08:04But I really, I really badly want open source models to be there. They're just not yet.

1:08:08Connor Heggie:Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to maxagency at langchain.dev. We want to hear from you.

1:08:23Thank you.