


Clay’s new “Clay” agent rebuild for go-to-market, shifting from task-focused agents (e.g., Sculptor) to an end-to-end, outcome-oriented agent built on a coding-agent harness.
Jeff Barg is head of AI at Clay, a go-to-market platform. He led the rebuild around a coding agent and discusses evals, harness engineering, sandboxing, and production monitoring.
Written by AI. May contain mistakes.
Discussion about the new Clay agent and its differences from previous versions.
“We actually started thinking about the memory piece before we started thinking about the agent piece.”
Exploring the distinction between task-oriented and outcome-oriented features of Clay.
“And I think, you know, that drove a lot of value.”
Insight into the user experience and interaction with the new agent interface.
“I mean, I feel like those types of proactive agents have come out a lot recently and people are paying a lot of attention to that.”
Details on the architecture of Clay's agent and challenges faced during its development.
“Is it the same concept and like each thread has assets somehow associated with it?”
Methods and strategies for evaluating the performance of Clay's agents.
“probably five years ago wouldn't really know what a cli is um you have like these rev ops teams uh or, you know, maybe like very AI forward, but like go to market builder teams that, yeah, I mean, they loved the CLI.”
Discussion on evaluating first-party agents and their performance metrics.
“and so helpful from that standpoint, but very awkward from the standpoint of like, the metrics were almost meaningless.”
Debate on the strategic choice of building first-party agents versus using third-party solutions.
“Yeah, I mean, I think like, so we have an agent as well that's very similar in some sense, an engine that runs in the background, looks at traces.”
Exploration of benefits related to orchestration, cost, and performance in first-party agents.
“you know third party versus first party, how you guys have thought about or advice you'd have for companies, maybe when to do it, when not to do it.”
Insights gained from previous project Sculptor and its impact on current agent development.
“I want to come back in a second to how technically you guys drive those outcomes, but maybe stepping back and setting the stage.”
Discussion on harness engineering and strategies for optimizing agent performance.
“When we launched Sculptor, I think we had a pretty good understanding of the gaps.”
Insights on collaborating with AI labs and the role of feedback in agent development.
“Can anyone make their platform, like, in distribution of what the models are good at?”
Learn how the project builds upon existing frameworks while ensuring durability.
“Like we tried to actually just keep it as close to pie as possible.”
Discover why durability is crucial for maintaining user trust in agent operations.
“So I think it was probably bad timing, but we re-implemented pretty closely, I guess, what ended up being PyDurable.”
Understand the role of BullMQ in enhancing agent performance and scaling.
“And you can have scheduled tasks or like event trigger tasks, and they should just run and you should have confidence that they're actually going to accomplish what what you want them to.”
Explore the rationale behind choosing Pi and its distinct advantages.
“One of the things that you talked about earlier is the importance of kind of like being in distribution of what the models are good at.”
Learn about strategies to enhance agent performance and efficiency.
“A lot of them are not, probably not rocket science.”
Examine the security considerations in agent design and sandboxing choices.
“And do you do the agent in the sandbox or the agent connecting to the sandbox?”
Understand how to effectively monitor the new agent's performance in production.
“Just, you know, if this agent does gain significant scale, that is a part of the cost structure.”
Discover methods for analyzing agent traces and enhancing problem resolution.
“So there's a lot of different things that you can do with it.”
Learn about the implementation of memory and its significance for the agent.
“What I've been surprised by is, you know, if I take that trace and put it into Codex and just ask, like, can you figure out what went wrong here?”
Learn about the Knowledge Hub's functions and advantages for businesses.
Explore how memory updates work within the Knowledge Hub and user permissions.
“And I think if you just looked at historical data, it might be hard to extrapolate those two priorities.”
Understand the interaction between company-level and user-level memories.
“And you can think of it as like, you know, just two folders.”
Assess how outcome-oriented Clay is compared to previous systems.
“You could maybe try to give it more outcomes.”
Discuss the challenges of utilizing AI for company growth and the need for human insights.
“And so if I think about growing a company, one, it requires a lot of insights from the people on the ground, from the field, from the founder of a company.”
Delve into the evolving relationship between agent infrastructure and functionality.
“And so I don't know, like, I think probably this is, you know, still harness, you know, harness skill layer on top of the models.”
Discussion on how the definition and functionalities of agents are changing.
“This is why I'm not a marketer, but I do view this as just sort of the agent, you know, like, yeah.”
Exploration of pricing structures and strategies for AI agents.
“of what the pricing structure, like cost structure looks like.”
Insights into the fine-tuning of models and its impact on performance.
“My general sense here is, one, there are two outcomes you can try to drive from fine-tuning.”
Challenges and solutions related to the usability of CLI and user approvals.
“I feel quite strongly actually that it needs to be trained alongside the harness.”
Discussion on the evaluation of agents and testing in simulated environments.
“We effectively added a layer on top of the commands that we had, right?”

Get the gist, explore an idea or ask about something said.
On Max Agency, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00Jeff Barg: I think most people can do it, honestly. I don't think, like, there's really any secret sauce here. Today I'm talking to Jeff Barg, head of AI at Clay, a go-to-market platform that just rebuilt its agent around a coding agent. If you look at the bash commands that these agents write, they are extremely complex in ways that I, as an engineer, if I'm typing my terminal, would not do. We get into what Clay learned from post-training a model first. You can see the graphs kind of getting closer to Sonnet or Opus-level performance, but it wasn't a step-function difference. I think the learning there was just, hey, there's a lot of juice that we can squeeze out of just the harness engineering.
0:32He shares what surprised him about reading traces and understanding where the agent breaks down.
0:37Jeff Barg: If I take that trace and put it into Codex and just ask, can you figure out what went wrong here? It's actually really bad at doing this. We also get into how you give an agent enough business context to make good decisions. We actually started thinking about the memory piece before we started thinking about the agent piece. And Jeff explains how he thinks about pricing an agent that's built to drive outcomes. I think whatever pricing structure we align on for this agent, it will be wrong. I do think there's actually something quite nice about charging for tokens. Welcome to Max Agency, the podcast that goes deep into how the best agents are being built by builders like you.
1:11Thanks for being here, Jeff. I feel like you guys at Clay have been in the agent space for a while by anyone's standards. You guys just recently launched a new agent, which I think you're just calling Clay. Yep. What does that agent do? And maybe how is it different from previous agentic things you guys have launched?
1:31Jeff Barg: Yeah, well, first, thanks for having me. Excited to chat. Yeah, so we just launched our new agent. We're calling it Clay. So I would say a big change for the company, which is, you know, we want our agent to be the front door to all of the go-to-market tasks that you do in Clay. and I think the big shift here from previous attempts that we've had is we really want the agent to be, one, comprehensive of all the parts of our platform, so should do everything that you would need to do to do an end-to-end good market task, and really more outcome-oriented versus task-oriented. I think the first big attempt that we had at this was a product we launched last year called Sculptor.
2:10Jeff Barg: I think the big learning from Sculptor was we learned a lot about agent building. first off, this was like 2025. And I think, I don't know, we learned a lot since then. But from a product point of view, I think, you know, Sculptor was very good at doing individual tasks, but not like doing end-to-end good, like driving go-to-market outcomes on your behalf. What's an example of like a task versus an outcome? Clay is generally a tool for list building, data enrichment, and then orchestration on top. So into various go-to-market plays. I think the things that Sculptor was good at was like, individual granular things like, hey, I want to find a phone number for each person in this list.
2:51Jeff Barg: And Sculpture could do that for you. And I think, you know, that drove a lot of value. But what it was not good at was, hey, like, I'm trying to grow my business, and I want to launch a new outbound campaign against the customers that are best for me, right? It's like, that's actually like a very multifaceted thing, right? You have to figure out like, what are your best customers? Like, who are your best customers? How do you launch an outbound campaign? Like, how do you build a list of those customers? It's kind of like actually very multi-step and uses like probably five or six different parts of the Clay system.
3:27Interesting. So what does the UX look like now? Like you mentioned it's the front door and you mentioned it should do everything. What does it look like?
3:34Jeff Barg: The main UX is unsurprising. It is still just sort of blank text box, full screen now, like agent experience. Is that the homepage of Clay now? It is, yes. So if you come to Clay, it's like large text box. We do still have sort of list of resources, but it is getting smaller and smaller. It's just sort of the, it's funny. I feel like our agent thing, our agent chat box just keeps getting bigger and bigger. But yes. So you go to the chat bot and then what do you say? Do you say like run an outbound campaign for me? And does it still do it like a task? Is it like long running and proactive and will like check in on things?
4:12Like how do people interact with it once it's kicked off?
4:15Jeff Barg: You're exactly right. So you do still say things like, hey, build an outbound campaign for me. It'll do that. It'll build what we're calling artifacts, which are just like different parts of the clay system. So like workflows or an audience segment. So it'll build these artifacts for you. And it will schedule follow ups against those artifacts. So you can imagine like build me an outbound campaign. it will run it like a task and then schedule a follow-up that will check in on the results of that task over time. And the follow-up, is that just another agent run that you kick off yourself? That's right.
4:47Jeff Barg: Yeah. Yeah. I think we erred on the side of being overly transparent. Like we haven't gone the full instinct muse route where it's just sort of proactive behind the scenes. I was going to ask about that. Yeah. I mean, I feel like those types of proactive agents have come out a lot recently and people are paying a lot of attention to that. How much inspiration did you take from that? Yeah, I think we've been thinking about this since OpenClaw. So since December of last year. And yeah, I think we've taken inspiration from that for actually multiple parts of our product, both at the sort of global level, but also at the account level.
5:18Jeff Barg: You can imagine there's like for every account, an OpenClaw like thing that checks in on like, hey, should we be moving this deal forward or should we be, you know, reaching out to different people in the organization? So we actually launched a product early in the summer called Account Agents. somewhat similarly, like background proactive agents to suggest next best actions for each account. I want to come back to those proactive agents, but for the main agent, when it creates these like assets, how are those like kind of like visualized in the platform? What I mean by that is it sounds like it's creating other assets that you could also just create manually or create through any other methods.
5:52Do you basically kind of like lump all of this agent's work into some, I don't know, unit and say, yes, this is the agent, this is what it does, or are those like kind of dispersed just throughout the platform generally?
6:03Jeff Barg: They're currently dispersed throughout the platform if you just use the platform. But if you go to the agent, like you'll see them kind of bundled together. So like, you know, I go to an agent normally and I see like chat threads. Is it the same concept and like each thread has assets somehow associated with it? That's right. Yeah. Between kind of like chat and full blown UI, how do you think about the pros and cons of each one? I think directionally what we wanted to have this be was still task oriented in some ways. And the reason for that is actually we wanted the agent to be able to do some things that are budget scoped, sort of like permission scoped potentially, where you give it a task that's like, hey, you can spend up to this many credits or this much money and just figure out how to reach out to people.
6:49Jeff Barg: And so it's like, I don't think you can really do this in the product today, but directionally, that's where you want to go. So those like the design choices were sort of kind of directionally aligned with that. Yeah, what is the, what directionally, what's the North Star in putting everything together? I think a lot of it is actually modeled off of how we've seen RevOps team operate, which is they generally have a bunch of people who are trying different ideas. They have some kind of shared context. So some kind of like shared idea of like, hey, here are the total list of ideas we want to try.
7:17Jeff Barg: And then they have some form of like intake or idea generation. So I think directionally, it looks probably like some kind of task list, some kind of agents like adding and reading from that task list, and then agents just kind of picking away tasks like, yeah. Maybe switching a little bit from UX to now kind of like the core agent itself. How is it architected? How does it run? I think a big reason we got started on this was, one, I'd say the company has been going through a fairly big shift since the beginning of the year. So same time frame as OpenClaw and Opus 4.5, a lot of people started doing go-to-market tasks and clawed code.
7:55Jeff Barg: And I think for many reasons, right, like it's very expressive, like there's a lot of things it can do. It's kind of unbounded in terms of its solution space. And for a long time, Clay has been only web app, right? Like we owned the front door of kind of go-to-market tasks and felt fairly strongly that that was important strategically and also better for customers in a lot of ways. since this year, I think we've realized like, hey, there's like, we should really be playing in the coding agent space and really participate in that ecosystem. And so all this is to say, we actually spent a lot of time, one, making the platform accessible to agents.
8:32Jeff Barg: So all of the things that the agent can do was made possible by, you know, a six month investment in making our campaigns product have, you know, agent friendly primitives to be able to build, edit, run campaigns same with our workflows product and audiences product which is for like our good market context layer and so i'd say that work culminated over the summer we launched a cli as a side note i think uh it was in some ways unsurprising to me but also shocking to me how much people loved the cli and i think like a group of people that you might not think of as you know probably five years ago wouldn't really know what a cli is um you have like these rev ops teams uh or, you know, maybe like very AI forward, but like go to market builder teams that, yeah, I mean, they loved the CLI.
9:22Jeff Barg: And I think, you know, so this was just sort of the natural combination of building the product around these agent friendly primitives, launched this, a ton of got a ton of traction for that. And then I think the logical next step was, you know, bringing this back into the product. I think there were actually a lot of people who would have gotten value out of the CLI, But to the earlier point, you know, even though people are quite excited about the CLI, there are still a big there's still a big segment of users that don't know what a CLI is and don't really want to be in cloud code. So, yeah, so I think we really wanted to bring that experience, like this kind of powerful, expressive, all encompassing, agentic way of doing go to market back into the product.
10:05Jeff Barg: And there's a ton of advantages of doing that in product as well, because you can integrate it directly, you know, to the points we talked about earlier, right? Like having artifacts, being able to hand off between manual creation and agentic creation, having things like, you know, resource level budgeting, which wouldn't be possible in quad code. All of these things were sort of, hey, we can actually build a really customer centric experience. I don't know. I think there's probably many different strategies. Everyone's trying to figure this out in some ways. I'm curious to get your thoughts on that.
10:36Jeff Barg: But yeah, I think for us, like we saw a really good path to actually do this in a way that would be really customer friendly. Okay, I want to get back to the agent harness, but maybe sidebar on this, because I think everyone is talking about this. I'm really interested in one thing you said about making Clay agent-friendly and bringing agent-friendly primitives in. For a company that wants to do that, what would you recommend? It sounds like it ended up culminating in a CLI, but what are the steps that it took to get there? And what made it a good CLI? What were maybe some things you tried that didn't work?
11:06Jeff Barg: I have so many thoughts because there's probably at least three dimensions to this question, right? I think it was, it was a big challenge organizationally, right? Because it's every single team had to figure out, hey, we've been building this product for maybe 12 to 18 months. And all of a sudden, we have to rethink it from first principles to think like, hey, what should the commands be for agents? Like, there's all the organizational challenges. I think there's all the sort of skill set challenges of, you know, many teams who are good at building a sequencer product might not be great at, you know, building evals for a sequencer CLI or, sequencer tool set.
11:40Jeff Barg: To that point, I think evals are a big challenge, right? Just figuring out what are the outcomes we want to drive and what are the ideal behaviors that we want to have the agents do. So maybe a question on that. So we, you know, at LinkedIn, we obviously think a ton about evals. I think most of the time when we talk to people about it, it's about evals for kind of like their first party agent. But we're starting to talk to people more and more about evals for kind of like skills, MCPs, CLIs. What do those evals look like for you guys? What advice would you have for teams trying to build those?
12:08Jeff Barg: I think this was actually another advantage of building a first-party agent was it made it way easier to build these evals in a way that it's actually a great flywheel. So actually first, just to set the stage, our production agent uses our CLI. And that was like a very conscious choice to one, keep these experiences, you know, to have as much parity as possible. You know, we're actually, we want external agents to have just as much control and expressibility as our internal agent. And I think all of the evals now, like all of the hill climbing we've done on our first party agent has fed back into the CLI.
12:42Jeff Barg: And I think that's been a really good flywheel. Yeah, in terms of the evals, I mean, there's been a progression there, I'd say. We started out, I think, really thinking about evals from a capability standpoint, right? So we had a bunch of different buckets of evals for can it, you know, build, run, edit campaigns, right? Sort of just like very rote, you know, testing the capabilities. I think then we shifted towards, I'd say like levels of orchestration, right? So sort of like, hey, here's a big bucket of tasks for which like, if we get pretty good coverage, we think the agent will be, you know, ready to get to some milestone.
13:21Jeff Barg: So it's like we had an alpha launch, we had a beta launch and kind of bucketing evals like that. And I think where we're landing is actually going back to user journey bucketing, right? So it's like, hey, there are users who really want to do inbound lead routing. And then within that, there's probably five or six different subtasks that a user would come in, use the agent, have these experiences or like outcomes you want to drive. And then having evals that closely align to that. And so I think that's been the evolution of these things. The middle one was probably the most awkward because it was very helpful for driving.
13:54Jeff Barg: We had our user conference. And so we really wanted to, you know, feel confident about, well, if we get to, you know, this date, we feel like we'll be ready to launch. and so helpful from that standpoint, but very awkward from the standpoint of like, the metrics were almost meaningless. You get to 80 % coverage on something. What does that, you can't market that to users, right? I guess you can, you can put it on sort of a chart, but I always as a consumer see the charts that like OpenAI or Anthropic put out and was a bit confused, you know, or at least like I don't feel like that actually translates necessarily to the experience of using some of these agents.
14:30And so do you eval just kind of like your first party agent or do you eval the CLI in some way separately from that?
14:38Jeff Barg: We have done some basically harness comparisons. So, you know, just using the CLI in Claude versus the first party agent. I think that's gone through some iterations as well. I'm curious to get your thoughts here because, yeah, I think to your point, like this is feels very new. It feels like people are trying to figure this out. Yeah, I mean, I think like, so we have an agent as well that's very similar in some sense, an engine that runs in the background, looks at traces. It also uses our CLI, the same CLI that's exposed. We also do kind of like harness comparisons. I feel like we do that more to like make sure that we're performing better than the other harnesses rather than to like make sure that that CLI performs well in those harnesses.
15:22But I don't think it would be crazy for us to, you know, start testing them in CloudCode, not to, you know, make sure we're better than CloudCode, but to make sure that CloudCode can use it well. But I don't think we've done that to date. It sounds like you guys are kind of similar.
15:36Jeff Barg: Kind of similar. I think it was very much kind of a comparison to the other harnesses, not like trying to, you know, get the other harnesses to be great. And philosophically, I think we do want the CLI to perform well in all environments, right? So yeah, not crazy, but probably not a priority for us right now. The other thing in this vein that I wanted to touch on is basically, yeah, like making Clay available to other agents versus building a first party agent as a strategic choice. And I think this is, again, something that basically every company, including us, are also thinking about. Again, we think about this a ton for Engine.
16:12I think there's another aspect as well which is like I think we can I think we can get better performance out of kind of like engine compared to cloud code you could argue that's through kind of like skills and sub-agents and prompts again we use the same CLI so that's not really a factor but you could argue you could bundle those all up into a plugin or something and bring that to cloud code but it feels a little bit harder to control in some ways and I want to you know dive in more into how you build the engine but I'm curious yeah for this like just general problem of you know third party versus first party, how you guys have thought about or advice you'd have for companies, maybe when to do it, when not to do it.
16:52I don't know.
16:52Jeff Barg: I have a lot of thoughts here. I think, you know, all of those reasons are definitely, I think, the, you know, the primary reason why we guide people towards, you know, the agent versus the CLI. I do think there are all sorts of other idiosyncratic reasons, right? Like, you know, coding agents perform differently in different environments. For a while, we didn't support Windows for a CLI, right? Sort of like these weird little like easy ways to guide people towards the agent where they might not use it otherwise. I think the other big reason, which is maybe not true today, but I think will be true in the future, is there are a lot of orchestration benefits of having a first-party agent.
17:31Jeff Barg: Like if you think about go-to-market, a lot of go-to-market is event-based, right? So you want to kick off the agent when someone replies to your campaign so that you can actually get an email back right away. and like that time to response is actually really important. You know, maybe there will be standards that evolve here, right? I think, you know, there's some interesting things in MCP or, you know, there are potentially ways that this will evolve to be, you know, external agents as well. But definitely today, I think there's a big advantage of having this in-house and having, you know, good control over that orchestration layer.
18:01Jeff Barg: I think also there are just a lot of advantages of that orchestration with respect to costs. If I think about Clay, you know, Clay spins up, you know there's this main agent thread but the main agent spins up a lot of other agents typically right so uh if i think about you know building an outbound campaign usually you'll find a list of leads and then you'll spin up an agent for each to you know research qualify do copywriting for those uh leads and the difference in cost between a good version of that and a bad version of that can be you know orders of magnitude um and so i think that having like very tight control over that orchestration layer is really important yeah i think both those things are great points another kind of like similarly like event-based orchestration logic which you know we only do a little bit today but i could see being in the future is things like memory and background and dreaming and things like that i think that you know memory will look different per application and so being able to control that makes a ton of sense totally agree with what you're saying around like optimizing for cost maybe maybe one last question on this and then i want to dive more into some of the technical details but like when you're comparing your first party agent to cloud code codex between kind of like cost accuracy and latency where are the places where you think it like really shines yeah i mean i'd say we really think about latency and cost right um i think there's a lot of reasons as well that you know a plugin so just a cli plus some skill files um they might work really well but can be very token inefficient i think can be very turn inefficient right if you're reading a bunch of skill files and then doing something it's potentially more inefficient than if you're on a surface just pre-loading those skills into the system prompt or you know doing all these sort of micro optimizations that are pretty simple but do add up in sort of you know in the aggregate so yes i i think it's it's much more efficient it's we've actually heard that feedback is that people use it in cloud code and it works great but it's pretty slow and then if they do in the agent it's you know we at least have visibility of what the latency cost trade-offs are so we can drive those outcomes much better.
20:04I want to come back in a second to how technically you guys drive those outcomes, but maybe stepping back and setting the stage. It sounds like what you're getting at is under the hood. It's basically a coding agent that you guys are running, but I'd be curious to hear more about the technical details of what that looks like.
20:21Jeff Barg: Actually rewinding a bit to the earlier attempts that we had at this. We launched Sculptor last year, which in many ways, I think, the goals of that project were quite similar to this. You know, we had a lot of learnings from Sculptor. You know, Sculptor was quite good, again, at sort of task-level outcomes, but not really great at these longer horizon tasks. And I think some of the learnings were, one, we really needed to build around a coding agent. I think that's what the models... Why? Yeah. Again, not obvious as a conclusion, but one, I do think there's something weirdly generalizable about these coding agents, right, in ways that are somewhat surprising.
20:59Jeff Barg: Some of this, I think we just observed from people doing go-to-market tasks in CloudCode, again, without actually any of the scaffolding. Like pre-KlyCLI, people were using CloudCode to build go-to-market plays in, again, ways that were surprising to me. Like this is not non-intuitive. So one was that just on a, you know, engineering level, Bash is really, you know, it's Turing complete. You can compose a bunch of things together. If you look at the Bash commands that these agents write, they are extremely complex it's you know can be you know 100 plus lines of bash that use you know conditionals for loops all these things to compose different cli commands together again in ways that i as a an engineer if i'm typing my terminal would not do and so um there's all of those benefits and i think all of just the post training on the model seems to have benefited those those harnesses And actually just another tangent here.
21:54Jeff Barg: When we launched Sculptor, I think we had a pretty good understanding of the gaps. It was like it couldn't do these long horizon tasks. It wasn't necessarily good at just some of the composability aspects. It would call a couple of tools and then maybe stop. And so one of the things that we had thought was maybe our environment is just pretty idiosyncratic. It's pretty, you know, distinct from, you know, what these models are trained on. And so we actually took a tangent into post-training at the end of last year to try to, you know, sort of like, hey, our model or our agent falls short in a couple of these ways.
22:32Jeff Barg: Is it at the model layer, the harness layer, you know, kind of, should we just hill climb on the harness we have? Should we like really reconstruct everything? and you candidly I think the model training that we did you know you could see the graphs kind of getting closer to you know sonnet or opus level performance uh you could kind of see all these things but it wasn't a step function difference and I think the learning there was just hey there's a lot of juice that we can squeeze out of just the harness engineering and it's probably like the best model would not, you know, make a step function improvement in this.
23:10Jeff Barg: It's just like, hey, we're not giving it actually the express ability to do the things that we needed to do. So you guys, as an agent engineering team, you talk about like step function improvements. How often are you looking for step function improvements versus like just iterating on like, you know, like smaller improvements on a base thing? Is it every month where you're like, hey, we got to make a step function? Is every six months, every year? Oh, I mean, it's hard. I think it's probably, again, specific to each kind of shift change. I'd say in practice over the past 12 months, I can think of probably three moments at least, right?
23:45Jeff Barg: There was like the platform re-engineering around the CLI. There was sort of this coding agent shift. So building our new agent around a coding agent. And then I would say the other big shift was actually, this is maybe part of the platform piece but uh we redesigned a lot of our platform to have dsls to represent them um i'm thinking specifically about our search platform so we have a a company and contacts data set of you know 900 million contacts 40 million companies and the way we had represented this for sculptor so a year ago was like we have all these filters uh basically have a structured output that like you know builds a bunch of filters and composes them together worked totally fine but again sort of like was really good for the easy cases not good at the like sort of long tail of like hey i want uh startups in new york that have raised under this amount of money that you know have been mentioned in the new york time sort of like these more complex queries um what we ended up doing is we built this dsl for what we call uh our search query language but it is like a sequel it is a SQL-like variant to construct companies' contacts queries.
24:57That makes sense. I think we're doing something similar with Langsmith now where we want people to carry traces and we're creating a SQL-like variant on top of that to make it easier and easier because models are really good at that.
25:08Jeff Barg: This was our thinking at the time was just there's what the frontier models are good at and then there's kind of like what your application looks like. And you really have two options. It's like train it on the things that you have or take the things that you have and make it closer to what these models are trained on. So, you know, I think models have seen a lot of SQL code and they know how to generate it. And it sounds like you guys are going pretty hard in that direction of making it look like what the models are good at. Yes, I actually think they're not exclusionary. Like SQL has a ton of benefits beyond just being what the models are trained on, right?
25:41Jeff Barg: It's also composable. It's also sort of, you know, modular. You have all these nice properties about it. So I don't think these things are mutually exclusive. But yes, I think the most juice we could squeeze was out of the sort of harness or, you know, DSL layer. I see that you guys have a pretty good relationship with the labs. How, like, can anyone do this? Can anyone make their platform, like, in distribution of what the models are good at? Or do you have to really kind of, like, be close with the labs and iterate with them? I think most people can do it, honestly. You know, we, to your point, like, we talk to the labs a lot.
26:18Jeff Barg: I think they offer a lot of guidance on, you know, how their models were trained and, you know, how to, you know, best use the models for our use cases. I'd say also, you know, we, I'm sure there's different philosophical approaches to this. Like, we have no problem sharing evals or sort of our performance of our agent back to the labs. I don't think that's necessarily true for every company. You can probably build an agent, you know, absent of that. That's really good. I don't think there's really any secret sauce here. In terms of the harness itself, what does the harness look like? We started with off-the-shelf Pi.
26:56Jeff Barg: And I think for a variety of reasons. One, on the point of latency and costs, I think it's much more lightweight than something like CloudCode or Codex. And I think what we liked was it was very extensible and understandable. like there was not a ton of you know crazy tricks that they were doing um it was just fairly simple software that we could start customizing over time um i think it really helped us get started quickly i don't think we um we considered briefly building a harness from scratch and you know certainly we'd had experience doing that but i think there was something nice about you know just getting something started that actually I was familiar with as a developer that, you know, seemed to be performing well on benchmarks for quality and cost compared to, you know, the more heavyweight harnesses.
27:51Jeff Barg: So, yeah, it got started there. And then I think over time it's deviated a bit where it's, you know, we've started to build on top of it. But as much as possible, I think we tried to build on the plugin system. Like we tried to actually just keep it as close to pie as possible. Yeah, I was going to ask, when you say you build on top of it, what does that mean? Does that mean the plugin system? Does that mean you've forked Pi and are making changes out like that level? We haven't forked Pi, so it is mostly at the plugin level. There are pieces of it that is funny. I think for one of the technical challenges for building this project was the durability piece, right?
28:25Jeff Barg: So, you know, actually, we have a host of fleets that run these agents. if one of the hosts goes down for whatever reason, if we need to do a deployment, these azins should continue running because maybe they're running for 30 minutes or more. So I think it was probably bad timing, but we re-implemented pretty closely, I guess, what ended up being PyDurable. And so there were sort of things like that that ended up being on top that we implemented. But again, the core harness is still off-the-shelf Py. Could you maybe talk about durability and why it's so important? And when you implemented it, did you use something like a temporal that's out there or did you build more natively?
29:06Well, first off, on durability, very important, I think.
29:11Jeff Barg: Mostly from a user perspective, right? Like it's really bad if these agents run for 30 minutes and then they all of a sudden just, I don't know, crash or like you see some like unexpected error. And so there's probably all the normal reasons of, you know, just customer trust. I think the other thing is just this is an advantage of actually having a first party agent. I don't know, you don't have to keep your laptop open, like as your clay workflow is running, like you can just toss off a task. And it will run for you, right? And you can have scheduled tasks or like event trigger tasks, and they should just run and you should have confidence that they're actually going to accomplish what what you want them to.
29:48Jeff Barg: Yeah, in terms of building it, we built on top of BullMQ. For better or worse, I think we've we've had pretty good experience doing this. Actually, many of our agents are built on top of BullMQ. And yeah, we've stretched that pretty far. I think for some of our other agents that run at extremely high scale, we're still running them mostly on BullMQ, and it's mostly working. One of the things that you talked about earlier is the importance of kind of like being in distribution of what the models are good at. I think there is an argument from the model labs that, hey, you know, we're going to build harnesses that are going to be best for the models.
30:28And you guys chose to go with Pi, which is not from a model lab. How do you think about that argument? I don't have a concrete view for our agent specifically,
30:37Jeff Barg: but it has been surprising to see, I think, how not true this has been for other domains. You know, for coding, I use Codex, I use Pi, and there are many times where I prefer what Pi does as a coding agent. And they are distinct. You can actually feel it as a user. And so that's been interesting. You know, from first principles, I actually really buy that argument from the labs. If I look empirically at what's happened, it doesn't seem like that's true. So there's some disconnect there that I don't have a good thesis on why that disconnect exists, but it certainly seems to have borne out. In terms of some of the context engineering that you've done on top of the base harness in order to get these like latency and performance and cost improvements, what have those things been?
31:24Jeff Barg: A lot of them are not, probably not rocket science. Like I think a lot of it is low hanging fruit around, you know, things like what things exist in skill files that could be loaded into the system prompt. Like what are things that, you know, how should we load skills into the system? How many skills does the agent have? I don't know the exact number, but it's on the order of 20. How many tools does it have? Probably around 8 to 10. I don't know the exact percentage here, but it is 85 % plus the bash tool and probably 5%, 5 % read and write. And most of the bash tool is calling the CLI or do you find that actually does a bunch of other like bash slash coding related things?
32:08Jeff Barg: It does a bunch of stuff. Yeah, I think we see it writing Python files. We see it writing bash scripts, like writing the output of the CLI to files and then doing the string manipulation on the file. So we see it doing a ton of stuff. What are some of the like plugins that you've written that do things beyond skills or tools? Yeah, a lot of it has been the actually the UX integration. So there may be like two buckets of things that we have. So one is around things that don't apply to the CLI, but do apply to the agent. So things like, hey, navigating to an artifact. So if you've built a workflow, actually just having the user like see the workflow as it's running or like see the structure of the workflow as it's being built.
32:50Jeff Barg: And there's a couple of tools in that vein. The other is permissions and approvals. We have sort of specific approvals for, you know, some of the CLI commands in Clay. And we really want those to be deterministically approved. So there's a set of tools around that. Coding agents need to write an execute code. How do you guys do sandboxes? What is the infrastructure behind them? And do you do the agent in the sandbox or the agent connecting to the sandbox? when we started this project, I think my, I was very excited about it. My biggest concern was the security boundaries of this, right? I think, uh, one, the security ecosystem, we can maybe talk about this, but the security ecosystem right now is crazy.
33:35Jeff Barg: Uh, it's a very interesting and scary time, I think, to be building software. But, you know, I think one One of the things we really wanted was a highly secure agent. And I think that led to a bunch of different design decisions around this. One of the big ones was kind of a split, having the agent run on, not in the sandbox, right? So the agent's running somewhere. It is executing, you know, bash commands or read commands on the sandbox that we're using. We thought a lot about actually, you know, did we need a sandbox? I think there's a lot of interesting projects around sort of simulated sandboxes or I'm thinking of just Bash from Vercel.
34:18Jeff Barg: I think Monty from Pydantic is sort of in this vein. Maybe I'm wrong about this, but there's a couple of these projects that - QuickJS. So that's what we use in Deep Agents is QuickJS. It's very similar like emulator of a language or something like that. Yeah, I think like these make a lot of sense. You know, I don't actually have a, you know, long-term view on this. I think in the short term, you really, we really just wanted full parity with what coding agents were. And again, it was sort of, there was so much uncertainty. Like, I think especially when you're building a new agent, there's a lot of uncertainty around like, what does great look like?
34:53Jeff Barg: And then what compromises can you take to kind of keep great, but, you know, do have some other nice properties. And so we started just being like, let's take a coding agent, put it in a sandbox. We know that that works really well. Just from like our user's experience, using the CLI. We do have an actual, you know, firecracker isolated sandbox that, you know, an agent outside of the sandbox is operating against. Are you building that sandbox infrastructure from scratch? Are you using a provider? How did you decide that? We're using a provider. We're currently using Daytona. I mean, the sandbox ecosystem is also very, oh, it's really hard to make that choice, I think.
35:32Jeff Barg: I think it's, oh, it's, it's really interesting talking to a lot of these different sandbox providers i think a lot of the features feel like they are somewhat commoditized right like there's um there's certainly different choices you can make around you know yeah is it is it a vm isolate like you know kind of what are what are the security boundaries but beyond that right you kind of have some costs like uh some pricing differentials right some uh some of the schemes are different than the others but the decisions are mostly around the trust that you have in the provider again like the only consideration I have really is the security boundaries.
36:10Jeff Barg: Pricing certainly is a consideration. Just, you know, if this agent does gain significant scale, that is a part of the cost structure. But, oh man, it's just really hard to kind of make these decisions because you are basically trusting whose security team is most on top of it. New Clay agent is broader than previous ones. How does that change how you think about monitoring it or keeping tabs on what customers are using it for, how it's doing in production? Yes. One, I think it's a challenge. I think it's a challenge because to your point, it is extremely broad intentionally. So there's a lot of different things that you can do with it.
36:51Jeff Barg: And I think we're the most confident that we have been on, it will actually be able to do, you know, a wide range of tasks. I think figuring out what people, you know, we have a thesis on, you know, what the tasks it is best at are and, you know, how we want people to be guided to using the agent. But then there's also the reality of how people are using it. And these things are not exactly the same. And so the agent's been live now for about three weeks. We do see this discrepancy of like what we thought people would use this for and then, you know, what people are using it for. I think like trying to figure that out.
37:28Jeff Barg: It is a combination of looking at traces. It's a combination of talking to people about how they view the agent, how they are trying to use clay. And I think the thing that we are trying to institutionalize more is actually, you know, categorizing traces at a, you know, from online evals or kind of a, just, you know, actually try to bucket these and quantify it a bit more. What are some things we can add into Langsmith to make it easier for you guys to do One of the things that I, well, one, we do use Engine. And I think Engine has actually really, has been really helpful, I think, especially at the earliest stages of, you know, just seeing problems in the agent and flagging those to us in ways that were, like, very actionable.
38:15Jeff Barg: I think I've been really impressed by just how little noise there is. Like usually when engine flags something, it's usually actually something that we need to take action on, which is great. I think one of the things that I've been surprised by is when we know there's a problem with the trace and we look at the trace, usually the trace is quite long now. So there's like hundreds of tool calls. You click into even an individual tool. It's like a bash command with hundreds of lines. And so, you know, it's easy to sort of understand the structure of the trace, but hard to understand, like, actually, where did the agent fall down and what was the breakdown in process?
38:57Jeff Barg: What I've been surprised by is, you know, if I take that trace and put it into Codex and just ask, like, can you figure out what went wrong here? It's actually really bad at doing this. Yeah. I thought you were going to say it was really good at doing that. It's really bad at doing it. It's surprisingly bad at doing this. if you've got to do things like first go through each bash command and summarize what the bash command is doing it can do it pretty well so there's probably some sort of a skill file that's like i don't know a trace analysis that you can you can come to i think poly was probably an attempt to this and i feel like it's actually great you know this is kind of something where i wish langsmith just just had something here it's like you know on a one-off basis like engine is good for sort of the aggregate level just on a one-off trace basis like what's what's going wrong we talked about earlier about kind of like step function changes i think right now a lot of the usage we see from engine is to your point like very actionable things i feel like those probably aren't the step function changes are are there things that you think um you know if we if we abstract engine to kind of like an intelligence layer over traces you've got all of this data but how do you take action are there things that would be like very helpful in like figuring out these step function changes?
40:09Or are those really coming from like just like industry and broader product things? And it's kind of hard to put that in an intelligence layer over traces.
40:18Jeff Barg: One of the things that, you know, I think is really hard about the step function changes is kind of quantifying like what they can help with, right? Like you don't want to invest time in a step function change if it only affects like 2 % of traces. And so kind of the work there that I can imagine is you know failure mode clustering and then quantification right so like hey like the agent is failing for workflow creation 20 of the time and if you look at why it's failing on workflow creation it's because it is trying to edit the workflow and it's failing on validation steps after right i'm just giving a getting an example here but those are things that those are really hard won insights right like we have to really kind of i don't know just go through a you know, have a lot of context across all of these to think like, oh, yeah, like that's related to this.
41:10I think this is something that Engine could, you know, if it could offer those insights, that would be very helpful. Let's talk about memory a little bit. Does Clay have memory in it?
41:21Jeff Barg: Yes. And we actually spent a lot of time on thinking about this. I actually think like we, if I think about the history of this project, we actually started thinking about the memory piece before we started thinking about the agent piece. In go-to-market context is very important, right? Both for, it really affects kind of everything, right? Like you have a go-to-market strategy, you have a customer that you're trying to sell to, you have things that have worked for finding and closing those customers. Yeah, being able to share those insights across the different things that you're doing in your go-to-market motion is really critical, right?
41:57Jeff Barg: Like I think if you don't do this, So there's, you know, you're just actually losing out on a lot of benefits. So we launched actually alongside the agent, something we're calling Knowledge Hub. It is effectively a, I don't know the best way to describe this, but it's kind of like a wiki style understanding of your business and, you know, what your business does, who you sell to, and different attributes that you have, you know, about your company. this is both auto populated and then also self-updating right so um you sign up for clay we'll actually build the knowledge hub for you based on like what we think we know about your business um and then as you do more things in clay it will actually improve over time right so uh i'll give an example which is i was pleasantly surprised to see this but um i think one is like it's hard to do this right it's hard to actually understand what your go-to-market priorities are over time um you know clay had most of its success that it start um selling to plg right so it was like a lot of uh smaller companies uh that were just getting started and wanted to build lead lists and um we still do really really well there and that business is you know growing very fast it's you know it's like a um a core part of the business but strategically we are you know we want to expand geographically so we're expanding into europe uh we want to sell to enterprises as well And I think if you just looked at historical data, it might be hard to extrapolate those two priorities.
43:28Jeff Barg: But yeah, it was really fun. We launched this Knowledge Hub, and I think a week later, because people were using Clay to target European companies, there was an update in our Knowledge Hub. It was like, hey, we think your ICP is European B2B SaaS companies. It was like, oh, yeah. What's the UX for that memory update? Does an admin have to approve it or does it just automatically go in? We started being quite conservative here. So an admin has to approve it. I don't want to claim too much credit for this, but I think there was skepticism on the team around like, I think everyone has probably had a bad experience with memory at one point or another with some agent, right?
44:06Jeff Barg: Like I'm sometimes very frustrated by cloud code memory. Like it just will, it thinks you want to do something one way and like, no, I don't want to do that. It's hard to correct it. and you know I've also seen really good memory systems I think like OpenClaw and probably some of the new you know Muse Instinct style agents are really good at this but yeah so I think you know we started off being very conservative admins have to approve it we'll offer suggestions the agent can do it for you so if you explicitly tell the agent like just update my memory you know update Knowledge Hub it will do it for you but otherwise it's suggestions only.
44:42Is there one memory per company or are there different hubs, brains at an org level, user level, task level?
44:52Jeff Barg: Right now there are two levels. So there is the company level knowledge hub and then there is agent level memory, which is user scoped. What does that mean, agent level memory that is user scoped? So does that mean like if I, so we talk about clay, I go in, I start a thread, that thread has its own memory or like every thread that I kick off shares a user scope memory for me? Every thread you kick off. Yeah. And you can think of it as like, you know, just two folders. One is like read only. That is the company level memory. And then one which is just read and write for the for the agent across all threads.
45:29When the company one is getting updated in the background, what does it look like? or sorry, what does it look at? Will it look at like my user level interactions? And if so, like how do you decide what is user? What is company there?
45:45Jeff Barg: This is, I think, an area of very active development for us. I think in theory, it's like actually most things should be considered, right? So we definitely wanted to prioritize things that were explicitly offered by the user. So as a user, you can come in and you can upload documents. You can be like, hey, here's my sales deck where here's like, here's my go-to-market strategy doc. And we should be able to extract those insights and put them into your knowledge hub. We do care about user-level suggestions and we have some process that basically takes all of these user-scoped memories and tries to extract what do we think is global to the workspace.
46:25It sounds like right now the agents in Clay are kind of like single player. Like there's one for me and it has a different memory than for you and we can't share threads and things like that. How high of a priority is it to make it multiplayer? And there's an agent that both of us can talk to and influence.
46:42Jeff Barg: Yeah. It's pretty high priority, I'd say. I think, again, just I'm thinking about our own go-to-market ops team. You know, they operate in Slack. They have a shared linear board. And that's kind of how they build together. and you can imagine that, hey, like the agent should be able to do one, operate in the same surfaces. So bringing it to Slack, bringing it to, you know, things like iMessage even, I think those are things that we want to do. But also, yeah, I think just trying to emulate that workflow as much as possible. So giving them kind of shared task context, bringing them into surfaces where you can collaborate both with other people, but also other agents.
47:22Jeff Barg: Yeah, pretty high priority for us. I think we'll do something here quite soon. Do you think there's like one, Clay, when you go to this multiplayer world, is there one clay agent or do you kind of like anthropomorphize them into Joe and Jim and Tom and things like that? One, this is such a hard design question. And actually my favorite question to ask our designers is to ask about, hey, if it were in Slack, what does that look like? Because I think that really clarifies it, right? Like, is it clay? Is it like, you know, what Dots has done where it's like, it's your dot? You know, I think actually a lot of the technical decisions kind of fall out from that UX.
47:59Jeff Barg: My guess is that it will look something like Clay globally and that you won't have sort of these distinctions. There might be some distinctions around, well, there certainly will be distinctions around permissions for each user, but there might be distinction around like the use case and rule. Like I'm thinking about sales reps do very different things with Clay than someone in RevOps who's like building these sort of more good market systems. On a scale of one to 10, Like how outcome oriented is clay right now? How outcome oriented was something like Sculptor? The reality was Sculptor was probably very low on this.
48:37Jeff Barg: So, you know, three out of 10. You could maybe try to give it more outcomes. You could brainstorm with it. It was actually fairly good at table analysis. So you could kind of, if you wanted to extract insights of plays that you wanted to run, might be pretty good, pretty decent at that. but certainly you couldn't reliably delegate responsibilities to it. Like it was just not in the realm of capabilities. I think now we're much higher up that list. Like I feel pretty confident that, you know, our own go-to-market ops team can just give it pretty open-ended tasks and have it generate something pretty decent.
49:15Jeff Barg: The tasks I think customers have been most excited by are actually generating things like account scoring. So you can be like, hey, look at all of the deals I've closed, all of the deals I've tried to close but have lost, distinguish between these, propose a scoring algorithm to predict whether someone's going to close, and actually backtest it, and then backtest it with additional signals or data points that you could add to it. So actually quite expansive and comprehensive, and a real go-to-market task that would drive ROI. Pretty much without fail, this delivers something pretty reasonable and within the scope of under 30 minutes for a demo call.
49:59Jeff Barg: So I think enterprise customers are really excited about that. It's pretty powerful. So I don't know. I still think there's a long way to go. I don't think this is something where if I were running a company, I could feel like this is my AI head of growth and it just grows the company and I don't have to think about it. So I'd say we're now probably, if I have to be honest with ourselves, about a seven out of 10. What's the blocker for getting to that AI head of growth? There's certainly tactical level things that we're working on. One, I think growth is actually quite holistic. And so if I think about growing a company, one, it requires a lot of insights from the people on the ground, from the field, from the founder of a company.
50:41Jeff Barg: I'm sure you experienced this where there's probably, you know, so many different ways to grow Langchain and actually like requires a lot of discernment to be like, this is what we're going to do. And some of that is actually choice, right? Like it's actually like, this is the company I want to build. If I think about Clay's mission, it's to grow every company to its full potential. To its full potential is actually a really, really important part of that mission. So yeah, we really think there's like, you know insights are required to kind of grow the business it's not like this can just like run on its own yeah so i think that that's you know that that's a big piece of it is it's not uh it's not actually you do actually need to be quite involved here if i think about that style of kind of like give it a very like grow the business or i i think like um i think i saw one launch from a startup that was around like give it a metric to optimize like arr or something like that.
51:38I think there's a few worries I've there. I'm curious for your takes on other concerns or how you think about this. But like, I think it's very rare that you can maybe like quantify everything into a single metric. And so over optimizing for a single metric seems like you wouldn't want to do that. You'd want to have it. And maybe this is solved by having it have kind of like some constitutional guardrails where it can like recognize that the metric isn't everything and suggest corrections. And then the other thing is like something you said made me think about it, which is like, I think the power of agents is they can do a bajillion things.
52:09But like, I actually think to build something, there is value in not doing a bajillion things. And instead, like, you know, if it's not working initially, don't get scatterbrained, like double down. And again, maybe that's prompting, maybe that's some constitution. I'm curious, like, if there's other things like this that you think could be blockers for this, like, yeah, like give it a very high level thing to do and just let it run.
52:32Jeff Barg: First off, I think that's that is really important. Like you do want to have guardrails. I think for this to work, you have to really trust it, right? Like you do actually need it to have some intuition around, yes, like this isn't working, but it will work if we double down on it. And I think that's really hard. And I think the reality is, you know, I haven't seen many products do that well. But, you know, I think directionally there's, that feels like a tractable problem. Do you think it will be solved at the model layer or at the not model layer? I mean, I think a lot of this is very domain specific, right?
53:06Jeff Barg: Like I, um, I'm quite bullish on vertical AI companies. I'm, of course, very biased. But, uh, you know, I think a lot of this is understanding your business, understanding the context about what has worked, and then having the intuition around, yes, like, these are the types of plays that might work for, you know, finding more customers that would want this product. And so I don't know, like, I think probably this is, you know, still harness, you know, harness skill layer on top of the models. I don't think, you know, it's necessarily a model. Like, I don't think GPT-7 would just solve this out of the box.
53:45How close do you think the infrastructure and harness are becoming related? What I mean by that is I feel like a lot of early frameworks for sure and even harnesses were pretty unopinionated about the infrastructure that they interacted with. And now we talk about durability. We talk about things like scheduling, which can only really be done if you know how it's deployed. Memory, I think right now is a separate system for you guys, but there's still some kind of like interplay there. One of the things that we've seen this also pop up a little bit is like asynchronous subagents and multi-agent things.
54:18you kind of have to know where they're deployed. And maybe there's like a protocol that pops up for that, but I still don't think there's a really... So I'm curious, like how tightly coupled you see kind of like the infrastructure, not of everything, obviously, but the infrastructure of this core, like agent server being to the harness, whether it's very tightly coupled now, still pretty loose. And whether you think, how much do you think that's going to change? Sandboxes are another example. Like a lot of these will write and run code. Right now, those are a completely separate thing. Yeah, I think right now these things are fairly decoupled, but certainly getting more coupled over time.
54:54Jeff Barg: Like, I think that you're totally right on the direction here. If I think about even today with sandboxes, right, like you have a lot of implicit coupling where the sandbox has a file system. The file system is, you know, it is a store of persistence for the agent that, you know, is actually, you might not think about it. But like, yeah, if you talk to the agent two days later, you probably want it to have the same files. And, you know, so you have this kind of very implicit coupling. The other thing that I think you're totally right about, and I think where there will be more coupling over time, is on the orchestration and scheduling layer.
55:33Jeff Barg: Right. Like the agent actually probably should have pretty tight awareness of how it's being scheduled, like all of the follow ups, like be able to change those things. like actually very directly. So yeah, we don't really do this today, but I think that will be. What would you call this new thing? Because like, I feel like it's more than a harness. Agent server, agent OS, agent kernel. To be clear, I'm asking you to do marketing work for us. Yeah, yeah, yeah. I don't know, mega harnesses. This is why I'm not a marketer, but I do view this as just sort of the agent, you know, like, yeah. That's the other thing.
56:12I remember sometimes like a year ago, I was asked, what would you call long running agents? And I was just like, it's an agent.
56:18Jeff Barg: Like, I feel like our definition of what an agent is, is just changing. Yeah, totally. Yeah. I mean, I do think we're in many ways, we're still in the kind of like skeuomorphic age of agents where it's, I don't know, like, I think Instinct did a great job of this. It's like, do you really need threads for each thing? Like, it's kind of like you just talk to the agent. and I think there's probably something similar with the infrastructure where it's like, do I really need to know that the agent has scheduled follow-ups? Like it should just kind of do the thing when it needs to. It is maybe just like the agent, like that is what encompasses all of these things, like the infrastructure, the harness, the model, like that's the agent.
56:57How do you guys think about pricing for Clay? I saw back and forth, I think, between Christina from Vanta and Parker from Ripley talking about kind of like margins on AI, kind of like usage. How do you guys think about that? And any advice for kind of like people building agents about how they should approach that?
57:15Jeff Barg: First off, it's so core to the problem, I think. Like, I think you need to think about pricing when you're doing, when you're building an agent. Because if it works, like, these models are not free. You do need to have some sense of what the pricing structure, like cost structure looks like. I think whatever pricing structure we align on for this agent, it will be wrong. And I think that is almost explicitly the goal in some ways is it will either be way too cheap or way too expensive. And our goal is, of course, it should be way too cheap for the outcomes it's driving. But, you know, I think like if you could talk to an agent and have it find, you know, five new opportunities that end up, you know, turning into, you know,$100 ,000 plus contracts, like you probably pay a lot of money for that if you're rational about it.
58:08Jeff Barg: How close are you guys to charging for outcomes directly like that? I think pretty far. And I actually like I'm not sure that that's the right approach in some ways. Like I'd rather just, one, deliver that outcome. And then, I don't know, I do think there's actually something quite nice about charging for tokens. So this is maybe contrary to Christina's point of view here. Tokens, in my mind, are sort of the proxy for the unit of work. And yeah, like all of our customers are businesses, right? Like they are trying to also deliver outcomes for, you know, work and make a profit margin on that differential.
58:45Jeff Barg: And so I think it's very reasonable that for us, like, hey, we think our agent is quite good. We can help you deliver these really great things. But, you know, we'll charge just for sort of the unit of work that we've done for you. And then whatever you can do on top of that, that's, you know, surplus to our customers. I think it's, yeah, very hard. I do think, you know, charging for tokens or some sort of, you know, some proxy of that is probably actually probably not bad. You mentioned you guys tried fine-tuning maybe at the start of this year. Are you currently doing any fine-tuning? And when, if at all, do you think you'll try it again?
59:24Jeff Barg: I think our view has evolved a lot since then. I think we learned a lot from that original experience. My general sense here is, one, there are two outcomes you can try to drive from fine-tuning. One is you are taking a specific domain or problem space and trying to get front to your performance, right? You are trying to match or exceed fable in a specific problem space. And you're somewhat cost agnostic potentially for that, but you're just like actually trying to be the best. I think Harvey is sort of trying to do, you know, play in this space. The second where I think probably there's been more success historically for companies is you have a very well-defined problem space.
1:00:07Jeff Barg: you are doing it at quite high scale and evals are are somewhat saturated for the best models but you're basically trying to cost optimize so it's like hey i want you know opus level performance for this very specific problem but i want to pay you know haiku level haiku style prices one of the learnings is just like that second problem is actually much much easier like it's really hard to do i think what harvey is doing and you know what uh other folks are trying to do here. Yeah, so one, I think it's like, that feels like a better entry point is that second type of problem. For us, I think we are exploring getting back into this.
1:00:47Jeff Barg: I think one, just at the level of inference that we're doing overall, like there are opportunities to save on costs and actually be quite material to the business. What we're starting with is actually a model router. So So I think this is also a great way for vertical AI companies or probably many companies just to just to deliver excess, you know, excess value to their customers is, you know, like we actually have a pretty good understanding of what people are trying to do in Clay for certain for certain things. Right. Like I have pretty good intuitive understanding of what people use Claygent for, like our go to market research agents, like the types of tasks, the type of quality that they expect.
1:01:24Jeff Barg: um you can think of it like i could probably hand write a bunch of evals for claygent and they would actually be pretty aligned with what our users want and it's kind of like hey if our users trust us to do that like we can translate that into a model router and you can actually get that level of performance for you know 50 of the cost and you pass on some of those cost savings to the customer and this is sort of a win-win for everyone um so yeah we uh i think certainly by the time this will go out, we do have an auto mode for Clasient now, which is a post-trained model router. It sounds like it's post-trained, so I'm assuming there's pretty custom logic about how you guys do routing.
1:02:00And then does that live like at the harness layer or at the gateway layer?
1:02:06Jeff Barg: Today, it lives for us at the harness layer. I feel quite strongly actually that it needs to be trained alongside the harness. I think in terms of like where it's served from, like this somewhat you know somewhat agnostic right the thing that you really want to preserve is you don't want to model route to something and then you know route to something else and then have a cache miss right like that's a disaster right like you you would lose all benefit from from routing if you did that but there are certainly ways to do this even at the at the at the gateway layer when we were talking earlier you mentioned some challenges around making the CLI kind of like usable and kind of like a safe and almost governed way?
1:02:45Like, could you talk about that?
1:02:47Jeff Barg: I think one of the advantages we have for having a first party agent in Clay is all of the things that we can do around the Clay specific nature of our CLI, right? If you're using the Clay CLI in Cloud Code, Cloud Code can maybe use the commands. It may have a good understanding of what primitives use when because of skill files but you don't necessarily have all of the protections around hey we want to enable this capability for you but we probably want you to ask like probably want to ask you for permission before sending as an example sending a campaign or sending real emails to customers that was one of the things that we really wanted to build into the agent like human in the loop yes exactly um you can think of it as like uh you know approvals framework were kind of on top of the CLI.
1:03:37Jeff Barg: One, I think there's a product experience there that's actually quite nice about this. You know, really nice that as a consumer of Clay's agent, I can come in and I can feel pretty safe about using it and not shooting myself in the foot. But I think a pretty hard technical challenge, because we're using the CLI directly, if you look at these CLI commands, they are pretty crazy, right? Like they are potentially hundreds of lines of bash. bash as a turing complete language you don't actually it's very hard to before executing a bash command understand whether it's going to call a certain cli command or certainly like call a cli command with what parameters and so that was you know one of the things we thought pretty deeply about was how can we actually do this in a way that you know we give people all the capabilities of what they would have in the cli but ask for approval for the right things and so you know spent a lot of time on that we should have like a couple of different technical ways that we debated doing this.
1:04:34But how did you end up doing it?
1:04:37Jeff Barg: We effectively added a layer on top of the commands that we had, right? So you can imagine calling the CLI with certain parameters, right? Like there's a way to execute that where it just does the command. And then there's a way which is like, hey, the user has an intent to run this command. We'll generate an approval object that is passed back to the agent. And then the agent will basically present that approval to the user. so you can imagine right like there's actually a specific tool for the the agent which is there's bash and then there's like present an approval that i got from this bash command which the user can approve so we'll call the bash tool the result of that the bash tool called cli the result of that will be like some uh logic that or some some language that needs to call the approval tool it will call the approval tool and then it calls the bash tool again or does that approval tool automatically unlock things?
1:05:29Jeff Barg: The approval to automatically unlock things, yeah. We considered, I think you're alluding to the other solution that I think other folks have done and we considered as well, which is we could just block the bash command and then have the agent basically rerun the bash command with elevated permissions that are only given to the, right, sort of this like, forget the exact term for this, but elevated permissions and, you know, just-in-time elevated permissions. So can you basically like pause a bash command and then pick that back up once the approval comes in? We don't do that, but we did consider approaches like that.
1:06:04Jeff Barg: I think, like, actually, like, that's quite a good approach, I think. And I think this is the advantage of some of these code mode, you know, solutions. I'm somewhat skeptical of those solutions. They're all pretty early. And, you know, so we don't do that. We effectively have the agent just re, you know, call the approval after. What if there's, like, stateful work that happens before the approval? The approval will basically try to run the command. And so if it fails, that will just go back to the agent and it can retry and things. When you guys run evals and end evals for this agent, it sounds like it's interacting with a lot of different environments.
1:06:42Do you have simulated environments? Do you mock environments? Like how does the agent run against these things?
1:06:48Jeff Barg: Today, we have just one environment that it runs against, right? So it is four evals. It spins up a workspace. it spins up sort of all of these relevant, you know, data sources that the agent might interact with. Like basically a dev clay deployment with some fake kind of like data in there. That's right. Yeah. We probably need to, you know, do more here, right? Different types of environments, more simulated environments. I do think we really want to have sort of simulated users at some point, which, you know, I don't think our evals are totally comprehensive of like these long conversations.
1:07:24Jeff Barg: And I think it makes it really hard to simulate things like compactions. It really makes it really hard to simulate things like create a workflow, edit it, then like look at the results and, you know, have these sort of like true long horizon evals. And so, yeah, I think this is an active area for us to really think through. Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to maxagency at langchain.dev. We want to hear from you.
Jeff Barg is Head of AI at Clay, the go-to-market platform for list building, data enrichment and orchestration, with a dataset of 900 million contacts and 40 million companies. Clay recently launched a new global agent, built around a coding agent that runs on the same CLI external agents use. In this conversation, Jeff explains why Clay got more out of harness engineering than post-training, how the agent asks for approval on 100+ line Bash commands, and why he’d rather charge for tokens than outcomes.
–
We also discuss:
Where a first-party agent beats a plugin on latency and cost
Making a SQL-like query language for Clay’s search platform
The case for picking an off-the-shelf harness
Why Codex is “really bad” at debugging long traces
Why memory came before the agent
–
Timestamps:
(00:00) Introduction
(01:11) From Sculptor to the global agent: tasks vs outcomes
(07:32) Making Clay agent-friendly, and the CLI RevOps teams love
(11:50) How Clay’s evals moved from capabilities to user journeys
(15:54) First-party agent vs plugin: when to build your own
(20:05) Why Clay built around a coding agent
(21:53) What post-training taught Clay, and a SQL-like search language
(26:49) The case for an off-the-shelf harness
(30:14) Do lab-built harnesses really work best?
(33:10) Running the agent outside the sandbox
(37:51) Why Codex is bad at debugging long traces
(41:16) Knowledge Hub: building memory before the agent
(46:25) Multiplayer agents, and what the agent looks like in Slack
(48:24) How close the agent is to an AI head of growth
(53:44) Where the harness ends and infrastructure begins
(56:57) Why Clay would rather charge for tokens than outcomes
(59:14) Fine-tuning, and a post-trained model router for Claygent
(01:02:37) Approvals for 100-line Bash commands
(01:06:36) Eval environments and simulated users
–
References:
–
Where to find Jeff:
–
Where to find Harrison:
–
Where to find LangChain:
–
Send feedback or questions to [email protected]







