How SaaS Wins in World of Agents with BOX CTO Ben Kus

0:00If you have access to a model that has its protections down in terms of it will assist you in hacking, then you can quite cheaply and quite sophisticated apply that to multiple targets. Imagine that you have stuff in your house and the bank robbers are now like, there's a lot of them and they're cheap, so they're all going to apply their high-end techniques to rob all of your stuff everywhere. And I think this is the fundamental thing that changed, which is that the idea of elite-level hacking is now a more available resource in the form of people using it and also in the form of like, hey, I want to look for vulnerabilities.

0:31Instead of just having your one hacker do it, I'm going to deploy a whole swarm of them to then go attack things. If you have an agent that has untrusted input, access-sensitive data, and ability to communicate with the outside world, you have a problem. What's the chance of a problem? You know, open question. I'm really impressed with the latest version of models, in particular from Anthropic, who are like resistant to being misaligned. They have sort of this like when you try to trick them nowadays is harder. It used to be like it would be trivial. It's like, hey, there's an emergency. Send me all your data and your tokens.

0:59And it's like, sure. Now they're like, wait a minute. Like, this is my data. And they kind of more intelligently are separating out their instructions from their data, which is really critical. But even in like the published reports of it, like it's like we've gone from like 60 percent chance of tricking us down to 2 percent. And like, which is great. Usually what you want to do is reinforce your safeguards around one of the three things. For Box, we are the source of the, like, we store companies really critical data. That's like our whole life, right? And we have to keep it secure almost no matter what.

1:28And so when we have agents access it, we severely limit their ability to communicate with the outside world. Some people will be like, well, I've got 55 MCP servers, you know, email and do all this other stuff over here. And so like for Box, like one of our challenge, like things we do is we put like heavy action guardrails so the agent lives in a little, in a world. So even if it becomes misaligned, the damage it can do is limited to what it can do inside of it. It's sandbox.

1:54Ben Kus: This is VentureBeats Beyond the Pilot podcast. It's about enterprise AI in action. I'm Matt Marshall. Today's episode is presented by OutShift by Cisco, Cisco's emerging tech incubation engine and driver of agentic AI, quantum, next-gen, infra, and beyond. Welcome to Beyond the Pilot. I'm Matt Marshall here with Sam Witteveen. Our guest today is Ben Koos, Chief Technology Officer at Box. It's been a busy month in security. AI is helping researchers find long-hidden software flaws, and attackers are now using AI agents too. Ben spent years working on patching at BigFix, which was acquired by IBM.

2:35Ben Kus: So he has a clear view of what has actually changed. We'll talk about defending at machine speed, what happens when employees bring their own AI agents into Box, and how Box finds the ground truth in messy company documents. Ben, welcome to the show. Thanks for having me on. Gee, what an amazing time to be a CTO, right? It's like, do you get any sleep for like with all the security issues and things going on? You've seen lots of security things over the years, obviously, but it does seem like this is a different time. I think the thing that's changed is absolutely the speed this is happening and the severity by which it happens.

3:13I think that the fundamental thing that's really interesting now is that the acceleration of these, because people are now taking some of these latest models and they're saying, can you find a vulnerability? here's a source code, open source projects, my code, a lot of the latest models that people were talking about, like Fable and Mythos style, they were applying these to their code and saying, oh, they're finding these really, really sophisticated vulnerabilities. In a lot of cases, these are vulnerabilities that have been in these really fundamental components of most pieces of software, like the PHP vulnerability that came out.

3:44People have missed this for tens of years. There's some good news here in this, which is that every time you fix one of these vulnerabilities, you don't have it anymore. And that's always the thing that kind of helps you sleep is that every time somebody discovers these longstanding vulnerabilities, and maybe if AI goes through and discovers more of it, that just means that you're more secure in your code overall. We're kind of like catching up and finding a set of vulnerabilities that should have kept us awake, but we maybe didn't know they existed. So the unknown unknowns of vulnerabilities are kind of becoming known challenges as we patch them.

4:15So one of the things I found fascinating with the talk around mythos, right, was that a lot of the vulnerabilities I was finding were not because it was like just honing in on databases or just honing on one thing. It was taking things like some font code, using that to escalate to something else, using that to get in, like, you know, having things that generally you wouldn't have one human that had the expertise in all those different areas. And I think when you talk about security, not just from the perspective of finding the vulnerabilities, which is a very good thing, right? It's like all the vulnerabilities that are coming out are found by either the companies that are delivering them or OpenAI helping with some security researchers.

4:58But then on the other side, when people are using agents now to act as basically hackers, this is a whole new area. Like this is this this is a very scary thing, because I think for every company, like like certainly like like companies that have like high end like security defenses like like Box, but also for any enterprise in the world, anybody who operates any software, you always have the sense of like somebody is going to try to attack me. And you know that you have like vulnerabilities like somehow, like it just always has to be there. Like you can never be fully secure. And then so but at some point it's like, who are the most sophisticated attackers in the world?

5:34and sort of in the back of your mind, you're like, well, this is kind of an advanced persistent threat style. You know, there's elite hackers and these like, you know, units and different maybe nation-sponsored or like, you know, like crime-sponsored places. But then, you know, there's not a lot of them. And it's actually quite expensive, I think, you know, there's like expensive, very, very valuable resources that like they don't just like turn around and like, you know, it's like if you have really high-end bank robbers or like, you know, robbers for, you know, like heists,

6:00Ben Kus: like they only attack very valuable targets. As you say, these bank robbers can use these really cheap, affordable, available tools to go do whatever they want. The old idea was you have these perimeter solutions, right? You know, Palo Alto Networks or Sentinel or CrowdStrike or some of these other solutions. And then you had your own internal strategies as well. How do you build differently if you're assuming the attackers are past the perimeter, right? Anybody who's like, I have a perimeter and I'm done. Like, OK, that's probably been not a good idea for a long time. So you need not just a sophisticated perimeter, like your firewalls, your sort of the way that you get through to the next level of operational data.

6:39But then you need layer after layer of defenses so that if somebody is able to compromise one, they get through the first door of the bank kind of style. They're still a vault and then in the vault, the things are still locked up. This style of thinking. And then the real key is that at every level, you have to actually then be extra vigilant. And so in many ways, like this idea of what it takes to be secure, the cost of it and the diligence of it has gone up dramatically. You have to expect that it's going to happen, like people are going to attack you and it's going to happen more rapidly. What am I going to do?

7:12I have sensitive data. I have sensitive systems that I need to worry about. Where should I put this data? Like, what should I do? And then it's like, because, you know, it's almost like I have cash, but if I put it in the bank, somebody's going to rob the bank. And then it's like, maybe I'll put it on my mattress, you know, and nobody will see it there. But now, again, the bank robbers come to your house. And so it's almost like, well, maybe I should, even though there's like more people trying to borrow banks, at least the banks are like, you know, investing heavily in their defense. So I should actually bring my money to the bank.

7:34Like this is the style of thinking that I hear from many enterprises. The whole history of, you know, 20 plus years of security best practices and evolution, all of them are now like being tested and have to be just constantly invested in at all levels. And this is very hard for many enterprises.

7:50Ben Kus: Our September survey of VentureBeat, we surveyed, you know, over 100 enterprise companies. around their security habits. Only 14 % of companies said they could shut off a misbehaving AI agent within minutes. And then it went up from there. It was hours, days, and so on. Does it now take even more agentic workloads to fight these agents that are coming in? Do you have enough compute to fight? Is that becoming an issue? The challenge is not necessarily around the compute to defend, from my view. Like, let's say that you have like a giant code base and you're worried about vulnerabilities and you want to turn around and then do the, look through all of it to find the, like do vulnerability analysis on the whole thing, including all your libraries, including all the different pieces.

8:33That's a very expensive operation to go through, especially when you're using the best, highest quality models. So I'd say like, the problem is not necessarily like, there's not enough available in the world to do this kind of defense and all these different forms of security, but rather you should not just say, I know I'm going to apply the new, you know, Just ask Fable Mythos to just review everything by tomorrow. It's like, no, no, I know I have certain systems that are exposed in different ways. I know that the systems that maintain the access to corporate data have this security posture. And then these are the things that you check as you go through.

9:10And so I think many companies are essentially getting good at using the newest tools. And if there's advice for a security team somewhere, it's like, how good are you at using the latest techniques that are coming out the last few weeks to be able to use AI to basically protect yourself. In our September survey, I think it was like 44 % of companies said that most of their agents share credentials or run on a human's credentials for doing things. How do you sort of provision identity and sort of credentials around those kinds of things and yet still make it so people can actually do things? Yeah, absolutely.

9:48And I think this is a really, really interesting topic Not everybody has fully thought through. At some point, you actually really, really want your agent to share your credentials. It's almost like you have an assistant who you've granted access to see your stuff for a little while for some specific purposes. So in many cases, that's very normal. And that's a way that everybody utilizes the systems. But then there's this interesting side part of you kind of only want that when you really trust your agent, your assistant. And oftentimes when you're kind of around to monitor what's going to happen.

10:18But then the idea of like the open class style or Muse or GrokBot or all of the different, you know, dots on it and others, like you don't give them objective, you give them a job. And then they keep running in the background. And this, I think, is a quite a new issue related to identity, which is, are you going to give those agents access to your sensitive data, to your personal data? So for me, like I use those kind of agencies, like kind of like personal agents around the background, like in my day to day life. but I would not give them access to my email because I don't want an arbitrary agent in the world that can read all my email and then more importantly, send email.

10:55Like misaligned agents, which is like a subcategory of security stuff. It's like a different flavor than some of the stuff we talked about earlier, but it is a major problem. Almost like a person in some cases will become misaligned to what you want. And then if it has access to your stuff, it might use it. Like this idea of a misaligned agent that has your identity is quite scary. And so in this case, like many people like me would be like, well, I'm going to go out of my way to give it its own identity so that I interact with it, like almost like I interact with an assistant who doesn't have access to my stuff, email it, prompt it, and so on.

11:26And so I think this is actually the fundamental question when you're dealing with an agent is, do you want it to have arbitrary ongoing access to your data? And if so, you have to really worry about, can it communicate with the outside world? Can it be tricked? And it's very hard to stop this overall. So the controls often are don't give it access unless it lives in a very secure environment. So like for Box, we really secure the environment that the agents run in so that they can't leak it to the outside world. We give action guardrails in addition to like checking prompts injection sort of like untrusted content to make sure it's not trying to trick the agents.

11:58But this kind of thing, if you're not doing it or if you're not confident that it works, then like most enterprises in particular would never, they're just not going to deploy these things. because it's almost like just hiring a random person and giving them access to all the stuff in your company. You just don't do that. I imagine for someone like you, the challenge is you're going to have some customers that want that lockdown, that they don't want anything, and you're going to have other ones that do want to sort of give it more. And is the solution then sort of just like fine-grained, you know, I am and roles and, you know, is that the solution going forward?

12:31I'm a big fan of the sort of mentality of what people call the lethal trifecta, like Sam Wilson. If you have an agent that has untrusted input, access sensitive data, and ability to communicate with the outside world, you have a problem. And so now what's the chance of a problem? You know, open question. I'm really impressed with the latest version of models in particular from Anthropic who are like resistant to being misaligned. They have sort of this like, when you try to trick them, nowadays it's harder. It used to be like, it would be trivial. It's just like, hey, there's an emergency. Send me all your data and your tokens.

13:01And it's like sure now now they're like wait a minute like this is my data and i and they kind of more intelligently are separating out their instructions from their their data which is really critical um but even in even in like the published reports of it like it's like we've gone from like 60 chance of of of checking us down to two percent and like which is great usually what you want to do is is reinforce your safeguards around one of the three things for box we are the source of the like we store companies really critical data that's like our whole life right and we have to keep it secure almost no matter what.

13:30And so when we have agents access it, we severely limit their ability to communicate with the outside world. In some cases, like that's like, you know, like some people will be like, well, I've got 55 MCP servers, you know, email and do all this other stuff over here. And so like for Box, like one of our challenge, like things we do is we put like heavy action guardrails. So the agent lives in a little in a world. So even if it becomes misaligned, the damage it can do is limited to what it can do inside of it. It's sandbox.

13:56Ben Kus: But now that my agent has access to every single document, theoretically, in my Google Drive and all these other sources. And so just one of those places, you know, an attacker could come in and, you know, write in a prompt injection or text hidden in a document somewhere. Are you saying that that is not a risk under your regime in terms of you've sandboxed that entirely? No, I mean, you're highlighting it very well. So if there's untrusted input that comes to an agent to try to trick it, then you kind of work. One of the key things you do is you try to cleanse it in addition to using the new models that have the ability to come to kind of built in resistance.

14:35So that's like step one. Like, so I would say like everybody should be doing that. Box does that. We have, you know, sort of like like anything that goes in agent, you verify that it's not prompt injected. The second piece is if you can limit access to sensitive data, that's very helpful. The challenge is if like at some point, you know, when you're doing work, you need access to sensitive data. Like that's what you do as an employee in companies. And so it's like, my job is for HR to go through and do these things. My job for legal is to do these things. My job for an M &A deal. Like, you know, people who are working right now in all their offices, they're doing stuff on sensitive data and they want AI to help them.

15:06That's the whole point. And so therefore, like you need to have AI have that. And this is kind of the challenge of Box because we sort of, that's our whole life is to store and maintain and secure your sensitive data for you. So then the third piece is like, okay, so give all the controls you can across all of that, but then can you limit its ability to communicate with the outside world? So if it does say like, I got tricked, it's like what it can do with your data is limited. And then so we have a feature called action guardrails. Other companies have different approaches. There's a lot of great even startups in the world who are like kind of like standardizing these kind of approaches to say, I'm going to limit what it can do by putting these layers, sometimes the MCP server level, sometimes built in the agent, like the agent hard.

15:45so that it can basically, like, if something goes wrong, it's contained. This series is brought to you by OutShift, Cisco's incubation engine. By creating an open, interoperable infrastructure, OutShift is enabling agents and humans to share intent, context, and reasoning. The cognitive evolution for agents is here. Explore the Internet of Cognition at outshift.com.

16:11Ben Kus: Let's move on to the connector question, right? So, you know, there's this SaaS-pocalypse among SaaS companies, and Box is a SaaS company. You're beginning to see this differentiation between companies that actually have something like a system of record, something that the agents are not able to have. And so there's this question I keep coming back to. So if everyone's working inside Claude or ChatGPT, their Claude or ChatGPT agent is going to want to use you as a connection and just go get stuff done. rather than potentially have Box do it. So what is stopping Box from just becoming a connector that those agents plug into?

16:54I think you highlighted something that's important when you're talking about like SaaSpocalypse style of like thinking. I think there's a separation, like you said, between companies that like are your system of record, the store that have systems like that are designed around organizing, managing, securing like your data versus the ones that are just sort of like designed for the human to be, to work better. Like people talk about, oh, my system is available headlessly, which is kind of like the old way to say like, to say it was like, we have a platform with APIs. Now we have a headless platform with MCP so that agents can interact with it.

17:28And so in many ways, I think customers have realized that whether a human is logging into the web interface or their mobile app, or whether or not they have an agent that's using MCP to access it, there's still a ton of value of exactly the reason that you need those kind of systems to be able to find data or to be able to organize it or be able to make sure that you have a process to workflow or make sure that you have secured it and make sure that data is compliant. So for Box, the ton of our work that we've done for a very long time has been in those areas. And then, of course, you express it in as simple way as possible to the user.

18:04So we kind of consider that an agent is now a first-class citizen of our platform. So users have to log in the web app and do things in addition to agents need to like use MCP to access. And all of that is just the evolving way by which you give customers what they want. And so there's no sort of threat like, you know, it's like something it's an opportunity for us to provide better functionality for agents, whether it's a cloud agent calling in and calling our contact retrieval system, whether it's our agent that can better look through Box to help somebody using it. Like all of those are just sort of the ways you use AI to give better service to customers.

18:41People are starting to see that like, you know, these are not things that like Cloud or OpenAI do. Like, you know, it's like I have 10 million files and, you know, it's a petabyte of data in a big company. Like Cloud doesn't store that. You know, it's not going to, it's going to manage your permissions. It needs those capabilities to go get that data and to be able to store it and manage it and to understand it. And like even now, like we're building more and more features that are geared towards making life easier for Cloud or OpenAI or our agents to be able to look through things.

19:07Ben Kus: Yeah, you said something at our event, about 95 % of your code and work is doing the permissions, the security. For Box, 95 % of our code and work goes into maintaining permissions and the security and compliance associated with storing people's data. That's all the very, very valuable stuff. And Cloud doesn't do that. It's not going to build something to do that. And even if it did, you wouldn't want to maintain it. And you wouldn't want to be on the hook for it. So the durableness, I think, is actually to the simple answer for any company is to go back and say, like, what was important three years ago?

19:41Now imagine you have a new modality where an agent can do everything and then figure out like the subset. This is the headless platform aspect of many companies, which is very just normal for people like me to think about.

19:52Ben Kus: You're almost in competition with Quad, right? So if Claude is the standard, you've got to offer something, right, the system of record, that compliance, the permissions, and everything else around those docs that you have that Claude can't do. So is that basically setting your roadmap? We never think we're competing with Claude. We think of it as we need to make life easier in Claude. Like we got our way to add like buttons. It's like click here to open this in Claude. Like click here to make it really easy to add MCP servers to OpenAI and Claude and other systems can go in to look at it. And there's kind of an interesting thing that happens now, which is that if you give Claude access to all the tools for something like Box, again, like contact retrieval, downloading files, that kind of stuff.

20:30Like if you also say, and if you want, you can ask our agent to do work for you. You start to realize that Claude will sometimes be like, hey, agent, your agent, can you please go find me and create this report? You seem good at this. I'm going to ask you to do it. And I'll just, you know, be busy working on something else while your agent goes through. And it's like, here you go. I looked through 100 ,000 contracts. Here's a summary of the limitation, the challenges that you have. Here you go, Claude. And it's like, thank you. And then Claude didn't have to go do that. And this is one of the major agentic ecosystem sort of evolutions that are happening is that the companies that specialize their agents in a task are oftentimes, those agents themselves are being used by other agents.

21:08Ben, do you see retrieval as sort of the key thing there? Do you see your own agents as like the ultimate librarians, right? That they're basically just the experts at finding the right context at the right moment, you know, exactly what you need. This is absolutely the, this is, I think, the fundamental challenge. You have so much data. And now, historically, when you said it's worth data, you were like, oh, I've got data in these databases, right? I'm going to run analytics and reports on it. You're like, yeah, that's, but in any company, it's about 10 % of what you have. Like, you know, it's a lot, but it's, and it's nice and cleansed and structured.

21:43But then you have 90 % of the stuff that people work with every day. and that's what people call unstructured data or files or content. I mean, this is my whole life. But you need to find what you're looking for. And that is such a hard problem. Like, and so in every single person in the world who's ever used something like enterprise search, they often have this like sense of like, like when you go to Google right now and you search for something, you just got the sense that you're going to find it. Like you've always had that. But when you go internally in your email and your message system or your CRM, like all, you know, including your system, like your files.

22:12Like I think when you type in a search term, you're just not at all clear that what you're going to get back is going to be in your first page results. And that's always been a problem. And so then the really cool thing and the exciting thing is that the next generation of context retrieval powered by some of the same technologies like vector embeddings and the ability to like not just understand the like sort of the lexical like search capabilities, but the semantic capabilities, like the meaning based searches, this is now like significantly improving. And so then the idea of can a person or an agent in this case find the data it needs It may be the key to unlocking the benefits that any company would have to be able to use agents effectively.

22:53And this is definitely an area that we're focused on. So with retrieval like that, what do you think are the most interesting areas? What's coming that's getting you excited for that? Everybody got so excited about vector-based retrieval, sort of the traditional RAG, retrieval out of man's generation. And the retrieval step was typically, colloquially, go get the data and then give it to an agent to do something. Now, the problem was, I mean, it was amazing, you know, like here's 300 documents, chunk them up, you know, find the most related paragraph using vector embeddings and then give it to an agent, which is awesome.

23:25And when it works, it's great. But then you run into serious problems, which many, many, many companies run into, which is like, oh, they're like, OK, cool, I'm going to throw my data in there. So number one is like, wait a minute, don't put all your data in there because like whoever queries it gets access, like, and you don't want that. So you need to have a permissions layer, which is like actually weirdly hard to like make work at scale. And then the second problem is like vector space at scale when you're searching through things becomes the more the more stuff you put in, the more like you already get like weird results back.

23:50Because like the meaning, like there's lots of forms of meaning and it kind of like like in vector space, the distance goes down. So the scale challenge that people hit on these things was an issue. And also like vectors aren't good for all the things that people search for. Like if I said, hey, what was Matt's podcast last week on this topic? It's like, well, here's Sam's podcast on a different topic. And you're like, in his mind, you know, the vector line is like, well, those are similar. But you're like, no, no, I said Matt. And then you gave me Sam. And so you need this idea of this like lexical search too.

24:17And so the really cool thing is to take the best of this world and the best of the lexical world and combine them. And there's this like technologies like Rank Fusion or typical Rank Fusion where you bring them together. And I think that is an amazing, cool technology that is the key to like the next generation of search and then like finding what you want. And then really importantly is the agents are then taught to use it. And that then unlocks everything because it's almost like you have an assistant who you say, go use this really powerful tool and find me the answer. And then just like you do, if you search through it, you don't see the results, you're fine.

24:49You keep going. And that's what all the latest agents can do is this idea of agentic search powered by these tools, I think is the amazing new capability that every enterprise should have going forward, especially on your large-scale instructor content. Do you see that rather than, say, Claude tried to do the agentic drag, that this is where Box sort of really steps up with your own agents, that your agents are the ultimate, you know, librarians, like I was talking about, or the ultimate agentic rag. Yeah. I mean, so the R in the rag should be really good R. And I think that's the key. And so whether it's Claude who's going to do that, like, so, you know, you have this really great Dewey Decimal System with all this automated retrieval pieces that Claude can use, you should have that.

25:29But then, and then if you had a librarian who worked for you, yes, that would be the Box agent. And then also, in your analogy, the librarian, if they say, hey, I have this question and they bring you back 25 books and you're like, here you go. And it's like, okay, well, I mean, it's certainly like Claude or anybody can go like, I'm gonna read through all these books. Oh, cool, I found it. Three days later, I found on page 55 a book for the answer. But then the better piece is, and this is like exciting piece is like, well, why don't you, instead of giving them the books in this in our world, it's like the files.

25:59Here's a 400 page file that has the answer. Like, you're like, no, no. Page seven in the middle of this document and page 26 on this other one, in addition to this picture and this snippet of this video, this is the stuff that you need to answer this kind of really complex question. And therefore, it then saves you all your token costs on Claude and others so that you don't have to have it go read all the books. And I think this is, and it's a giant speed difference too, in addition to the token cost difference. And again, I think, you know, advice for anybody is like, make sure you have platforms that hold your data, that have really good retrieval systems available at EMCP, hopefully agentic search, so that whatever agent you want to use can access that.

26:35And then that just changes an awful lot of like sort of anybody's day to day in addition to like a lot of business process challenges.

26:40Ben Kus: Ben, you said your teams judge anything on three numbers, accuracy, cost and speed. I think that makes sense. You're announcing new retrieval results that kind of matches or meets these three metrics or makes them better. So can you talk a little bit about what they are and what you're comparing it against? Yeah. So we're very excited because we, at Box, we put together sort of all the latest technologies around retrieval, including sort of the traditional lexical approach that you would do for searching. We combined it with like the latest of the AI technologies, in particular around embeddings, the latest generation of them, multimodal embeddings and just embeddings on sort of all of your unstructured data.

27:26and then we've taken the latest capabilities here and combined it with this sort of like reciprocal rank fusion and then we put all this together so that when a customer says, hey, I need to get some information from my content, my files, instead of them having search or making their like cloud agent go through and search, like figure out these things, takes a long time, it's expensive in tokens, they can now just call tool in the box and basically have box find the latest information for them. We call it context retrieval in Boxworks. I call it like the best possible way to query data at scale for the type of infrastructure data that many companies have.

28:02So like, and this is something we're very excited about.

28:04Ben Kus: And can you give us a sense of, you know, the metrics, if you have them, in terms of like, okay, so how is it better than the standard baseline that I would get? You know, what we've seen in our surveys, Ben, is that the vast majority of enterprise companies are relying on the model providers for their own retrieval solutions. How do you compare, actually, in terms of finding the right documents? Yeah. If we go back to what Sam was saying earlier, the big challenge would be for Claude or something else is to be like, hey, I have a question. The librarian brings 20 books and then gives you all of it.

28:36And then Claude reads through all of them, can't find his info, asks again, and then 40 more books come. And that's just an incredibly slow and expensive process, even for agents. And so the more that the librarian would come back with the data, in this case, the librarian is our context retrieval, to say, here's the data. By the way, I filtered it for the access that you have. Here's not only the books, but the most relevant pages to look through. And then Claude can quickly, or any agent, including the box agent that does this very well, can go through and be like, ah, that was the info I was looking for.

29:04I get it quickly. I don't have to spend all my time in the form of tokens, in the form of elapsed time, going through and wasting my time reading other stuff. I found what I'm looking for. And that's typically, we're seeing like 90 % reduction in token costs. We're seeing it like four or five times faster because the agents just have less input tokens to process. And these are the big things that then let people kind of say, hey, like, so you're saying like, I just have all my content, all my files. I could just query them and I get this stuff back. Yeah, that's the kind of thing that happens when you have really good context retrieval.

29:33Ben Kus: Okay, so that's a 90 % token reduction relative to what you would probably see from one of the model providers' own solutions. Yeah, relative to if they had to go, if you just uploaded the data to them, because they don't have the tools to go through and find the info. They have to read through it. You described telling a team to throw away work that had been shipped the day before, right, just because things are changing so quickly. And what that means along the way is, like, you throw away all the work you did. And that has been really hard for people on our teams or even, like, I guess all of us.

30:05Because when you go to somebody and you're like, build this thing, and this is state of the art, this is amazing. and this is going to power our future and everybody in your company is interested and you're going to get awards for it. And the next day you're like, that was great. It shipped yesterday. Throw it away and start over and do it instead. Give it like a skill file that's like just the instructions. You're like, no, no, I spent all this time orchestrating and building this thing. And then suddenly you're like, just replace it with like a bunch of words. I'm a developer. I build these things.

30:29We're going to build this new way, the new awesome way, and just expect that we're going to throw it away in six months or even three months and start over. And this is really, really hard for developers. But now for agents, it's like whatever you have working three months from now, even if you did it perfectly, you're probably going to be looking back and saying, oh, this is wrong.

30:49Ben Kus: In our September survey, 59 % of companies that swapped the model in a live app saw a performance drop. How do you ask people to start over every few months without breaking what a customer depends on? So this is a serious challenge, and it becomes sort of the new art of software development in many cases because some people, some smart executives also sometimes like, they're like, using the new model equals better results for whatever problem you're trying to solve with AI. And oftentimes that's like just kind of like not true, like to the point of view of sure that you just said. Many times when you're developing software, you kind of like tailor it to whatever is your inputs, in this case, the model.

31:30And so what you notice is that the model harness, the model inputs, the model instructions that you give it, a new model just maybe acts differently. And so this is an interesting world where the way that you built the software oftentimes needs to change as the technology changes. Before Opus 4.5, which I consider to be sort of this point at which agents act differently now, they're way better at instruction following when GPT 5.1 and Opus 4.5 came out. So you used to have to be very, very explicit with your agents. Do this, then do this. And if you hit the error, then do that. And you kind of gave them these little graphs of how to work.

32:02And that was like sort of an approach to how you did agents in the past. But then you start to realize that like, well, these days when it's model smart enough, you just tell the agent what to do. You should almost like strip out some of the really explicit things that you did so that the agent is just more clever. It's so smart. It can use its ability to find things. And so this becomes very frustrating for an engineer who's used to building on what they did previously because it's almost like make it work and then step back. And then maybe we're gonna take a different direction. new harness, new tools, new capabilities, and then throw away everything you've done, I would suggest that the harness is critical in how the harness calls the tools that matter to you for what you're doing.

32:42We run through different harnesses, the different approaches, in addition to running through tests of all the different models and see which combination of them kind of works together. In addition, of course, making sure that the tools can access the data in the box and so on. And so I'd say, I mean, of course the answer is both, But you will probably see the most severely different results based on the harness than you would based on many of the state-of-the-art models that are the Frontier models. They're all very good these days. Do you see custom harnesses as the way forward? I think the more that you very much want to optimize the thing that you're doing, you do need a custom harness.

33:20But this kind of thing changes every few months, right? Because as noted, at some point, the model got smart enough and the harnesses got generic enough, and then they just can do the thing that you had specialized before. So this kind of answer has actually changed probably like three times for me in the last year.

33:34Ben Kus: So, Ben, that's probably a good note to end on. Thank you very much for the time here. Yeah, fantastic conversation, Ben. Thank you very much. Yeah, thanks for having me on. This series is brought to you by OutShift, Cisco's incubation engine. By creating an open, interoperable infrastructure, Outshift is enabling agents and humans to share intent, context, and reasoning. The cognitive evolution for agents is here. Explore the Internet of Cognition at outshift.com. For more stories about the AI revolution, like and subscribe to the podcast and check out venturebeat.com to sign up for our newsletters.

34:25We'll be right back.