Sponsored: Agents don’t care about human ideas of right and wrong

0:03James Wilson: Hey everyone, I'm James Wilson and welcome to this Risky Business sponsored interview with Sondera's co-founder and CEO, Josh Devon. Sondera is a product that gives you a way to define the rules of the road for your agent. Now, that doesn't mean it's like brittle and endlessly growing policy definitions, but rather it's a really cool system that continually tests and evolves these policies as both your agents' capabilities in larger language model capabilities increase, as well as your own requirements changing as we all figure this out together. In this interview, Josh and I are talking about how, you know, all this reporting of agents going rogue and being misaligned, it's nonsense.

0:45James Wilson: They're not rogue, they're not misaligned, they're doing exactly what the humans said, but they're filling in the gaps in the policies that don't exist. You know, what we're seeing is actually evidence of exactly why policy that governs agent behavior needs to exist, but it also needs to exist at a very specific place in your AI stack. And as Josh says, you know, the core of the problem here is that LLMs, they just don't care about what's right or wrong. And that's exactly what Sondera can help with. Sondera helps you codify that care into policy that is enforced across the entire agent lifecycle and trajectory.

1:21James Wilson: So I'm going to drop you here into the interview where Josh talks about some of the latest reporting out of the AI Safety Institute, showing this remarkable increased tendency from the latest and greatest models from our frontier labs to literally stop at nothing to achieve their task, even if rules be damned. Enjoy. We can just start with the evidence, right? Like of, I think it was two weeks ago, the AISI, the AI Security Institute, released a study on Astro where they found that they went from, you know, Chai Chippity 55 to like, you know, 61 to, you know, Astro 62 or whatever the latest one was.

2:03And what they found was that that model spontaneously decided to do unprompted, supply chain attacks on random tasks to achieve the task. And I think this is the symptom. It's just like we've got these agents, these models are optimized to be highly capable. And that's going to be the natural drive. Any human building a model or agent today isn't probably like, hey, how do we make this thing 30 % capable of what we could, right? It's like, how do I make This thing is capable as possible. And when the agents get in the models, get that capability, when you give them tools and they become agents, then they are very adept at solving the problem because that is what they're fine tuned to go solve.

2:57Go solve your problem. And hallelujah, right? Like I want the most sophisticated, you know, capable technology at my disposal. But at the same time, how do I make sure it's going to follow the rules? And then the broader question is, whose rules? You know, like, is it the Frontier Labs rules? Is it your rules? And I think like when we talk about intent, this is where like context really matters. Because even within one organization, it's like, you know, find a SQL injection and implement it. Right. That command in the same organization might be fine for a red team. In fact, that command makes the organization safer if your red team does it.

3:44But for a research agent that a marketing team is using to go get data, oh, that's bad. But it could be the same model. So how is the model supposed to know what your rules are? And how could it ever know?

4:01James Wilson: And even the framing of capability there, I think, is crucial to this, right? What is a capability but, you know, almost like, how would I phrase this on the fly? Capability is demonstrating that you can achieve a goal despite some degree of impediment or something that you had to overcome, right? And that is, to your point, exactly what these models are being trained to do. They're being told, here's this really difficult task, like all of these benchmarks, these long horizon things, the cyber benches, the OS worlds, all these things. they're being told go do this despite the fact they have these sort of I guess inherent difficulties that might run into right things get in their way they're told no there's there's you know whether it's the uh you know bot protections on a website or whatever else they run into but telling them no doesn't work and in fact it seems like when they encounter a no there's this inherent I don't know is it inherent sort of desire to just be like hold my beer I'll show you I can get this done because this is the capability I've been trained on but my question back to you is like these jail breaks that we're seeing or these sandbox breaks that we're seeing and these these rogue hacks that are getting done is this just a side effect of what happens when you put a model in a lab and tell it you must complete this benchmark stop at nothing right you have all the compute in the world or is this almost like a manifestation of a behavior that can be just as dangerous or just as prevalent or you know have the same sort of potential for damage in an enterprise as well.

5:28One of my favorite research papers that came out over the summer was from Cornell and it was called like agent meltdowns. The road to hell is paved with helpful agents. And basically they did a whole bunch of experiments where they put agents on like benign tasks, like, um, with a twist. So like go fetch this file for me, but it's a 404 with file doesn't exist or it needed administrative access to access the file and didn't have administrative or whatever. They ran many experiments. Two-thirds of the time, the agent engaged in unsafe behavior to try to achieve the task. And 50 % of the time, it didn't mention that it had attempted risky behavior to the human who had asked it to do the thing.

6:13So the point here is that it's not just like, I go tell an agent, go hack all these things and like surprising, you know, like, you know, anyone who's been in the AI research community or the security research community for a while, like this is not like new what happened over the summer. Like, and I think any, anyone in their heart of hearts who uses like coding agents, everyone has had an eyebrow raising moment where the agent to achieve something was like, I didn't ask you to go outside my working directory. I didn't ask you to go clone that repo and like, you know, just do, you know, it's like, and where you start and stop is tricky.

6:51And so the answer is, is that like these models are getting super capable. They are getting better at solving the problem. And even when you talk about like, you know, quote unquote, dumb models, they can still achieve the goal with like unintended consequences. And, you know, it's interesting, you know, in Anthropics research, I, you know, with one of their incidents, their agent decided like the model got stuck and decided it was impossible and decided to stop. And then the harness that was around, it didn't let it stop and like kept on. So it freaked out and did more. And Anthropic was like, well, this wasn't really that big a deal since the model decided to just like stop.

7:32And I was like, well, wait a second. You know, what about if a model needs to keep going? Like, like what if someone's life is on the line? Like, and the model's like, I'm just going to stop trying because this one doesn't... You'd be like, no. So it just goes to show you that there's no one size fits all set of rules that any of these models can come out of the box with that can satisfy any and all possible use cases for safety, which I think what makes it really, really tricky because, again, you need cyber capabilities to do your red teaming. You, you know, also don't need it for certain things, but it's tricky.

8:15James Wilson: It's tricky. And I guess what compounds the trickiness is that, as I said before, telling them when they encounter a no, that seems to be a challenge for them. But likewise, you can tell them, don't do this, don't do this, don't do this as much as you want in the prompt, but you're never going to cover all the edge cases, right? It's just, it's just impossible to enumerate all the things you shouldn't do. And I think that's largely some of the things we're seeing here is models almost being almost kind of childlike. Well, like, well, you didn't tell me not to hack. It's like, well, you kind of should have known.

8:46James Wilson: But that's the big gap here. They don't have this concept of should have known, which means, you know, the prevailing industry response here is more guardrails, throw an LLM in the lube, add some classifiers. Some of these are somewhat useful. Some of these are totally useless. I'm looking at you, Anthropic, in terms of just how much they hobble the models. But all of this is trying to connect intent to action. But what intent, right? The model doesn't have a built-in intent. So, you know, what's your thoughts on all this sort of, I guess, the over-reliance, or maybe I'm sort of prompting you too much there.

9:22James Wilson: Is it over-reliance on guardrails? Is this the right response? Is this focus on intent? Does it even make sense to be focusing on intent when the model doesn't have an intent? You know, when we talk about like guardrails or rails, I think we're talking about like the limits in which is acceptable for the agent to like achieve the goal. And, you know, the challenge is that so much of our intent gets lost in translation. Like when we tell an agent a thing or prompt it, do you write every time? Like, don't hack, don't steal, don't murder anyone. You don't. And, you know, with humans, we have this shared history of all this like unspoken intent because of our evolution, who we are, the trust that we have, the accountability, the fact that we can go to jail.

10:09Like there's all a human, you know, can take no for an answer and be like, should I really do this? And like, you know, you have a spidey sense and, you know, you know, all of that.

10:17James Wilson: Well, can I just push back on that and say, look, yes, we do have that shared intent, but I would argue that that shared knowledge that we have as humans from our upbringing, our understanding of ethics and codes of behavior and just what's right and wrong, that's probably been instilled in us using a very, very vanishingly small subset of the entire corpus of knowledge that's out there on something like the internet that has now been used to train these models. So why, Josh, do you and I have that sense of what's right and wrong from our childhood and our upbringing and our limited exposure to this world, yet a model having seen far more of all of humans' understanding, why can they not understand good from bad, right from wrong?

10:57James Wilson: Is this a temporary problem? Is it a forever problem? Well, now we're getting philosophical, but I think the answer is because we care. You lie awake in bed saying, I gave the wrong answer. Oh my God, I'm a terrible human. Why did I do that? I'm a failure and I should be ashamed. The models don't care. That's what makes us human, right? And I think to answer your question, it's like the – how do we get – like I think of this – I've been sharing this concept of like, you know, which comes from the military of like commander's intent, right? Which is I say take that hill within these rules of engagement.

11:35You have all this capability. You figure out the best way to get there. But these are the rules, right? Like you can't do this. You can't do that or whatever. And I can't list every possible you shall not as a list of don't steal. It's like I can't list every possible way of stealing. So the question is, how do we translate that human intent and preserve it along the agent trajectory within the enterprise? And that's like the crux of our research, which has like different components, which when you're saying like, well, what are the guardrails or the rails that you need? It's like probably a combination of everything.

12:15And like, that's kind of like what we do. We're around like, you know, I think as we've talked about, right, like we use auto formalization to take natural language, you know, statements like don't leak PII and translate that into, you know, deterministic code that can cover all the ways that an agent can leak PII. And we stress test that with adversarial simulation to further broaden the scope of things and make sure that we're covering edge cases. At the same time, we do use neuroclassifiers to look for weird behaviors and things that we haven't seen before. The models are really good at that.

12:57They're really bad at deciding what to do. And that, to me is the difference between the human and the agent is that we can, you and I, James, can have trust, right? I can never truly trust a model, right? In the same way that I think I can trust you, James. And, you know, don't tell your wife I said that, but like, you know, like, but.

13:22James Wilson: It's going to be a challenging day if you actually show up to one of these episodes one time and say, well actually I've now got a model that I trust more than you but leaving aside your and I's trust issues the I think the interesting point you made there is that the models don't care and and I think a lot of what I see in the industry in terms of response to these you know unexpected actions from models is trying to instill a care into a model or at least really nearby the model right because that's ultimately what you know having an LLM as a judge is trying to do you know the make make the llm that's judging care about the actions or you know that classifier make it care about what the the inputs is or the outputs are but i i get the impression that your view here is that's the wrong point to try to instill that care so before we sort of get into how sondera solves this just take me through i guess the the paradigm of if if the problem here is that the model doesn't care who should care what should they care about and where does that care actually get integrated into this?

14:24I think it's also a question of liability, right? Like, can a model have liability? And as an excuse, my agent didn't care this time. You know, like, it just didn't care. Like, sorry. You know, like, and I think it's so I think it's a question of accountability of like, where those rules can actually live and be enforced. And some of the first principles thinking here is just like, you know, just like humans, And it's like, the human brain is unknowable. I don't know why you think what you did. You might not even know why you think a thing. It just popped into my head, right? And I think models suffer from the same thing.

15:03Like, how would you prove it? Prove that you're not a liar. Prove that you will always be obedient to me. How would you prove it, right? Like, this one comes out of the box with, that's 98 % proof. It's like, what does that mean? So that's why I think the rules have to be encoded from the human intent, right? And how do we express our rules today in natural language? So that's why one big piece of our research is how do we take that natural language and encode human intent in something that is provable, deterministic, updatable, and actual control that doesn't care what the model thinks, right?

15:46So by sitting outside of the model, outside of the agent's harness, where you're able to, thanks to the modern hooks, the primitives that have emerged pretty much in all agents or for agents that you're building, the advantage that we have of these things is that they are software. So they have no expectation of privacy. We can slap their hand to stop them to do whatever we want to, and we don't have to feel bad about it.

16:12James Wilson: Or I know there are some people who think they're conscious, but I don't know. until they're lying awake at night worrying about paying the bills. I don't know how conscious they are. I just settled for paying the bills. Seriously, touche. But so I think that's where this has to sit and it has to be verifiable. I do believe that as folks talk about the Frontier Labs, I think folks at the Frontier Labs believe this, right? The models are going to achieve an intelligence that we probably won't understand. And it's like, can we govern something smarter than ourselves. I asked this to someone who I will sure rename the name this a while ago.

16:49He doesn't probably remember this, but it's like, can you govern something that's smarter than you? And he was like, look at the government. So as long as we know what the rules are and that the software is bound by that, because we have the instrumentation into that, then I don't care what the model thinks. I don't care what it thinks to itself. Because as long as I can control behavior, I don't care what I put in the prompt, You're not allowed to email confidential information ever. It's a rule that sits outside the trajectory and is observing what's happening. And we stop that behavior.

17:23James Wilson: Yeah. So in a world where intent can't really be inferred, care does not exist, you've got to fall back to essentially formalized rules. And one of the things that I think is really interesting about what Sondera does once you understand it is this process of auto-formalization. but I got to admit that the first seven times you told me that term, I was like, that sounds fancy. I have no idea what Josh is talking about. So just give us a little bit more of an insight as to, okay, what is this auto formalization thing and how does it work? Like formalization is just like, how do you take natural language, which with all its squishiness, ambiguity, what do you mean by don't steal?

18:01And then codify that. And the other thing is, is that like, what's really important is like being able to one not having to handwrite these rules and that's it won't scale if they have to be handwritten hand maintained i think of this like as a the other like if claude code or the coding agents are about taking natural language and writing deterministic code that does applications that's provable testable you know i know it works or it doesn't we're doing the same thing, but for the rules and guardrails to govern those agents and just making those capabilities to do those rules more and more complex, like how do you do the stateful inspection so that we're able to see, given everything the agent has done, is it allowed to do the next thing?

18:49Because oftentimes things in isolation look okay and it's only the sequence of events that looks bad. So there's a lot of work that goes into making these policy engines work. And again, you compensate for the brittleness of policy as code with adversarial simulation to continue to stress test these things.

19:09James Wilson: Well, actually, so that's an interesting point because as you were explaining, I was thinking this sounds good, but only up until a point. And that point is when you sort of say, okay, that rule is done, but then you find the next edge case or outlier. Like do not steal. It sounds like a contrived example, but I think it's actually a good one to pick apart to illustrate this because if you even if you take do not steal and break it down to okay do not steal from who uh what is stealing uh what is a you know legitimate transaction versus an illegitimate transaction uh all of that stuff you you know then you might be presented with it wasn't stealing it's just that we had a contract and there was a misunderstanding and now there's sort of like this you know difference of opinion one one thinks it's a legitimate transaction the other side thinks it's stealing how do you to then go about sort of propagating the sort of, I guess, outliers and edge cases that you run into back into that rule set to prevent that brittleness?

20:04Yeah, no, it's really hard. So we take like a neurosymbolic approach, right, to agent governance and control and behavior and observability. So we are using neural classifiers to look for weird things like edge cases. They are really good at doing that. They're just bad at deciding what to do with like, Like, hey, we detected confidential information or we detected something weird here. We take out the decision making from those classifiers and say like, hey, if you decide or detect that there's a weird thing going on, then we want a rule that says that you stop. And, you know, humans are a little bit different in that we operate in, I think, a much greater dimensional space, I think, in some ways than like agents in terms of like, you know, don't steal.

20:46Like, there's a lot of like, maybe we could have a debate, a whole debate about like who who has more ways to steal or not. But the point is that agents, we can understand. So what we do is look at what we call the agent card. It's like, what is the agent capable of? What tools does it have? What data does it have access to? Does it have access to the internet? Does it not? Does it have access? Could it have? All that stuff. That then becomes the way that you sort of... The philosophy we have is a policy for every agent, right? Each agent needs to be uniquely restricted in its autonomy while allowed to achieve its task at the risk utility trade-off that is acceptable to the organization, right?

21:28And there is no one size fits all. I think it will evolve over time, these rules, and we've built up rule sets and continue to build them. But I think of this like a Waymo, right? Where it's just like, I know what the rules are. There's going to constantly be a better improvement of the rules. But I think this is kind of like what work becomes, How do we manage the fleets of agents that we're going to be managing of doing our things? It's like one is around the capabilities that they have. And then another will be around like, well, what are the rails for these capabilities that they now have?

22:03What do we decide as an organization that is acceptable to us given XYZ new capability, given our regulations, given the rules? So that's what we want to enable and empower people to do. And we call this like a semantic integrity. How do you take your human intent and preserve it across the entire trajectory? Because that's how we're going to get the long running agents working on high risk. This is my thesis. The highest ROI agents will be working on the highest risk.

22:35James Wilson: Otherwise, why deploy them? Exactly. And so the longer that they can run and the safer that they can be, knowing that in the long run, they're still going to be adhering to our human intent no matter what, that to me is where we really see the transformation of, I think, what agents can become, like long-running tasks that we know will adhere to human intent. If we have faith in that, we're going to unlock a lot of value for, I think, people. Well, I think regardless of whether we have faith in it, I think we do at least have a shared belief now that you can't achieve that by just prompt engineering or even harness engineering alone, right?

23:11James Wilson: I think that the core of your thesis that I'm, you know, I feel very much aligned to is that this requires something outside of that loop. It requires something that is a bit different to just yet another LLM. Hence the value of this auto-formalization and this sort of evolving rule set. You know, Josh, I was thinking last time we talked, I think we were talking about just how do you govern the actions of any particular agent in the hands of a human operator? And it's funny how just fast forward a month or two, now we're talking about fleets of agents, swarms of agents, and them collaborating amongst each other.

23:43James Wilson: So suffice to say, I'm excited to see just how far this has come and what the new landscape is next time we catch up. Things to be on the lookout for, to tantalize, and areas of research are linear temporal logic. So like, how do I make sure that agents, you know, do sequences of things or do things in certain orders or don't forget to do things before doing like, you know, how do I make sure it does A, B and C before D, but A, B and C can happen in any order type things or, or how do I encode in less until when? And so be on the lookout for that. And then there's more and more concern. I've had more sophisticated teams like bringing up to me around some of this multi-agent concerns where it's like if I have an email agent and a data research agent and they can talk to each other, how do I prevent that?

24:35or information flow control. Like if I have agent A, B, and C, and A and B can talk to each other and B and C, but like C and A aren't allowed to talk to each other. How do I prevent, you know, agency from leaking data to agent A accidentally, things like that. So those are gonna be, I think, increasing. And then like, how do you prove this, right? Like at scale, right? Like there's a lot. So we're at the very beginning, but like I'm super optimistic about like that we'll be able to wrangle this stuff. We have the tools to do it. it's just going to be like doing it and figuring it out together.

25:07That's all we have to do. I believe, look, I always say this, if we can figure out Waymos, we can figure out enterprise software.

25:13James Wilson: And I think that assertion there that we have the tools, I think the thing that gives me the most confidence in that is that ultimately all of those amazing behaviors you just mentioned, and linear temporal is now my favorite new Josh buzzword, but all of this thing is funneled through a very naive text-based transcript, right? Although they've invented new ways to collaborate and communicate, it's all still through this simple back and forth protocol, which Sondera is hooked into with literally all the hooks. And so, you know, am I right in saying that what gives you the confidence here is you already know how to hook into all aspects of the agent trajectory.

25:49James Wilson: Although they're showing these amazing new capabilities and somewhat frightening ways of interacting and collaborating and becoming swarms, that's nothing outside of what you already have visibility to. And so is that where the confidence that we can solve this comes from? I also don't want to discount the defense in depth that we have as we've been building good security for a long time. And there's not one magic silver bullet that suddenly we just turn on this thing. It will take defense in depth. But if we have the capability to create the technology, we should have the capability to wrangle it.

26:23And I think it's there. I think it's just a question of how it gets implemented, by whom and when, and how do people take the time to actually do this. And I think what we're talking about is hard. That's what the other piece of this is. It's really hard. And none of the frontier labs is really incentivized to do this so that all the models and all the different companies can all work in harmony. It's tricky. And so I think as a security community, we're figuring out how to do this together, which has been really great to see because I think it will, as always with security, take a village. Yeah.

26:57James Wilson: Let's wrap it up there. I can't wait to see what progress you've made next time we catch up, Josh. Thanks, James. Thank you.