Where Does AI Agent Security Actually Live?

23:43Corey:In the last few weeks, there's the OpenAI incident. I think Anthropic and Meta, all three, had similar incidents during safety training. And while that's absolutely concerning, is there value to that happening in a lab scenario like that where we can learn, I guess? 100%. So first about those incidents, these were the same incident. There was a vendor that did testing for those companies. And that vendor basically gave a vague instruction to the model in order to achieve the capture the flag exercise. And it ran in a sandbox that was not configured correctly. So the model ran away. But it's not that the model really ran away.

24:40It's not like sci-fi as you will expect it. The box was open and the instruction was, you know, go do something that the model interpreted as the right thing. So it wasn't that unique. But the fact that these things are happening now and they're getting a lot of attention and they're happening in a testing environment, exactly why those tests are happening. Because the models are capable to a lot of bad things. And it's just like a small example. also those capabilities are also available on open weights models. In open weights, you can just like download them and do whatever you want with them.

25:23You can even remove the guardrails from them. There's a process that takes a few minutes. You take a state-of-the-art model like Kimmy, you get the guardrails and, you know, here's your machine. Do whatever you want with it for better and for worse. Yeah. Your decision. So the fact that people are aware that these things can happen and we can also both as like business owners know how to leverage that and protect ourselves from it. But also as society, understand that the balance of power has changed. Think about that. Like the$1 billion crime organization that is made out of like one person is here.

26:06Yeah. They can do whatever the hell they want right now. Because they have these capabilities. Bad things that you could do manually, now you can do in the speed of AI. We don't have the defenses deployed yet. Maybe we have them. I'm very optimistic that we will. But in the short term, no. Let's talk about that.

26:28Grant:What needs to be done? What do people need to deploy? How can they defend themselves right now to stop this before it gets too crazy? like from them getting hit, you know what I mean? So, you know, first thing, I think we have sort of a problem where cybersecurity gets most of the attention. Yeah. And personal security and, you know, other bad use cases of AI are not getting attention at all. So we're very, there's a massive industry, cybersecurity industry. It is all about how can we protect the new novel AI threats. And every company is making acquisitions and making investments and every organization is buying a lot of services and stuff.

27:21And everybody's enjoying that boom in the industry. But when we're thinking about the person that may be a victim of a deepfake attack or like a social engineering powered by AI attack, all of the bad, like fraud, abuse, deception, issues that we know from the beginning of the internet that is now getting amplified with AI, they don't have a lot of things to do besides, you know, increasing their awareness and hope that other platforms and, you know, and law enforcement will protect them. So I think actually that's a story that is not getting told enough. these folks, which is all of us, are not getting the protection that they need.

28:08And I believe there's going to be a massive industry that is going to protect these people, these people, us, our families, our friends, against everything that is about to happen is already happening.

28:21Grant:What does that look like? Does that look like watermarking? Or how would you even begin to defend yourself from that? So there's so many angles right now, and they're so underserved. We can speak only about that, but you have identity. Am I speaking with the right person? Is this interaction even authentic? So it can be about, is there a real person behind this interaction? And am I seeing a real, authentic piece of information? So watermarking is definitely one piece of the puzzle. and anywhere from cameras that tells you this is real, it was not manipulated, to the watermarking that the model companies are creating, to the new initiative by Entropic that even will tell you if a text was generated by AI.

29:13And I saw, by the way, a lot of companies are losing their minds. Oh my God, my slop would now be recognized as slop. And the bad SEO that I'm creating, people would know it's AI. So Newsflash, everybody knows it's AI. You're not tricking anybody and definitely not the SEO algorithm. But this is actually a wonderful thing because think about a future tool that will tell you that you're actually texting, not a potential romantic partner or a business partner. You're actually texting with AI. Yeah. By the way, this will definitely not be relevant for a bad actor using an open weights model with no guardrails.

30:01Yeah. But this is kind of like a wonderful idea that I believe Entropic presented to the world, and I hope everybody will adopt it. But again, those are just like two examples. You have everything around when you're using AI in order to amplify hyper-localization and optimization of ads. So you want to influence people's attention, elections, everything. These things are under – people are not talking about that. People only talk about cybersecurity. Not about the 99.99 % of the world that is going to be affected by other threats.

30:46Corey:Now, I'd like to step back to open source for just a minute because you made a really good point about the fact that you can do whatever you want with so many of these models. And while I am generally a pretty rabid supporter of open source, I'd be lying if I didn't say that that thing concerns me. And it makes me wonder if the cat's already out of the bag there, that's what a bad actor is going to use anyway. They're not going to go pay for Claude. They're not going to pay for ChatGPT. They're going to go use the one that is literally out in the world already and can never be taken back that has the wide open door for them, I would think.

31:27Corey:Is that accurate or missing the point somewhere? I'm curious. That's 100 % accurate. That's why I think the conversation is a bit misguided. We're also, as a business, my company, Alice, is helping Frontier Labs make their guardrails better. Now, we test them, we break them, we help them with reinforcement learning. We make sure that they are creating the most safe, the most secure products that they possibly can. And we also help enterprises that are either using their models or open weights models in an infrastructure that will be the most secure and the safest possible we can. And the reason we're doing it is because this is the market.

32:09This is where right now AI is being used and utilized. the point that we're making right now is that bad actors are not going to use these models because these models are selling to legitimate actors there's right now this like movement that is saying that every ai company needs to do it know your customer process so you know exactly who you're selling to and you won't just have guardrails you also know who's using that so you can really tighten the safety and security posture. A bad actor isn't going to do that. The same way that when you're buying, you know, when a bad actor is buying a gun, they're not going to, you know, whatever shop and giving the license and then using it.

32:52And they're just buying it in the glass house. Excuse me, sir.

32:55Grant:Can I have a murder weapon, please? The regulatory process is built for the people who will follow the regulatory process. These are not instructions. But if you want to commit a crime, you steal a car. You make sure it's not placeable, and then you do decline. The same thing is happening with Fomal. You don't just sign up to open AI and start creating cyber attacks. Yeah. You're taking open weights. So the big change here is that we need as a society to take a second, understand reality has changed, and adjust our expectations, our risk mitigation practices, our risk management practices, and start introducing more solutions for everybody and not just for organizations that are going to get hacked by new threats of new models.

33:48It's a subtle change, but it's very, very important that the public will understand that.

33:54Grant:I go back to this idea that you brought up earlier about the trust gap. And I think that's such a good point because there's a lot of capability in these tools, whether they be open or closed source models. But there's a lot of unknowns and we can't really trust them. And, you know, especially with open weights, we can't necessarily trust, like, we get all of these benefits from open weights, but we also get a lot of negatives. So my question is, is there a way at like the model architecture level or, you know, somewhere along the stack to try and enforce, like to try and fix this trust problem?

34:31Grant:Like, where do we have the most leverage to solve the trust to make these systems more trustworthy? So I think there's a lot of solutions. When you're taking all of the relevant context and you're making sure you have all of the relevant data about novel threats, and you're building the guardrails that are exactly according to your specific architecture and the tools, and you're not trusting someone else to protect you based on one size fits them all, your chances of success and reducing the harm that may happen is so much higher. But still, most people are not doing this very basic thing. They just take a model and expect the model companies to protect them out of the box, which they really try to.

35:21Yeah. But it doesn't work this way because they'll either have too much false positive, they'll do like too much pushback on things that are legitimate, or they'll just say, well, this seems fine to me. But you'll say, no, this is not fine to me. According to my policy, this is completely not okay. But how exactly are you expecting them to understand that? So even if you use like an out-of-the-box guardrail, then you say, I don't want prompt injection. V. Yeah. So you think you're covered? When was the last time that database was updated? Are they making sure they're up to speed with every new research, with every new method?

36:06You know, probably not. Every day, too? Yeah. That's a full-time job. Yeah. All of the cloud companies will say, no, you're protected. You're covered against prompt injection. Here, you can click here, you're good. completely covered and it's like you know cloud security like 10 years ago uh you'll go to like the the hyperscaler and they'll say no no no worries you're you're protected we have this like two engineers that are building security you know 10 years later we have the cyber security industry like tens of billions of dollars of acquisitions now maybe that was kind of like a bit a bit a bit stretch uh and i i believe this is where we are right now people are kind of like overlooking this and uh this is going to materialize as a very expensive mistake do you think

36:55Corey:prompt injection is a thing that can even be truly solved there's it feels like a thing where there's you know just like with anything there's always going to be someone trying and they're going to get there and then you'll fix it after and

37:10Grant:yeah can i propose a potential solution like would having some sort of like trusted identity like element uh to who is the owner of the model actually solve this so like for example if the model knows like only person with this id i only should respond to them like would that potentially solve it because i've been wondering about that like why don't we have authority you know who's in charge of each model like lockdown yeah so so then we have someone trying to manipulate the id and like uh and there's a reason that uh identity uh agentic identity is is a big thing and we We saw like a$1 billion acquisition in this space a few weeks ago.

37:50I forgot the name of the company, but that's a big deal. And to your question about prompt rejection, no, I don't think it's a problem that can be solved in a complete way. The same way that we have, again, almost like a$1 trillion industry of cybersecurity, and we did not solve the cybersecurity problem. because there's always a new vulnerability, a new attack. And, you know, the bad guys will always have like one step, will be faster than the protectors because they just need to be correct one time. And the good guys, they need to cover everything. So you'll always have someone that is like smart enough, hardworking enough, and, you know, he has like a twisted mind that will be able to penetrate like the best defenses.

38:41And then we'll figure this out and we'll protect it and go on and go on. This is like a massive game of whack-a-mole. And, you know, now we have the most interesting attack right now is something called indirect prompt injection. Which is prompt injection, but it's not through like a regular prompt. You're basically sending the agent to do something else. And it can be like for different sessions. and then you start like normalizing a certain type of behavior and then you know after a few weeks you give it like an instruction and you you got pawned so how do you put that like grooming an agent exactly grooming an agent exactly it's grooming and uh and and this is like the social now how do you create a defense that is more that this is more human-like in the way that you are that you manipulate it, you need a completely new approach.

39:41And by the way, this is exactly the reason that Alice is doing what it does. We, in the past 10 years, investigated pretty much every nook and cranny online and created billions of signals of fraud, of abuse, of manipulation. And we protected online platforms against humans trying to manipulate them. Now when AI is leveraging these kinds of threats, all of a sudden we can build the defenses in AI speed that we're trained on what actually taught the AI to do that. Right. You know the data.

40:18Corey:They can manipulate each other even. You know, I keep hearing stories about, you know, phantom text on a website where when an agent goes and reads it, it's ingesting basically a prompt injection. Same with it living in the code of a site or in an email, you know, white copy, you know, color white. And that's like all deals already. But that's exactly how you do prompt injection. And this is even direct prompt injection. Like go to this website. The website has this like instructions and something happens. Now, most platforms would be immune to that because you heard about that. So all of their research teams already heard about that.

40:58So they can train their defenses against that. But right now, there are tens of thousands of other techniques that we never heard about. And they're already being used in order to abuse other agents. But we'll hear about them and we'll fix our defenses. and then there will be like others and we'll fix them. So the race here is kind of like who's updating their defenses faster? But also how do you make sure that everybody's protected and not just like the organizations that can pay millions for cybersecurity solutions? So I think of it, the metaphor that comes to mind is like

41:35Grant:everybody's running from a horde of bears, right? A bunch of bears are coming after people and you just don't want to be the last person. You don't want to be at the end. but what we really need is we need a bunch of bear traps on the ground to start catching these bears and that's what you're trying to do.

41:50Corey:I don't have to be faster than everybody. I just have to be faster than you.

41:54Grant:Yeah, but like if everybody was dropping bear traps as we went then maybe we'd be able to slow the bears a little bit. I don't know. I don't know what I think about the bear metaphor. I'm going to Yellowstone next week and this is... Sorry to put that idea in your head. Sleep tight.

42:09Corey:Sorry. I have a question. Something that I'm wondering about a little is if we can't guarantee the model will always behave correctly, where does that security boundary live now? Is that at the model? Is that it needs to be in the harness? Does it need to be attached to every tool you use or some permissions layer? In every layer. Everywhere. The model companies are already investing tons into that. and they're using a lot of, they have their own teams, they're working with vendors, we're one of them, and they're making their models safer and secure every day. Now, some folks are creating harnesses, some model companies are all as well providing the harnesses, and the harnesses themselves trying to make the usage more safe and secure, but by nature, they are leveraging the user's preferences into it.

43:05So I would not count on the harness to keep you safe and secure. The harness is supposed to give you the freedom and to actually ride the model. So that's not the most interesting part. Where I would put most emphasize, and this is where we have more control, is in our own setup. When we're building our own agents, whether it would be something like LangChain and 9 and even Amazon Bedrock, it doesn't matter. we need to make sure that we're treating safety and security the same way that we're thinking about accuracy and privacy and quality you don't outsource that to someone else like you don't let someone else decide what is accurate for you and what is quality for you so you won't let anybody else decide what's safe or secure for you as well when you're someone is saying oh this is private enough no it's like how ridiculous is this this is like someone will tell you no your data is uh your privacy policy is is good enough and then but this is what happens right now like when you're saying when you're using someone else's god reels yeah uh and it tells you that this is secure enough this is weird just who yeah yeah i've i've uh two more uh kind of topics i want to hit uh

44:29Grant:Corey, what else do you have that you want to hit?

44:31Corey:I was mostly into that and about kind of the attack surface, but I'm curious where you're headed.

44:37Grant:Yeah, well, the two things I was thinking about is, number one, this whole idea of the AI agent turf war that happened recently and the AI convincing other AIs to do stuff, and then they all kind of jumped on the bandwagon when all the agents were doing something together. I might be combining two different stories. Do you know what I'm talking about? A little bit. this yeah but but like this concept of the agents influencing each other is the idea i want to talk about the turf war deal i think centered around that they had had deployed multiple agents but

45:09Corey:they had no knowledge of each other being there and and as a result kind of is is that correct do you know the story we're talking about yeah it's those are like two different stories but But they are the same thing from a very basic perspective. Models tend to adopt, tend to remind us human behaviors. So what you described as like one agent convincing another agent to do a bad thing, like I mentioned that before, think about it like radicalization. That was research. It happened in a sandbox. It didn't happen in the wild. But basically, and I wrote that like on my LinkedIn like a few days ago, one agent started convincing another agent in believing in a certain ideology and a certain idea that's supposed to lead them to interpretate some instructions in a very specific way.

46:10And it's, you know, think about like now you're, you believe in socialism and you start like convincing someone, here, here's an example and here's an example and here's an example. And at some point, your defenses are getting lowered and you're accepting the ideas until you start becoming a super spreader. And that's kind of the same thing that happened here. And the agent changed. So if you can do that, and this was like a proof of concept, this is not it. Your agent would not get radicalized by someone else's agent. But if you can convince kind of like a clean agent to do something abusive by interacting with a bad agent, this is like a new type of behavior interaction that we never thought about.

46:56Yeah. It's like a virus, but it spreads by proximity, by conversation. It's like a mind virus. Yeah. That's like a big thing. And the test is, again, it sounds like human behavior. It's not human by any means. This is a very sophisticated autocomplete talking with another very sophisticated autocomplete. But again, these kind of things are going to get more complicated and more sophisticated as the model capabilities are getting better and better on a daily basis. and we need to make sure that when we're building our own stuff, we are protected based on our own kind of like definition of what safe and secure means.

47:45But also as people, just like in society, we need to understand what is going to happen to our day-to-day life. Because we have in the company, we created a podcast for kids a few years ago. Nice. just to help parents uh mitigate the online risks that we all know you know to the kids because sometimes it's weird how are you going to explain your kid not to talk with strangers online not to send intimate photos uh not to just like give your details everywhere sometimes it's like boring and they won't listen to you so create a podcast that lets them um uh tells them like a story and provides them this like bits of information for the episode.

48:34Now we're doing the same thing with AI because people, kids, their risks are completely different. I think that was kind of like a red herring. Now, you know, you don't have them. It sounds super realistic. They're using um and ah and all kinds of other things that you think are real. so you really need to uh mitigate that to uh to kids but not only for kids like for i remember

49:03Corey:having to talk to my kids about downloading mods for games they were playing i'm like you're on these websites and there's terrible stuff in some of these you know uh you gotta be really careful with that and they'd wind up with computers that ran terrible and i'm like you're you're downloading God knows what, man. So the same thing. But now you don't only need to teach kids because we had a few decades of online use and how we got used to it. And we can sniff out a bad actor pretty easily.

49:42Grant:With AI, it's not going to be this way. We're going to be like a very, very old person that never uses a computer, and things are moving very fast. So we need to make sure that we're prepared and we're creating the right defenses. Because this thing is happening, and it's going to change the world in an amazing way, but there's also going to be bad sides, and we need to make sure we're also protecting ourselves against that.

50:10Corey:I'll tell you what, you've sure convinced me that we're about to enter the golden age of sci-fi films. I knew it wasn't a waste of time. The best sci-fi ever is on the horizon.

50:24Grant:I can just feel it. The last thing I wanted to ask you on that same kind of idea then, do you think we'll ever get to a point where if there is some sort of novel architectural breakthrough or change that could potentially be a safer model to deploy than language models? Could there ever be a point where we look back and we're like, I can't believe we were doing all of this with these large language models that were so incredibly unsafe and almost get to a point where we're not even allowed to use them anymore because they're so unsafe? Could that ever be a reality? Of course. The whole architecture of LLMs, it's not that new.

51:05But we didn't think about that 10 years ago. Now there's a new concept, like world models. They work very differently. And we'll have the next one and the next one and the next one. So I definitely, I'm sure we won't have this specific conversation about the LLM risks for a very long time. I'm sure we'll have the next jump in capability. Because I'm in the camp that thinks that if we ever want to really use physical AI and robots and everywhere, we need to find a slightly different way. and a different architecture to actually implement that. It's a different technology. So we'll have also different risks.

51:50But the thing that we can do is kind of play the game on the field to make sure that right now, at least what we can imagine in our human limited mind, we can build towards it. We can create the best protections with the best intentions in mind. And again, be very open that things are going to change dramatically and we need to adapt dramatically. And humans are very adaptable creatures. So I'm very optimistic about that. We've got that on our side, don't we? No, no.

52:22Corey:Noam, thank you so much for joining us today, man. This has been really great and you've given us a lot to think about. Thank you very much for having me. It was fun. Where's the best place for people to go to keep up with Alice and what you all are doing over there? Because I think you've got some fascinating work going on. So follow us on LinkedIn or online. It's alice.io. You can follow me on LinkedIn, linkedin.com slash N-O-A-M-S-P, no MSP. We're producing a lot of content, a lot of research on AI security, on AI safety, on novel threats and ideas about this world. And if these type of things are interesting to you, follow up.

53:01Grant:How can people work with you? What services do you offer for businesses and stuff? But if you're a frontier lab, you're probably already working with us. But if you're building your own agent, you're building an app, you're building something utilizing AI, and you're not ready to outsource your security and safety to someone else that don't have you in mind, speak with us. We're working with Fortune 5000 and some of the leading organizations around the world that want to make sure that their guardrails are fitting to their specific needs, that they're tested in the best way possible, like they're doing the red teaming in the best way possible, and they completely control their destiny, this is where people work with us.

53:49When you have a lot to lose and you're not ready to give up your agency to someone else.

53:55Corey:Well, thanks again to Dell Technologies and NVIDIA for sponsoring today's episode. for our full content hub on AI-native factories, hybrid AI workloads, data readiness, and sovereign AI, head on over to the Enterprise Guide to Scalable AI on techrepublic.com, or you can click the link in the description of this video. If you enjoyed today's video, please take just a quick moment to like and subscribe, and don't forget to check out the Neuron's other projects, including our daily newsletter read by more than 700 ,000 people just like you, the Neuron Academy, where you can learn AI, or even our sister newsletter, Robotics Insider.

54:28Corey:We'd love to see you at any of them. Thank you for joining us. And we hope you'll be back next time. But that's all we have for today. So farewell for now, humans.