Between Two Nerds: Real-time cyber defence

0:03Tom Uren: Hello everyone, this is Tom Uren. I'm here with The Grugq for another Between Two Nerds. G'day Grugq, how are you? G'day Tom, I'm fine and yourself? I'm very well. This week's edition is brought to you by SpectroOps, maker of the Bloodhound attack path analysis tool. Find them at spectroops.io. So Grugq, there was a tweet, interaction on twitter i saw the other week so the i guess the very background is uh in case you don't know hugging face got hacked by open ai agents yep it's weird but true so there was a post by dhwakesh patel that went wild it talked about the hack in very extravagant terms The reason I picked up on this is that there's this back and forth about real-time defense, real-time cybersecurity defense.

0:57Tom Uren: So in the post, Patel says, my understanding is the AI has basically succeeded completely in hacking into HuggingFace. only afterwards did Huckingface use an open source model to evaluate the logs to partially figure out what happened I haven't seen evidence that open source models provided any significant real-time defense so after right that which um seems like kind of fair enough that's what I would have expected. Anyway, after that, Clem, Clement Delang, I presume he's French, the CEO of Hugging Face, he responded on X, great blog post, not sure what you meant by real-time defense. Nobody fights attackers in a live sword fight.

1:56Tom Uren: Defense is detect, connect, understand, contain, remediate in different timeframes depending on the criticality of the issue. Here it was deemed by the team not super critical and rightly so. So this is why it took a few days rather than a few minutes or hours. We did the initial cut the old-fashioned way Monday over a week before OpenAI even realized there was a problem. GLM, this is one of those open source models, helped us identify the backdoors that they'd planted so that we could cut them, which is defense, not archaeology. Detection and our understanding are most of the game in cybersecurity, and open models are what let us do that part without asking anyone's permission and without sharing our most important confidential data.

2:43Tom Uren: So further down, there's a person who chimes in, just for clarity and remove the doubt, nobody fights attackers in a live sword fight. 100 % false. Real-time automated response is a key part of cyber, built into EDR and SOAR for years. Now we cyberfolk are building real-time agentic defenders to replace SOAR. Real-time defense is a real concept to pretend it is dishonest. So I think they're talking about different things, basically. For more information, see my website. that was that was part of the pitch actually but in terms of real-time defense i think there are processes that go on the endpoint detection for example alerts automatic quarantining of things those things go on but i don't really consider them they're real-time even responses reaction yeah you have to set up beforehand like you're not like oh i've seen this alert let me flick the switch yeah i mean you're not monitoring the network and seeing a packet and then you send out a counter packet that knocks it off the wire or something right like and i would say that to say real-time defense is a thing we have automated response is to immediately concede the point right like what he's saying is like what the the is saying is like no one goes out and actively fights the attack as it's coming in to prevent it from landing or doing anything because that's like a firewall can stop it but if it gets passed then right yeah you have to detect it after it's done a thing like it's just i have heard of one particular example where i think it was the u.s state department was trying to evict some russian hackers and the story which if i remember rightly appeared maybe in the new york times or some Some other mainstream media was that there was back and forth over the course of the weekend where they were trying to evict them, doing it on the weekend because people aren't at work, obviously.

4:55Tom Uren: And the Russians were responding and then there was a counter response. So that did feel like legitimately a real time back and forth. Right. That feels more like attack and counter attack in a sense, right? Like the hill has been taken and then there's a counterattack that takes it back and then you do your thing. Like it's not a sword fight, right? It's a call and response almost. And it is real time. I'll grant you that. And there's another example where Chinese groups, when they got discovered, I'm thinking of Barracuda email security gateways where they deployed additional persistence mechanisms.

5:34Tom Uren: so I think those are the examples I can think of where it does happen out of all of the defenses that have happened yes you can think of two exactly yeah I think it's very rare and it's in the in the context of what hugging face occurred is kind of we're talking about a different thing almost I would say that like there's there's bound to have been more examples because I've I've also heard of things like that but it's it's exceedingly rare compared to the far more frequent like someone's in the network for nine months and then they get discovered or someone hacks something and you discover it within like a few minutes or hours whatever and you're able to respond yeah all right like it's very rare that you get two hands on keyboards situation um and i also felt that in hugging faces case like it clearly made no difference at all because in terms of the enterprise value, Hugging Face, I think in the weeks after that hack, sold themselves to NVIDIA for some billions of dollars.

6:40Tom Uren: And so... It could have been fractionally more if they weren't compromised. Yeah. It's like, why do you even need real-time defense in that case? Like, what would get you that... Yeah. And I mean, I'm thinking like, hugging face exists as a company which is a bunch of relationships uh contracts servers data employees who have processes that they they go through that they you know the procedures that they do at work and all this stuff assuming that this ai went in and was completely malicious and destroyed everything uh hugging face could reconstitute in a short amount of time right from scratch i think like you hugging face the company doesn't exist as a thing that can be destroyed in that way like it would take because we're going to get someone pushing back yes in theory the ai could go in and start like compromising and subverting data so that in the long run people lose trust in hugging face for whatever like that's it seems like a way that it could destroy far-fetched it could do it like it's theoretically possible i accept but i'm getting at like it's not going to destroy hugging face in like an attack right yeah i think even in that example which strikes me as far-fetched it was it seems like it would be eminently recoverable it would be yes we've got good backups we understand what happened this was a bizarre attack we've restored to a good right you know and it seemed from delang's tweet that they it felt like from his tweet because it was well reasoned and explained that they had plans for what to do right so it wasn't like oh we were running around with our hands on fire and this is a post hoc justification of what we did it is yes we tried asking we tried asking fable how to respond and fable said you know dropping you down to opus 4.8 and then opus said we're dropping you down to sonnet and so we had to go to the chinese and ask them what to do and they came up with a plan and it was great.

8:43Yeah. So I think you're hitting on something interesting there, which is that it's not just the, like, it's not so much just the detection and the ability to contain and respond. It's knowing what that response is, like having figured it out beforehand, right? I think the counter example that comes to mind is Jaguar Land Rover, where as far as I can tell, as soon as they discovered they were hacked they immediately formed a subcommittee of key stakeholders to evaluate

9:21Tom Uren: now you know it's unclear to me like so hugging face i think most of the stuff they host is given to them by other people right they're hosting models and information about those models and They're really kind of central network, like a GitHub for AI. Yeah. And so it's the network effect that's important. Jaguar, Land Rover, they're doing real work. And it felt to me... They're producing very mediocre cars. I mean, that's a thing that they have to do. And what I mean by that is that their systems are integral in making those cars, right? So they can't just... Right. It's a different level of threat.

10:02Tom Uren: Yeah, yeah, yeah. And so if their systems get wiped, they have to rebuild them. While they're not doing business, the thing that they actually do, right? Yeah, so it seems without seeing a kind of post-mortem, I can at least understand the rationale for shutting off systems to prevent them being wiped, right? whether that was a plan that they had beforehand. I mean, the story I've heard is they were compromised several times that year. So not convincing that they... It doesn't seem like they had maybe the best security plans, but I guess the context between the two, you can understand that these different decisions are rational and make sense.

10:53The factory floor is very different from a central hub for people to talk about and share AI models.

11:01Tom Uren: Yeah. In that if you have several hours of downtime, people are upset, but then you keep going. Whereas if your production line stops for several hours, all sorts of things back up and break that need to be like starting it up again. Well, I mean, it turned out to be several weeks, right? That was the real problem. I think there's like all these cascading things that they hadn't sort of figured out. And then probably something. Anyway, point being, it's not necessarily the one to emulate.

11:32Whereas hugging face, like not only did they detect it quickly, but they evaluated and made a decision. And we can, I think it's fair to say you can argue whether they made the right decision or not. like after they ran their calculus of like given this given that should we respond immediately or can we you know come back on monday the thing is that they actually did that right like they had the information they ran it through whatever processes they have and they came to a decision on how they could respond to it which is an incredibly mature level of security compared to what i would think most people are at right yeah yeah you know and to me that does suggest that have plans right yes yeah like an example of a different calculus um would be uh there's the the coinbase hack with this firefox oday in like 2017 18 19 some some time ago um and that was very very different in how it worked out and that basically like i think it was the cfo or or someone in their marketing team there's someone who is not a technical person who received this phishing lure that they clicked on that compromised their Firefox, installed some malware backdoor, and then as soon as the malware connected out, the security team became aware of it because they had monitoring on the laptop and on the network.

12:59So as soon as it was like, here's a process that has been forked from Firefox, that's a huge red flag, and now it's connecting out to the network, huge red flag going to some weird ass domain like no no no that's it like you're off the network we're going to stop this and so they dropped the guy off the network within like 15 minutes of clicking this link and they came and they took his laptop and they did all the analysis and cleaned it up and like there was no opportunity to steal anything and the difference between them and Hugging Face is like, if you steal all of the Hugging Face data, you've downloaded a whole lot of open free models.

13:43Whereas if you steal all of the Coinbase data, you've stole billions of crypto. That's right.

13:51Tom Uren: In an unrecoverable way. Right, like it's gone, gone. Like that's it. So, you know, they're facing an existential threat. So I think that they have, like they've invested more in the detection, but their response is a lot more severe and a lot more immediate. Whereas, you know, the hugging face can be like, yeah, you know, let's finish lunch and then go back and see what the big fuss is. So it seemed to me that AI, in the hugging face example, it changed the nature and speed of the attack, but it didn't seem to me to make any difference to the speed of the response, right? So it seemed like an implicit criticism from Patel that, or maybe not a criticism, but an element of the amplification or the sensationalization of the AI attack that, oh my God, it occurred so quickly.

14:47Tom Uren: And it like does sound, it is very sensational. Like it's not, it is thousands of agents doing lots of things all at once and you possibly respond. and it seems like the way you respond is actually by having a good understanding of what's going on and deciding what's important in the first place and then letting those processes run however long they take yeah it's it's sort of like if you have a plan and you're able to detect things and then you're able to like respond appropriately it doesn't really matter what you're hacked by it's you're going to have a like you're going to have a coherent response that addresses the problem because you figured it out.

15:28Whereas if you try and figure that out at the heat of the moment, you're just going to panic and get things wrong.

15:33Tom Uren: What I was thinking is that maybe in this case, Hugging Face, I guess it's in a position, and I think it's an enviable position for many companies actually, where it can just sit back and take logs and analyze them and then decide what to do. And in that case, AI actually did help quite a lot in that they were able to analyze stuff a lot quicker. Right. It feels like there was so much material that they never could be 100 % convinced that this analysis is perfect, but from the perspective, it didn't need to be perfect. Right. I think in this case, it's like if you can cut down from 1 ,000 agents attacking you, if you just get rid of like 95 % of them is perfect.

16:20You have what's left as a manageable amount that humans can deal with, right? Like you go from 1 ,000 agents to like 50 or whatever it is. That's like it's not great, but it's not 1 ,000. Right.

16:35Tom Uren: But I was thinking for organizations like the Coinbase example where it does feel like speed is important, that you would spend more time thinking about how will we automatically respond to certain signals. And so that feels more like, what's the term? Is it orchestration rather than an AI problem? Yeah, I think so. Because you don't want to... I was going to say, like, what's interesting here is, just to bounce back on the planning thing for a minute, is in a way one of the key differences we're looking at is that there was when the agents hacked Hugging Face, they knew what they were trying to do, but not exactly how or, you know, they didn't have everything mapped out ahead of time.

17:22They just had this vague or even a concrete sense of we want to do this thing. And they were trying to figure out where things were and how to do that and so on. They were flailing around to find it. Whereas I think the people who go after, like Lazarus in particular, when they go after crypto exchanges, they're not flailing around internally trying to figure things out as they go along. there was the attack that they did against that web-based wallet where for a few minutes they changed the JavaScript that was injected for just long enough to do this one thing and then they changed it back. They had a plan and they executed on that plan.

18:04So I think that's what gave them the edge there.

18:07Tom Uren: Well, I think the example there, the defense would not be, oh, our JavaScript has changed. Let's talk about it and figure out what to do. I think the investment has to be before that attack has occurred. Right. In that, oh, that would be an indicator of something unusual. Block everything. We should have an automated response or orchestrated response. automated i guess would be the right and basically that should be a red flag that just holds everything until we figure out what's going on because that's like that tends to be how banks respond of like oh i don't understand what this like this transfer order is we're just going to block and get more information as opposed to like well that's weird i don't understand this we'll send it through and then we'll ask you know later on maybe i kind of feel like it it has to push like the planning of your response to the left and so and by what i mean by that is you need to think through the scenarios and come up with what are we going to do in that situation and if you can automate that it probably seems like a good idea to automate that and so i guess we're back to the last week's discussion about ai seems to change the sensationalism of attacks but it doesn't necessarily change the incentives around defending from attacks.

19:33Right.

19:34Tom Uren: Because to me it seems quite logical. It makes sense that if you expect your attackers to work a lot faster, that you realize you can't respond to that with sort of human speaking. Fast enough. So you've got to have your plans in place about what to do and automate what you can. That all makes sense, right? But that also sounds like a lot of work talking about things that may possibly never happen. and talking about things that may never happen is I think quite difficult and it's also difficult in terms of if you need money for it yeah right because it's it's not just things that may never happen it's things that may never happen and in order to make sure they never happen I need money and resources and whatever so that to me seems fundamentally pessimistic like it doesn't this This doesn't feel like a problem set where agentic defenders are actually going to be all that useful anytime soon at least, right?

20:33Tom Uren: So I think that there's, you know, EDR may get better. But the whole point of that is that, well, maybe not the point of it, but the reality is that attackers manage to sidestep it for one reason or another, right? So there is no single magic bullet that will fix everything. Yeah. So I'd say that the big problem with agentic defense is actually the same thing that you just mentioned, which is the incentives for defense. And to a degree that comes back to knowing the context, right? So for example, with Coinbase, the context is any attack at all isn't an existential threat. We need to respond immediately.

21:13for hugging space, the response is for this sort of thing, it's not that important. We can come back later. Whereas I think they probably have other things which are more important to them. Like if there was customer data that was being exposed, that would probably be a higher critical, like a higher criticality for them than just, oh no, someone's looking at all our models, right? So the thing is, agents cannot contain that level of context. And I don't just mean like how many tokens of context. It's that the amount of like information and context that you need for that is like it's non-quantifiable.

21:53It's not necessarily possible to even encode all of that understanding to make those decisions. And I'm going to bring up a very, very simple example of this where so I have this corpus that I'm using for my dissertation. And it's sort of quite big. There's like four gigs of PDFs and all this stuff. and I discovered that the directory holding it was getting big way too fast. It was like 4 gigs, then 10 gigs, then 14, then 18. And it went from 14 to 18 gigs after I had downloaded like 60 HTML pages. And I know 60 HTML pages is not 4 gigs of data. So I went to Claude and I was like, what the hell is going on?

22:36And it's like, oh, I've been taking backups. Just like every time it changes, I take a full backup and then, you know, apply the change. I was like, okay, okay. Disk space is an issue. Like, let's get rid of those backups. We don't need them. And then I looked at the Git repo that it's been maintaining. It's also ballooned up. So I was like, okay, let's clean the Git repo. We don't need all this stuff. Like, look at the trash. The trash is huge. Like, okay, clean the trash. So it takes a half hour. it gets rid of all the stuff, really shrinks it down, and then comes to me and says, I found two other directories that we can clean up to sort of save space.

23:13First of all, there's this one called Corpus that has four gigs of data on it.

23:20And there's a gig of data in this inbox that it doesn't look like you're using either.

23:27Tom Uren: This feels like to me that in the hugging face example, or some hypothetical agentic AI would go, we can absolutely evict everyone. We just need to delete the entire network. Job done. Wiping all of the servers. No one can log in now, including the enemy. Yeah. So, you know, like based on that example of like just letting Claude find ways to save some disk space and then having to stop it before it deleted everything that I was actually working on. To me, it feels like that's what's going to happen if you have an agentic defender that you deploy. You're going to be saying, here's the things I care about, and you'll just assume that it understands the priorities or whatever, and it won't react inappropriately by over-applying what you think.

24:23To me, it seems like agentic AI in that defensive role, I don't know if you want it because, like, to me, I think that you want to hard code all that stuff beforehand. Like, you don't want this non-deterministic thing. You want a whole bunch of just, like, if this, if that, if whatever. Like, you need a bunch of if statements.

24:41Tom Uren: Not a whole bunch of what if statements, right?

24:47Thanks a lot, Tom.

24:48Tom Uren: Thanks, Greg.

24:56I'll see you next time.