Soap Box: HD Moore talks OT security, frontier fearmongering and more

0:04Patrick Gray: Hey everyone and welcome to this special Soapbox edition of the Risky Business Podcast. My name's Patrick Gray. For those of you who are unfamiliar, these Soapbox editions of the show are wholly sponsored and that means everyone you hear in one of these Soapbox editions, they paid to be here. But that's okay because we still get to talk to very, very interesting people despite this being sponsored content. Because today we're chatting with HD Moore, who's a bit of an industry legend. He, of course, is behind Metasploit. He released Metasploit back in 2003. We actually talk about that a little in this interview, actually.

0:41Patrick Gray: But he's probably best known today for being the co-founder and chief executive of RunZero. Now, for those of you who are not familiar with it, RunZero is like an asset and systems-like discovery platform, which can go out there and really find all of the devices that are connected to your network, servers, desktops, IoT, whatever it is, and really paint an accurate picture of what it is that you actually own, which is obviously very useful to a security program. And RunZero has actually just recently been acquired. Like the transaction closed very recently. I should disclose to that I am an advisor.

1:22Patrick Gray: Well, I was an advisor to RunZero, which meant that I received a payment after this transaction closed. But yeah, they got bought by Accenture alongside a company called NetRise and also Dragos, which does OT slash IoT security stuff, right? So that begs the question, why did a sort of enterprise IT discovery platform wind up being gobbled up and sort of placed together with a very large OT security company? So as part of this interview, we'll be chatting about why RunZero and how RunZero gradually became a product that does a lot of stuff relating to OT. We talk a bit about what the vision is for the sort of combined entity that is being created.

2:11Patrick Gray: But we also have a really fun conversation about some other stuff. We talk about the Volpocalypse. And we also talk about, I guess, you know, how this AI Doom stuff. We've been there before, right? So 23 years ago, when HD released Metasploit, people were calling him a cyber terrorist and saying he was giving machine guns to toddlers and stuff like that, right? So we've been around for the hype cycle, him and I both, once before. So we have a bit of a chat about that. But look, I'll drop you in here where HD sort of explains where RunZero came from and how it got to where it got to. Like why it wound up being something that was so highly relevant to OT and IoT networks.

2:56Patrick Gray: So here is HD more.

2:58HD Moore: Kind of going back to basics, the reason RunZero started was we're all sitting around the pool in Vegas, what, eight years ago. chatting about this stuff. And the idea was like, you know, hey, folks have kind of abandoned network discovery. They've kind of assumed that you can, everything on the network is managed and everything unmanaged doesn't exist. And the problem was that unmanaged stuff, that unknown stuff kept getting hit again and again. That kept being the kind of Achilles heel of networks. It was the stuff that the security team had no idea that existed that they didn't have an agent on.

3:23HD Moore: So, you know, going forward, that became more and more of Run Zero's scope became OT because that stuff is mostly unmanaged. It's PLCs, it's random industrial routers, things like that, things that don't run your CrowdStrike agents that don't have any other kinds of visibility into it outside of maybe past discovery. Even then, you don't really know what's on the network. You're only kind of seeing where you happen to get a span tap or a cacher. So in that sense, like Brunswick got really kind of deep in the OT space in terms of helping folks who had no idea how those networks even configured, figure out what's out there, how they're wired up, where the ingress points are.

3:53HD Moore: And that stuff really kind of hit the fan in the last year or so. And where that kind of dovetails with, you know, corporate enterprise is that every enterprise out there has some kind of equipment like that, even if they don't realize they do. It tends to be building management systems, your HVAC controls, everything's like your building access systems, the things that you don't think of as IT products, but nobody else thinks of it as theirs. That's where you tend to see the biggest problems. So places where you typically have your real estate firm managing IT are where a lot of corporates get hacked because it's like, okay, well, I don't manage that.

4:19HD Moore: The building people manage that device, that controller, even though it's on your network. Yeah, that HVAC.

4:24Patrick Gray: I mean, that's how Target got owned like a decade ago or whatever. It was actually the HVAC controller, right? Like that has just admin password or whatever. And that was, but I mean, like, you know, back then I didn't understand why that wasn't on its own VLAN. Right. And, and look, I got to stop being surprised and disappointed, you know, cause I've been at this for 25 years and yeah, I got to stop being surprised and disappointed by that because when given a choice between convenience or a secure, sensible architecture, that's not really that much more work. People just tend to do the easy thing.

4:55HD Moore: It's gotten really bizarre. So one thing we did a couple months ago at Run Zero was we put together protocol gateway discovery so you can find devices behind other devices, even if they don't speak IP. So BACnet controllers are a really good example. That's mostly your HVAC. You connect to a thermostat, and that thermostat lets you talk to everything else in the BACnet bus, which can include your fire, safety, sprinklers, all that kind of horrible stuff you really don't want attackers to get to. So we did an internet scan. We found about 5 ,000 devices directly on the internet, but behind them, we found over 110 ,000 devices that you can control.

5:23HD Moore: So the stuff with the internet was pretty boring. It's like thermostat. But the stuff behind them is actually really important, like door locks, relays, sprinkler systems, all the safety systems, the early warning for shooters, active shooter drills. So it's pretty wild how much of that stuff is out there. And nobody had looked for it, effectively. Evan says, OK, there's a thermostat in the network. I don't care. Not realizing that one layer behind the thermostat was everything else you do really care about.

5:44Patrick Gray: Well, it's interesting there that you said that that's not even IP, right? because some of the earlier work that you did that was very interesting with Run Zero is you could discover devices that were UPnPing their way onto the internet, for example, right? So you had a lot of people doing discovery who would say, give us an IP range, we'll scan it, we'll find all of the IP cameras. Now, of course, at a time of war, etc., like this has become quite an important thing that a lot of countries care about. They want to find insecure IP cameras that their adversaries might use to do airstrike battle damage assessments or whatever.

6:19Patrick Gray: So it's important stuff. So a lot of these discovery places would just look for Hikvision. And, okay, we found them. What your stuff would do, though, and it was really interesting when you looked at the scan results, you'd see Hikvision running on Cisco. And it was because it had UPnPed its way out. So you've always been really good at doing that extra hop. But that was with IP-based stuff, right? So could you tell me more about how you were able to go from a thermostat through some sort of question mark protocol and then enumerate stuff and connect to stuff behind it? Like what protocols do they use?

6:52HD Moore: Yeah, the two that are most common for HVACs are, in the States at least, it'll be BACnet. And it's basically almost like a serial bus. Like once you connect to the first gateway, you can just enumerate devices by ID number and just kind of go through that. And if you find another device in the network, you could then keep routing through that and change all you want to. On most European networks, it's called KNX, and it's a little bit different. It's not quite as actively routed. In both cases, you're establishing yourself as a remote foreign router to the network and then routing your own traffic into it over these bizarre protocols.

7:19HD Moore: On the OT side of things, it's mostly Ethernet IP slash SIP. And that protocol is really neat because not only can you do serial chaining, but you can do serial to IP to serial to IP to serial to IP and keep working around a network through a combination of Ethernet ports and serial ports and just relay as long as you want to. You get to the point where you're making like 20-second connections balancing through five devices. And in fact, you can turn a lot of Ethernet SIP devices into like remote port scanners or subments you can't directly route into. And it's a lot of fun.

7:44Patrick Gray: It's really funny because I can guarantee you right now that you are blowing up. Well, not blowing up, but you are being indiscreet about someone's tradecraft right now, right? Like I reckon if there was some sort of leak out of NSA a la Snowden again, this is the sort of stuff you would see. I mean, would you think so?

8:00HD Moore: Yeah, I mean, it's not like it's hidden. It's just that people haven't implemented tools for it. Like you don't see any commercial vendors implementing anything beyond the first step of these enumerations. Or if you do, you see them being very structured tools by the HVAC company to then monitor your equipment. But there's no one really putting those two together. So if you look at like a lot of the team management -

8:17Patrick Gray: Well, there's no one doing it publicly,

8:17HD Moore: I guess is my point. Oh, fair enough, yeah.

8:18Patrick Gray: I'm guessing that like, you know, and no one's sort of snapping this sort of stuff in a forensics context, but I'm guessing people are using this in the wild is what I'm getting at.

8:26HD Moore: Yeah. And we kind of go into your earlier point about finding unexpected things, like UPnP devices, cutting a hole in the firewall by default, it was a huge thing. It's still a problem today. A lot of your HikVision, your other cameras will automatically cut a hole in the firewall and UPnP forward by default, whether you do anything or not. So one thing else we're finding more recently with a lot of the OT attacks, that devices are being hit through cellular router connections that no one knows IP of because they're on a public APN through a telco. So you never know what IP they're going to pop out of in the first place.

8:52HD Moore: So how do you even find them, right? So I think we've been spending a lot of time on Run Zero is being able to scan the inside of the network, fingerprint if anything uniquely, then scan the entire internet and match the two up. Say, hey, this device inside is a device outside. Even if we don't know what range is yours, we know it's yours because we can match those two signatures.

9:06Patrick Gray: Yeah, I mean, this is something you've done a lot of. right? Like even doing stuff like matching, you know, key fingerprints inside and outside, right? By scanning the whole internet and then doing that targeted scanning of like, yeah, internally. Yeah.

9:20HD Moore: And we can be pretty lazy about it. We can pull a lot of data from existing third-party data sets. So for example, Shodan has about nine of the 10 fingerprints we need is already in the Shodan data set. So we just pull them out to start with. Another fun thing about Shodan's data set is that there's holes in it and the holes are really conspicuous. Like you'll try to figure out, well, why doesn't Starlink exist in the Shodan dataset? It's a really good example. And then what does Starlink have exposed that is not a new public dataset because it's been excluded from all the public ones? So an example of interesting, I'll get to the punchline, 4 ,000 Fortinets is what's there.

9:51HD Moore: But very interesting. Yeah, so the Starlink IP space is almost all Fortinet gateways, and it's not in Shodan at all, which is kind of a weird confluence of things. But the stuff like that you run into when you start doing a inspection of this stuff and say, okay, here's what we expect to see. here's what's actually in these different third parties. Let's just scan the difference and figure out where they went.

10:08Patrick Gray: Now look, I just want to switch gears here for a second and talk about vulnerability scanning because you did a big sort of pivot into, well not a pivot into vulnerability scanning, but you added a lot of features to RunZero that made it quite a useful vulnerability scanner. You did this great talk at a Decibel event. I was there in San Francisco next to RSA where you pointed out like the incumbent vuln scanners basically haven't changed the way they do anything for 20 years. And they don't really give you much useful information. They just dump a bunch of CVEs on you. And that's one of the reasons vulnerability management is such a mess.

10:46Patrick Gray: So the idea with Run Zero is you could scan, you could find stuff that attackers can actually reach, you can get a better sense of the impact. But it feels like now, even that is just untenable. vulnerability management has just got completely untenable because of this AI driven vulnpocalypse. Right. And you and I, you, you and I have talked about this, like it's got to the point now where people are just getting owned by stuff that doesn't even have a CVE. So, you know, how is vulnerability scanning supposed to save you? Right. So it's, it's, it's, you know, we've swung back to discoverability and control, but walk us through the, you know, walk us through the current state of exploitation out there, if you would.

11:27HD Moore: I mean, the dirty secret is CVE hasn't mattered in a long time. Most of the bugs people are exploiting don't have CVEs when they're being exploited. So the, you know, bonepocalypse will is accelerating that. So you may see, what is it, like 65 ,070 CVEs so far this year. That's not including the 200 ,000 in the wings right now. They don't even get a CVE yet. If you look at the stuff that Anthropic presented, for example, they said they found about 20 ,000 in one point. But long story short, about 83 have got a fix patched. About 130-something have not been fixed yet, but have been disclosed and accepted by the vendor.

11:54HD Moore: And there's a giant pile of ones that the vendors have not accepted in the first place. So I can tell you, we did a disclosure process for OpenBMC, which is kind of the underpinnings of all modern BMC implementations these days. You know, Dell, iDrive10, Supermicro, newest boards, you know, Huawei, Huawei3Com, or sorry, Huawei is using OpenBMC, but Huawei3Com is using, or H2 is using OpenBMC. Long story short, we found an auth bypass directly to privacy relations. So you go from zero to root on every device that's running this BMC stack that is now the most popular stack in the world for managing every server out there, including the hyperscalers.

12:26HD Moore: And we announced it, or we told the vendors about it 60-something days ago. They did not patch it. We disclosed it a couple days ago. No one even noticed. We've been reporting vulnerabilities in run zero for this thing for almost since 60 days ago. We've been telling customers about it, hey, this is something you need to turn off really quickly. There's an exploit coming out for it ASAP. No one cares. And more surprisingly, the vendors who are downstream of OpenBMC, the commercial vendors that rely on OpenBMC for their own stacks, They don't care until it's a patch available. So we've got CBEs for it.

12:54HD Moore: But again, even in the case of a CBE that we've been telling customers about for almost two and a half, three months now, is not really on anyone's radar. So in a lot of ways, and this is just one of 40 bugs that we're trying to get fixed right now. These are the only two that have been published of the set of 40. So it's a nightmare out there. I've been looking at working with other folks in the kind of home discovery space. And, you know, everyone's sitting on hundreds of thousands of bugs. Most people will never get a CBE. Most people will never get patched. So the question is, okay, if CBs no longer matter and exploitation is only an example of what happens when something is exploited enough that someone else notices, like put it this way, most of the volumes going forward will be exploited exactly once in a linear infrastructure and that's it.

13:30HD Moore: Like no one's going to go burn their good bug by sending it across the world.

13:34Patrick Gray: Well, you don't need to either. You just got to find some enterprise crapware, right? And then throw tokens at it until you get a bug because it hasn't been QA'd that well. So like, you know, I was talking recently about how I think where we're going to land is say you're an airline, your ticket booking website, that's going to have a lot of tokens burned on it to look for bugs, to make it secure, to refactor it, to make it as good as it possibly can be. And everything else that was touching the internet from your org, you're going to try to get that off the internet. You're going to go back to like zero trust network access principles.

14:03Patrick Gray: You're going to, you know what I mean? Like it's all about like attack surface reduction at the moment. But what you're saying, seem to be saying right now, is that there's not much that can be done, right? Like, even if you've got the run zero vuln scanning and everything, like, people aren't just, they're just drowning.

14:20HD Moore: Yeah, I mean, the important part isn't that you've patched everything on the CV side, because that doesn't really matter anymore. The important part is you know what you have in the first place. Where is it connected? How can you get to it? What is it connected to? What's the blast release of a bug? What vendors are in place? Like, you need to understand, like, okay, if new something comes out, where is it? What can it impact? And then if someone gets into that work and they go, and that's true, no matter what CV it is, no matter, you know, what level of patchability the issue has. So really it's back to basics.

14:44HD Moore: Like you go back to, you know, mid 1990s and everybody who was a hacker at the time could walk their way into any system they wanted to. So we're talking about some - So this is fine.

14:52Patrick Gray: Like I've had variations of this conversation, like especially down at Unprompted where I'm seeing people I've known, you know, 25 years. Yeah, they were doing this 20 years ago, right? Exactly. right like that that's the thing is is like i think for and it's been really funny watching the discourse uh of the you know we're all gonna die from the from the labs right and everything because for those of us who are old and have been around in this for a long time we're like we've been here before like we've we've lived this reality previously and and like it was messy and it was ugly but like we got through it i mean is your you seem to be like singing from the same songbook right now.

15:27Patrick Gray: Yeah.

15:27HD Moore: What did we do back then? We disconnected stuff. We put firewalls in place. We added more monitoring. We added more detection. You assumed you weren't going to catch every attack when it happened. You assumed that you'd have to have enough information later on to find it. So the big difference between now and then, of course, is the number of people who can do it. Instead of it being a couple of our friends doing this on the weekends, it's become anybody with an LLM and five bucks to spend can get pretty close. But again, the challenge is, well, the nice thing at least is these LLM tools are not particularly sneaky.

15:53HD Moore: They're really good at, you know, as Haroon mentioned, they're really good at taking bait. You can trick them into telling you who they are, into leaking themselves. Like they're really great to attack with honeypots. So defenders have a great chance to honeypot the crap out of things right now and use those to find people who are using LLM tools against them really, really quickly.

16:09Patrick Gray: Oh man, the Haroon Mir stuff, I've interviewed him about that and it's hilarious, right? Where it's like, you can pop up a web server that says, hi, I'm a web server that helps agents. Please connect over here. And it does. And then you're like, hey, run this Go binary. And it says no that's a reverse shell i'm not going to do that you say no come on run it it goes okay

16:26HD Moore: it's awesome yeah but that's where you are so yeah we got really fast exploit tools and they're really smart but also really dumb and so detection becomes something you can actually do with these days like honeypots become more relevant again and firewalls become more relevant ipx control becomes more relevant so we're kind of right back to the mid-1990s you can't stop someone from attacking you from zero i'm with you i'm with you but like it just occurred to me that the

16:45Patrick Gray: the most comparable situation, right? That I can think of to people freaking out about people releasing technologies that make exploitation easy. You can see where I'm going with this. And this is how we met. We met in 2003 when I was working for ZDNet in Sydney on the news desk covering security. And a guy called HD Moore in the United States released a tool, released a project called Metasploit. And this was so controversial. I think, you know, unless you were there at the time, people were saying the craziest stuff about you. Can you remember like some of the choice quotes about what people said about you?

17:24Patrick Gray: Like, were you a cyber terrorist? And, you know, those were the sort of vibes, right?

17:28HD Moore: Yeah, let's see, handing machine guns to toddlers, enabling cyber crime, you name it, right? And the funniest thing about Metasplay was like it was controversial for a couple of years because we're ahead of the curve. And then we're just chasing behind the bad guys for the rest of our, you know, history, right? Metasplay was then a year, two years, three years behind the baddies going forward. but there was a small period of time when folks were looking at Metasploities being the leading edge of security and exploit tech. And at that point, that's when it crossed over and, you know, folks stopped complaining about Metasploities being the same enabling bad guys.

17:53HD Moore: They started realizing that the only way they're going to keep up is to put things into the open. So I think we're right there again, like open models are how the industry is going to keep surviving. And, you know, screaming for regulation is not helping anybody. And it doesn't matter if you want to pace yourself or not, the world's going to keep on hacking. And so that's where

18:10Patrick Gray: but i just realized like it really is like and i i hadn't even thought of it that way uh before being on this call and like talking about it with you but it really does have the vibes from that time has that occurred to you previously i'm guessing it has yeah it's all that especially

18:22HD Moore: the more last of last week or two with anthropics and you need to pace yourselves and so on like come on man like look back at the history of this like open always wins and acceleration always wins you can't say no no guys let's slow down like yeah yeah yeah technology technology doesn't slow down

18:36Patrick Gray: for anybody, right? But like, I'd also note that like none of the people calling for a slowdown and everyone who's like inciting panic right now, none of them know anything about cybersecurity, right? Like they are from completely different fields. And I think that's, it's wild, man. Cause I get asked, you know, I get texts and messages and stuff from like some pretty like influential people and asking me for my opinion on this stuff. And I'm like, man, we all just need to chill. We need to chill. There's going to be, it's going to be bumpy, right? it's definitely going to be bumpy but you know we're not all going to die get a grip i mean i

19:12HD Moore: think that's another parallel with security is that ai has the same kind of like mindless panic that cyber security did you know everything like it's like it's like i keep thinking of the satanic

19:20Patrick Gray: panic from the 80s right it feels like that a bit right and it's worth it like it's it's worth that

19:25HD Moore: to the commercial firms to incite that panic we're seeing it because it actually drives sales yeah anyway that's interesting sorry it's okay like i love that we've just taken a bit of a bit

19:35Patrick Gray: of a trip down memory lane there talking about this. But look, while I've got you here, you know, why don't we talk a little bit? Because you've actually shipped a bunch of new stuff. Talking about technology, you haven't paced anything. You haven't done any pacing. You've released a whole bunch of new stuff, new OT stuff. I should point out too that like, I think one of the big Run Zero innovations, underappreciated, mind you, was the fact that You figured out, you did all of the reverse engineering that allowed you to do active scanning of OT environments without knocking stuff over because you reverse engineered all the protocols, understood how they work.

20:11Patrick Gray: So previously it used to be, oh no, you can't do an active scan of an OT environment. You knock everything over. You know, you figured out how to do that safely. Still meant you got pushback in a lot of deals where people are like, the graybeards, like no, you're not going anywhere near our network with that. But a lot of people did, you know, like you've really sort of changed the culture around that and now people accept that well if you've got a well-engineered product you can do that but you did all of that pre-ai and now that sort of reverse engineering work that sort of development is a lot easier uh so you've basically gone turbo mode uh the last four months why don't you tell

20:48HD Moore: us what you've what you've shipped sure all time stuff so going back to our 4.9 release we added all the ot protocols including all the ot uh base or um backplane protocols so speaking non-ip so You find a device network, then you ask that device to find everything else behind it, even though it's not an IP. So BACnet, Modbus, S7, all kinds of fun stuff. Yeah, that's the stuff we were talking about earlier. We have that nice, big, fancy interactive map, a little 3D map you can go pew-pew with and run around and see it all. So showing the connectivity of Layer 2, Layer 3, and these serial protocols all laid on top of each other is really cool.

21:18HD Moore: You can actually see the path from serial to IP to another gateway to a multi-home host, all the way to wherever you want to get to. And then we did a huge pile of work around bone remediation, bone tracking, exposure management, just kind of all the meat potatoes, like, you know, dashboard, shareable reports, being able to help you, like, take the fun stuff you see inside the product and get it to somebody else who doesn't know what they're doing. Like, give them a report, show them exactly what they want to see and not just giant inventory, right? So that all worked really well. And then more recently, we did AI.

21:45HD Moore: And just want to say, like, as always, Patrick is right. We did not want to do AI stuff for the longest time. And then finally we realized we were wrong and just sat down and said, okay, well, how would we do this in a way that customers are actually going to be happy with? Because we have some pretty conservative customers as well who do not put AI in the product by default. So we took it over. Dude, it's so funny.

22:03Patrick Gray: I literally had this conversation the other day with the Airlock Digital guys because I was like, hey, you probably want to do agentic allow listing and whatever. And they're like, our customers don't want it. I'm like, oh, I think they will eventually. And that's what happened, right? is they were right to resist it at the time because their customers were like, don't you under any circumstances ship AI features here? And now all of a sudden those same customers are back and just like, when are you shipping? And it's like, okay, right? Like that's, it's been a big change in the market, I guess is what.

22:30Patrick Gray: And you had to do both at once, right?

22:32HD Moore: We had to be investigating and learning how to use the AI tooling on one side while also resisting the demand to put it in the product at the same time. And once those two things crossed over and said, okay, we actually know how to do this well enough that it's safe, let's go ahead and do that. So one thing that we built that was really cool recently was an AI integration generator. So what people always hate is integrations. Like no matter which product you use, you may be missing the one integration you need. And everyone's like, okay, we'll build your own or use your API. No one wants to do that.

22:55HD Moore: So you can throw AI built-in APIs, but we found a better way to do it, which is like build a better integration engine into the product directly, and then have the AI tooling spit out integration for you live. So you give it a link to the documentation URL if you have one. If not, it'll figure it out. You go to an endpoint, and it'll go just create an integration for you. And you save it off, and off you go. So you can build integration to literally anything, custom, commercial, you name it. We put about 120 publicly online so you can get a sense of how they work. And we made a really decent API.

23:20HD Moore: So things like TLS pinning and all that's kind of done for you. You can chain stuff. You can set user agents. All the normal junk you want to do is kind of pre-baked. You don't have to reinvent the wheel for it.

23:28Patrick Gray: So why don't you just take a step back and define for us what you mean by building an integration when it comes to RunZero?

23:33HD Moore: Yeah, there's kind of two ways to do it. One is you're trying to pull data into the product. So you're saying, I want to go connect to my network switch.

23:39Patrick Gray: CrowdStrike or whatever or CoreLite. Okay, right. So that's what you mean by integration.

23:43HD Moore: You can pull data in or you put it back out again, or you can just manage the tasks. So we have folks who use integrations for outbound. So push data to Splunk or do a data lake or pull data inbound for full management. We also have folks who use integrations to push and pull data between two run zero instances. So they'll run external scans in one node. They'll push that to an internal node through a jump box. And that whole thing runs as an integration inside the run zero kind of tasking, if you will. So it's kind of neat. It's almost just an arbitrary language. You're going to do whatever you want to with it.

24:07HD Moore: We added a SHWMI. WMI is over S &P, MSRPC, and HTTP as well. So there's a billion different ways you can talk. So you can build an integration that shells into random boxes if you want to at this point and dumps data and then brings it either to run zero or out of run zero at that point. So it just really kind of took the restraints off a little bit and let folks kind of create anything. And of course, there are some restrictions in terms of where it runs and how you set credentials and encryption, things like that. But we basically made it very easy to build anything you want to in the product just by typing in a sentence saying, go do the thing.

24:36Patrick Gray: I didn't even know WMI was still a thing.

24:40HD Moore: Yeah, it's mostly HTTP these days, but you still see it over, you know, DCOM, DCRPC here and there. But it's kind of neat, like, putting all these old protocols and exposing them directly to the scripting engine. So the scripting engine will, you know, soon be able to do things like IPMI, Modbus, you name it. So we're looking at how do we take the same engine and bring it down to, like, the scanning layer as well and let people, you know, really get deep in the weeds of the stuff. You know, the AI integration we did was a little different. It's always BYOK. So you kind of take your, you know, your engine and you plug in the credentials to it and the URL to it and we use whatever you have.

25:09HD Moore: So if you have a management layer that's looking at your API request and steering it through either managed OAuth or NLL listing or any kind of filtering or logging system, you can drop that in and get the observability you want. So a good example would be like OpenRouter. You can plug in OpenRouter URL and turn around zero and then you get all the observability and hotel logs out of OpenRouter if you want to or whatever your backend happens to be. That way you don't know about data location. Sorry, you said BYOK.

25:34Patrick Gray: What's the K there? Oh, sorry.

25:36HD Moore: Bring your own key. So we say take whatever account, whatever URL you want, and we'll run all of the run zero logic through your own endpoint, including for report generation, AI kind of question answering, integration creation. And then we kind of show you the behind the scenes version of that. So you can go into the AI threads part of the product and say, oh, you know, Joe built an integration. Here's actually the entire thread that he built. Here's the output from it. I want to go rebuild that thing from here and there. So it's kind of like letting you do like arbitrary agentic work within the product like using your own AI backend of choice and being able to see the kind of internals of it in the product.

26:08Patrick Gray: Yeah, right. So you've got like a decent, yeah, you've got all of your transcripts and everything there, just from what the product is doing.

26:15HD Moore: Yeah, and we started shipping about a quarter of the product's total content now is like Markdown shipped in the binary because we built this whole self-reflective layer where it knows how its own code base works and it was all its own APIs. Something else we did is we took the MCP layer we had before and MCP layer we had inside the scripting engine in the kind of report engine and made them one. So it doesn't matter how you're calling the product. You can call it from Cloud directly or you can call it from the integration engine or the in-product UX. And either way, you can call the same tools.

26:41HD Moore: Like, you know, find me the assets that are at least seen, find me an OS with a particular configuration, give me a report and everything running in port 22 that's not SSH. So anything, or, you know, go through your CrowdStrike data and find me the CrowdStrike machine whose BIOS is out of data. You can do pretty much anything with that engine. I'm curious, how much use is your MCP server getting?

27:00Patrick Gray: You know, because I'm kind of working on something you know a little bit about it, but like how much, you know, how much are customers actually connecting to the run zero MCP? You know, they're connecting with local agents. They're connecting with hosted agents. Like what sort of queries are they doing? Like, you know, is this something that people are actually embracing or is it early days?

27:18HD Moore: Both. So when we first launched it, no one used it. And we realized they didn't use it because it sucked. We realized that the API calls were not great. They were slow.

27:26Patrick Gray: Or you're chewing up half the context window with like, you know, MCP instructions or whatever. That's a funny one.

27:30HD Moore: or CSV exports were eating your entire, coming back out again, all the data coming back out is pretty effusive. So way to come back and say, okay, let's build a new toolkit where it does like summarization at the API level instead. And of course, we had to change our access control. We had to change our API key management or usage management. So after we kind of just redid the whole thing about a month ago, it's now getting pretty popular. We're seeing folks hitting it all the time. We let you do things like use our natural search translation engine now from the MCP API. So instead of saying like, I want to find OS colon Cisco or whatever, you can say, find me the Cisco boxes and give me a search query for it.

28:02HD Moore: And then it gives you the actual API query that you need to do the rest of if you want. So a little mix of

28:07Patrick Gray: both arbitrary MCP. Well, you've taught it how to use the product, right? I mean, I think that's the... It returns smaller pieces, right?

28:15HD Moore: Yeah. Definitely where you kill it is either you stuff a million things into your context window or you spew so much data back out again that you can't make any decisions because you blew the whole window on your output. So trying to fix those two things has been fun. But if you make it too short, then of course the product doesn't work. You can't answer any complex problems. So we have things like turn limits. I think our default turn limit is about eight or so. We can set limits on how many records we search by default. And as long as you have enough context that tells the MCP engine or the agent when to call it the tools and what their limits are, it's pretty good about working around it.

28:45HD Moore: Say, let me do a sampling across these records or give me the top X to bottom X of this particular table.

28:51Patrick Gray: And where does your MCP live? Is it like, does it tend to be like behind the firewall on an enterprise network on-prem or is it in the cloud or like where do people tend to do that we're

29:00HD Moore: brave it's directly in the product so wherever your product is installed it is also the mcp server so it's remote hd streaming so you connect to whatever your runs or instances whether self hosted the cloud and off eco and that's it and there's no separate standard io or whatever to manage it also uses what sort of what sort of agents are people plumbing through to it right um cloud's probably the most common one we see some folks doing like either cloud cli or cloud desktop. We also see a lot of codecs. We had somebody trying to find, like, phones of the product a while back, and they're just, like, spewing the API request into it.

29:29HD Moore: So, fortunately, we cut things off after, like, if you're especially with a free account, we'll turn off the API request fairly quickly, which also shuts down the MCP access as part of it once you go over the limit. Something else we noticed early on is that folks, like, customers that we want to be using the product, we're blowing through the API limits per day, like, hundreds of thousands of requests per day by using MCP. We're like, what the heck is going on? And we found out that MCP was just, like hammering our backend for the dumbest reasons until we fix the context, give it better tools.

29:53Patrick Gray: Yeah. So, and how are you handling like the, like the permissions and the OAuth scoping and everything for that?

30:00HD Moore: Sorry, I know we're getting into like product engineering, but you know,

30:03Patrick Gray: this is, this is relevant to my interests at the moment.

30:05HD Moore: Yeah. We had like really granular permissions until about a month ago, you could say like read only or read write and that was about it. And then admin mode. So we went back to let's, let's do it from, you know, from the beginning again. Let's create every little scope permission, granular access, like inventory read, inventory by user managed, user read, like every category of data in the product that has multiple privilege levels. And we map those back into roles. But you can now do custom roles for everything in the product. So if you go into SSO and you log into a particular SSO group, you can then assign that group to a very specific set of roles.

30:33HD Moore: You can do X and Y, but not Z across a whole bunch of different organizations or mix of them. And then the MCP OAuth layer is all the same thing. You create a custom credit that can only do exactly whatever you want it to do. So you can safely give your MCP server something that has reaccessed everything. And it reaccessed only to certain orgs or certain types of data. And that's all it's over to touch. But rewriting that whole thing was a monster. So AI is helpful when you're doing massive bulk rework like that, especially when you need to do comprehensive backtesting on it.

31:01Patrick Gray: Well, the reason I'm asking you about it is because you tend to be two years ahead of everyone else, right? And I might be dabbling with building a product in this space, and I don't want to know where I need to be in a year. So that's cool. And it's interesting that you say you're getting the adoption now that the MCP actually works properly.

31:21HD Moore: Yeah. You can't just throw an MCP on and expect to be good. You have to actually try the experience. If it takes you five minutes, get a result, try again, figure it out, chop it down, shrink it, add better contacts. It's not enough just to bolt something onto it. Same thing with our in-product reporting for AI. You can now create arbitrary AI reports, whatever you want. But if you start off with just your engine and what the user types in, you're not going to get there. You have to give it charting tools. You have to give it the ability to draw like mermaid diagrams. You have to get a color guide, style guides.

31:47HD Moore: There's a whole lot that goes into it. Well, you can't just say like, you know, Claude, give me a reporting engine. You really have to go pretty deep to make it actually usable.

31:53Patrick Gray: But I am curious, right? Like, because you're in this situation where you're having to develop these like dashboards and, you know, visualizations and pew-pew maps and make it really cool. But I'm thinking most interaction with a platform like Run Zero in the future, it's probably going to be MCP or an agent using sort of run zero specific skills to go do X, Y, Z through an API or whatever? Like, is it, you know, where do you see the future going in terms of the split between, you know, dashboard versus agenting?

32:23HD Moore: The challenge with, I think, you know, we've had lots of customers reach out and say, hey, we built this cool report, put it in the product. And our take is like, that's great. You can put it in the product yourself. Copy and paste the prompt in, off you go, have fun. The reason we don't do it is because the reports that you specify like that only give you what you want to hear. So you would say, give me a report showing X. It's going to do exactly what you told it to do and to ignore everything else that's contrary to your goal. And so it's really difficult to build a report, this objective that tells you things that might surprise you if you're only asking to tell you things that you know about.

32:50HD Moore: So that's one of the biggest challenges. So I think you do need a set of visualizations of the product that you don't necessarily agree with. You need someone to tell you your baby's ugly. You need someone to say, hey, you've got my gender like crap. You need to go fix. I mean, that's really important. You can't say, give me a report that makes me look good. You need a report that tells you what's actually real. So to some extent, you need to provide all the customization in the world in the product. You also have to have an opinionated visualization, whether it's your dashboard, your queries or charts that helps users really kind of pin themselves to where they should be in the world.

33:16Patrick Gray: Well, I mean, you've always been a pretty accurate forecaster of how these things go. So I'm going to go ahead and just agree with you because it usually turns out to be right. We're going to wrap it up there, HD Moore. Thank you so much for joining us. If people want to have a look at RunZero, you can still try it for free, runzero.com slash try. but yeah man congratulations on selling the company because it has closed now and you know good luck with all of this right because it's a lot and I can't wait to chat to you again soon thanks for joining me

33:48HD Moore: thank you and thank you for being our advisor