The Future of Offensive Pentesting with Mark Goodwin

0:00Offensive penetration testing, or offensive pen testing, involves actively probing a system, network, or application to identify and exploit vulnerabilities, mimicking the tactics. of real-world attackers. The goal is to assess security weaknesses and provide actionable insights to strengthen defenses before malicious actors can exploit them. Bishop Fox is a private professional services firm focused on offensive security testing. Mark Goodwin is the Director of Operations at Bishop Fox, and he was previously an officer in the U.S. Air Force where he did cyberspace operations. Mark joins the podcast with Gregor Vand to talk about Bishop Fox and the future of offensive pen testing.

0:40Gregor Vand is a security-focused technologist and is the founder and CTO of MailPass. Previously, Gregor was a CTO across cybersecurity, cyber insurance, and general software engineering companies. He has been based in Asia-Pacific for almost a decade and can be found via his profile at vand.hk.

1:13Hi, Mark. Welcome to Software Engineering Daily. Hey, thanks for having me. Yeah, so great to have you here today, Mark. You are at the company Bishop Fox, which we're going to hear all about and understand about what is called the Cosmos platform and really just what Bishop Fox does in general. I think one of the first things would be really just interesting to hear about, obviously some details I don't think you'll be able to go into, but just your background is very interesting and sort of how you then ended up at Bishop Fox. So maybe if you just sort of introduce yourself a little bit in terms of, I don't know, straight out of college or high school, like what did you do?

1:48Yeah. Again, thanks for having me on. My name is Mark. I've been with Bishop Fox for four and a half years. I graduated college back in 2012, 2013, immediately went into the Air Force to be an active duty officer or got to do offensive cyberspace operations. So a lot of late nights, a lot of long days, and learned a lot about threat posture. How do we go after hard targets in ways that keep the toolkit safe, keep the platform safe, and go after nation state objectives? So I did that for six years, six and a half years. And at the end of my time in the Air Force, I started applying and Bishop Fox was starting this new service called Cosmos.

2:29And it was all about continuous offensive security at scale. And definitely very exciting, very, I consider it the future of offensive pen testing to be able to stay up with nation state actors, follow emerging threats, and identify attack surface that customers may not know about. So like I said, I joined Bishop Fox in March of 2020, started as a senior analyst running the analyst team, which is all about discovering attack surface. Did that for about a year and a half, led the operator team, which is all about taking that intelligence that we derive, that attack surface that we see, prioritizing potential vulnerabilities and providing true positives and true negatives, right?

3:08So true positives, here's a finding that we know exists, that we've validated and we've proved you're vulnerable to with a finding, or we give them a true negative to say, we went and identified that this host is out there. We thought it was going to be vulnerable and it wasn't. So you know, you're good, right? Your defensive team is doing the right things. They're applying the patches. This misconfiguration that we thought we saw didn't actually exist. So we're able to provide that safety blanket to say, even if you're not getting findings, you should be getting something to say, we're looking at your attack surface and we don't see anything.

3:40Yeah, super interesting. So I'd love to understand. So you've given a great sort of quick overview, I guess, of Cosmos. My understanding is that Bishop Fox started as more like consultative company and the platform aspect has kind of developed over a significant amount of time. I believe Bishop Fox started back in like 2013 or around then. I believe it's older than that. I believe we're an 18 year old firm. So yeah, they have been doing consulting for a long time. That is how we've made our name. We bring on some of the best offensive mind in the country, across the world, and let them work on really interesting projects.

4:16We dedicate time to training and developing tools that is going to make the offensive cyber community better. I have to shout out Joe DeMessi with his Sliver tool kit that he's built. Pretty exciting. And we use that almost daily. So yes, Bishop Fox started as a consulting. It's still our primary way to get after securing customers, keeping them safe and protecting their customers' data and trying to find vulnerabilities. We have our consulting firm and then we have the Cosmos firm and they're both continuing to grow. Amazing. What was the, I guess, do you know kind of what was the moment when it was thought actually bringing, turning sort of this knowledge into a platform made sense?

4:56And like, what was the thought process behind that? Yeah. So when I joined in March of 2020, we had been doing, we had called it CAST, which is like continuous attack surface testing. And then we had been doing that for almost a year. And I think it started from this mind of like, you know, how do we take our insanely skilled consultants or operators and give them a suit for Ironman, right? Like we don't want to build Ultron where we're just automating everything, but how do we instead enable and make our operators better? And so that was kind of the driving force. So it's on our very old documentation that, you know, from the way back times, We have that, you know, we're not building Ultron, we're building Ironman, you know, to propel us forward.

5:33And so part of that was looking at enterprise customers, right? Their attack surface is huge. How do you look across all of it when they just need help identifying what is the most important thing, right? We only have so many hours in a day, in a week, in a year. How do we find those things that advanced actors are going to go over, right? Hackers by trade, very lazy. We know that if we can take the knowledge of 10 operators or 10 consultants, consolidate that data, make it easily attainable, take off some of the scanning that every external pen test starts with, what if we instead offload that to automation?

6:07So it is on a weekend or smart scanning as we identify new targets, it gets scanned and then it prepares that data for us. So I'm not wasting cycles waiting for a scan result to return. Instead, that data is there ready for me. And then when I go to pick up the work, because we've had five other super sharp consultants or operators do a similar investigation, we get a jumpstart on what we're looking at. Yeah. So I think this is where it gets super interesting. So my understanding is the platform has it kind of has like three parts to it. So you've got the automated attack surface. You've got the automated application pen testing.

6:41And then does this understand also automated internal testing and pen testing? Is that correct? Right now, we're not at the automated internal pen testing yet. That's definitely on the roadmap, though. how do we do that smartly? How do we make sure that we drop into the right location? And again, for Cosmos, the entire point is scalability, right? I need to be able to do this on five targets, 500 targets, or 5 ,000, and way beyond that, right? So right now, the main focus is that web app pen testing, doing the research on customers' attack surface, we know it makes up the majority of their attack surface, right?

7:11There is going to be SSH and FTP services out there that may be vulnerable, and we're going after those. But most of a company's attack surface is going to be web-based. So that's what we spent the first four years really focusing in on, making sure that we can automate what we should, right? Like, hey, this is an exposure, you need to go fix this. Preparing the work for the team of, hey, we've identified that there's a version mismatch or there's a potential SQL injection here. We're not going to throw that. We want to keep that to be a human decision, right? Because at the end of the day, the last thing we want to do is knock over customer infrastructure, right?

7:46That's one of the worst things that could happen. So using this insight, using this prioritized attacking, we're able to say like, hey, we're going to automate these things that we don't care about. You need to fix it. Directory listing is something that you should not have. That's not a good configuration because of the potential to put something sensitive there. But we don't necessarily have to go look at that. We can build automations that will look for directory listings, tell you about it, and then look for things that were exposed and identify like, was there something sensitive here that needs follow-on work.

8:18So, you know, continuous web app pen testing is what we're started on. Continuous external pen testing is kind of that next thing of, all right, now that we've got web app pretty much locked in, now that we know what we're looking for, how do we look for other services, you know, SQL and other endpoints that may be a little sketch to be out there. How do we start targeting those and take care of those? So when it comes to the automated pen testing, and I might just sidebar for a second. I mean, this is what could also be classed as effectively automated red teaming? Is that a good way to think about it?

8:47I think it's definitely nuanced. I think there are some portions that could be considered that. When I think of red teaming, again, with the background that I have coming from the military, I always think of red teaming as this very high stakes. I need to move silently. I'm competing against that blue team. If I'm a red team, I'm competing against the blue team. I don't want to get caught. So a red team will have to operate and act and make different OPSEC decisions. right? For our continuous external pen testing, for our continuous web app pen testing, I hope you find me. Like that would be a good thing, right?

9:20Like if you have smart filtering or smart logging to say like, Hey, we see this thing, this isn't good. Drop those packets. Like that's a win for you. So that that's kind of the nuance I would say, but the methodology is still there, right? Like I want to be safe. I want to make sure we're not knocking over infrastructure. I want to make sure that we can get the data that we need out without causing logging to occur because it makes our job easier. Yeah, I think that's a good nuance. I mean, some of our listeners, whilst very technical security is probably not something they actually have had to touch a lot.

9:52So yeah, the sort of term of red and blue teams, yeah, red being the offensive and blue being the defensive. And I believe there's even at this point now, purple teaming, which is... Yes, there's purple teaming where, you know, the blue teams come together like, hey, we want to see if we can identify this attack patterns, right? The offensive landscape is huge. You're not only competing against White Hat, Or even gray hat, you know, security researchers, you have black hat actors out there that are willing to cause harm and willing to like ransomware is on the rise and at an all time high, right?

10:21So purple team is this way to address that where you take red team folks and say, run your playbooks with us, right? Like tell us when you're going to launch an attack so that we can look at our logging and figure out like, do we have these systems in place to catch that? So yeah, sorry about the nuance, right? Like red teaming to me is a very like specific subset of offense, but I would say everything that we're doing is on that red team offensive side. No, that's great. I think it's really good to be able to explain these, as you say, it is nuanced. So I'm just kind of interested to understand maybe more how did Cosmos evolve?

10:54So I mean, it's Cosmos, I think you said it's now been in, I guess, production for four to maybe five years. Is that correct? Yeah. Yeah. Could you just walk us through almost, I don't know, year by year, if that's possible? Yeah. Where did it start and like what was added and why? Because I'm just super curious how you take sort of this human process and then decide what and when that's going to get automated and brought into this platform. So yeah, if you could walk us through that. Yeah, it's an ever changing process. I know, you know, since being here in 2020, what we do now is not vastly different, but it is significantly different than what we did, you know, back then.

11:31And when I joined the team in 2020, we're operating out of just the goods, right? We're only going to provide findings for things that are highs or critical severities, right? This is the pulling your folks. They don't get to take a weekend. They have to fix this now. Everything else was a notification of, hey, you should probably fix this. And as we continue to work with customers, we realize there is a wider range of severities, right? It goes from informational all the way up to critical for a reason. So around 2021 or early 2022, we started reporting low to critical. We still don't do the informationals just because there's so much noise associated with that.

12:06And again, Cosmos is all about the like validated findings. If we're writing you something, it's something you actually need to go take a look at. So after about being here for around a year, we started reporting these low to criticals. And so that was more work coming in as we were getting more customers. And that's where we started looking at, all right, how do we identify those vulnerability types that we should automate? Like, does it really bring value? I gave this example earlier, but does it bring value for me to look at a directory listing and say like, yep, there's folders in there with files, right?

12:36Like that doesn't really always equate to something good, especially if it's something that is relatively benign, like a CSS folder, right? Like very rarely is there ever going to be anything sensitive in there, but you do need to be aware of it so you can fix it in case, you know, you have an intern or even somebody super senior that goes to drag and drop and puts it in the wrong folder, right? Like there has been findings that we've reported where like, Hey, we found passwords in a docs folder, right? Like that's not good. You need to make sure you don't do that. So those become two findings.

13:06And so it was, it really became, you know, out of survival and scalability, what do we need to automate? And so it is those, like these low level, medium level of, this is an exposure that you need to fix. This is a system configuration problem. and we're going to save that harder follow-on work for the operator cycles, right? We let our analysts make sure the target's in scope, and then our operator will do the work to make sure that like, hey, here's the full business impact of what we found in these files. As we continue to grow around 2022, we started automating subdomain takeovers. Again, this is one of those vulnerability types that doesn't take a lot of time, but a customer lifecycle with Cosmos, You get onboarded into the platform.

13:47My analyst team goes out and identifies all of your attack surface that they can find, right? They use open source intelligence. They use publicly identifiable information to figure out like, all right, this person is in your org. This person has registered this domain through who is information. So we believe this domain follows. We do this work and we would always get this huge spike of trash DNS records, stale DNS everywhere, domains that were for sale that used to be registered and had clear ties to a customer. Or just, again, misconfigurations where you have subdomain takeoverable subdomains.

14:23And so because we were at a point where we were getting a lot of new customers coming in, we looked at that and was like, all right, let's prioritize subdomain takeovers. And so we spent some cycles to report it out as fast as we could because there's this other entity out here that we're working with called Bug Bounty. There's a lot of Bug Bounty organizations out there that are also trying to help keep customers safe. And so we bring in a mature, validated findings. We have quality assurance checks. There's a lot of like, not a lot of bureaucracy, but there are a lot of checks to make sure that everything you get still is that Bishop Fox standard.

14:56Where with bug bounty, there's not always that, right? Like it can be five people just trying to get first report in. So they're not considered a duplicate. And so that's how we've kind of driven our automation approach. It's, all right, what are the low weight things we can knock out? And then what are through a customer lifecycle? What are the high impact things that take up too much time that really don't need to be there, right? Like if I can write a script that can go out and validate this vulnerability, do I need to go look at that? Or can I use my creativity somewhere else? And that's what we always try to lean back on is, is this a creative exploit or is this just something that with enough time and repetition, we can train and build a good enough tool that can tell a customer about it so that they can go fix it?

15:37So yeah, I think a lot for the listener base, the subdomain takeover I think is super interesting and also very relevant given some news that came out recently around a company called Watchtower, who's actually based in Singapore, being able to take over the.mobi, who is Records, I believe. Can you elaborate on why was subdomain takeover so important for Cosmos to focus on? And what does that even mean? And what can someone do if that is brought into effect, I guess? Yeah. It's not the first order. It's the second order effect, right? You know, if I was hosting something clients.example.com, and then for whatever reason I misconfigured or the application, like the hosting provider that I was using lapsed and that clients.example.com instead gets hijacked or taken over by somebody else, it can now become a phishing angle or, you know, a way to like harvest credentials from customers that are trying to log in.

16:37And so it's bad in that, you know, it hurts customer, like it would hurt our customers, like public perception and making sure that like we're keeping our customers, customer safe, right it's we try to approach it from that angle but when we talk about i guess subdomain takeover i guess is there a reason why the subdomain is more vulnerable obviously if this subdomain but you know if you have access to the dns of a domain then you can just create whichever subdomains you choose so are we talking about both like effectively domain takeover and subdomain takeover or how does that look that's a great call out so it's both right like you know azure subdomain takeovers, you're not going to be able to take over a domain that points back to an Azure.

17:19But if they misconfigure their Azure instance, then it would be possible, right? Given that the right things appear when you do a dig, you're able to investigate like, all right, this is vulnerable because of the configuration that they have. And then you can do work through the API gateway to take it over, right? There are other applications where you go out and it's a third party that you're hosting on. And if the third party is vulnerable or you didn't complete a setup, Shopify is a great example, right? If you don't complete a Shopify setup, you're able to come in there and scoop it up. And now that's your subdomain.

17:53Congratulations. So there's a lot of examples out there and they kind of run the gamut of, yes, they are harder because you actually need the domain. And that's why we also report on like, hey, this domain is for sale, right? Like it used to belong to you. It would make sense that either you make it very clear that it no longer belongs to you or you go out and you purchase it just defensively. There's a subset of offensive security called like brand protection. That's not something my team does, but our customers ask us about it and we try to lean in and help where we can. But because it's like more defensively focused, we don't do it, but we're able to say like, Hey, you know, the common things to look for are typos, squatting, you know, off by one character, those kinds of issues.

18:33Yeah. That's a very good way to explain that. So when we think about, you know, you mentioned having enterprise customers and yeah, if you go on, you know, bishopfox.com you can see a whole bunch of names there how do you approach that when it comes to like companies with such large attack surfaces you know whether you maybe want to bring out any examples that you can or just sort of talk at high level about a sort of industry you know i've worked on a tax surface but i'd say probably this was a tax surface of fairly small companies so you know i mean maybe the largest i think i worked on was singapore airlines and you know it was large-ish, but I wouldn't say I think for probably the kind of cases that you work on, it was what would be defined as large, actually.

19:13And, you know, we did find a kind of couple of interesting things, like tons of staging sites and that kind of thing. And to your point, you know, it's not that critical, but it's, there were various things where you sort of said, look, if someone is looking for as much information as possible to try and say spoof being Singapore Airlines, like they've got a ton of resources now. So you might want to just like lock that one up. So yeah, talk to me about like maybe some of your largest attack surfaces and kind of what were the actually, because I think Bishop Fox talks about sort of the needle versus the haystack.

19:42You know, we find the needle, not the haystack. What were some of the needles, I guess, in enterprise? Yeah, that's a great question. You know, our largest attack surfaces are six to seven digit targets, right? So that is a scheme, IP or domain, and then a port, right? So HTTP on an example.com and then HTTPSExample.com on 443, that would be two targets. So we look at it as a whole though, because we've seen through investigation and the reporting that we've done that port 80 and 443 may serve up different information. And so we want to take that unique approach to each one of them. The way that we find that needle in the haystack goes all the way back to my analyst team.

20:22So they are the ones that are tasked with identifying attack surface, right? So we do this breadth first, and then we go depth. We start with breadth of customer A has a lot of subsidiaries. So let's go out and identify each one of those. We use things like SEC filings if it's a US-based company. If it's not, we get to research like, all right, how does Germany do their filings for what customer owns different companies? And so it's a lot of fun investigative work there to figure out this customer that has the service actually has 10 other companies underneath it. And so we do this iterative approach until we get to the point where like, all right, we've identified all of the main companies that they would own that have a domain.

21:03From there, we start using reverse who is information to figure out like, all right, this company generally uses these email addresses or they have this as their org name or their tech name. And we're able to just search back through current information as well as historical information to figure out like, all right, did they have these domains and then they privacy protected them? We're going to bring those in and have a conversation with the customer to say, like, we found these. Are they still active? Because they're private and we don't want to include them if they're not yours. But if they are yours, we do want to include them.

21:35So we try to get that whole approach of, you know, we call it domain based investigations, right? Like we're going to go out and, you know, yes, we're going to cover your CIDR blocks. We're going to research ASN filings and make sure that like we're grabbing slash 24s or slash 20s and bring those into the platform. but we really want to make sure that we're identifying your domains because that's, again, how most of the things talk to each other on the internet. From there, we're going to do sub-demand enumeration with lots of different tools. The cybersecurity community is not shy when it comes to creating a tool and throwing it out there.

22:06And then we get to go through and figure out like, is this useful? Or is there a nugget in here that's good that we can take and make better? And in some instances, we'll go back and do a pull request to help the community. In other instances, we've built a lot of things in-house that, hey, this works really well, and we're going to do this for our service to give back to our customers to make sure that we're finding as much as we can. We let the platform help with reachability checks, resolving checks, so we never cut off a target too soon. We give a target several chances to become validated and become something that we look at.

22:39So it's a really long-winded way of saying we do a lot of prep work to build out a haystack that is less haystack and more needles. I know if I'm looking at something, it's definitely theirs. And then from there, I leverage my T team. They're like my threat enablement team. And they create scanning engines. They create things that we're going to go look for that are threat informed. We use the CISA known exploitable vulnerabilities list. That is one of our driving forces because customers care about that. I may not care about this thing that a security researcher found because it's probably not going to be out there.

23:16but I know that my C-suite or my senior executives are going to be asking about these things. So again, we always want to partner with the customer, meet them where there are, you know, get those baseline things taken care of. And then we can start working on, oh, and by the way, here's this and here's this and here's this. And we continue to get better that way. So it really goes back to, you know, good attack surface discovery and then looking for the right things so that when the operators pick it up, they're guaranteed to have a needle. That makes a lot of sense. I mean, when it comes to the, I guess, continuous aspect of this and sort of continuous i'd almost like analogize with like being proactive versus reactive in the sense of you're able in theories as of as it sounds continuously understand the attack surface and i guess match that up with especially with any newly declared vulnerabilities how does that work i mean it must be a lot of i'm just talking total lame in terms here but like a lot of filtering going on where you're like we looked at this yesterday so we're not we're not going to look at it again today?