StackHawk and Shift-Left API Security with Scott Gerlach

0:00APIs are a fundamental part of modern software systems and enable communication between services, applications, and third-party integrations. However, their openness and accessibility also make them a prime target for security threats, and this makes APIs a growing focus on software teams. StackHawk is a company that scans and monitors source code to obtain the full scope of an organization's APIs and applications, and runs tests to identify vulnerabilities and address them pre-production. Scott Gerlach is the co-founder and chief security officer at StackHawk, and previously worked at SendGrid and GoDaddy.

0:37He has an extensive background running security operations and engineering, and in this episode, he joins the show to talk about the challenges around API security and leading-edge strategies to address them. Gregor Vand is a security-focused technologist and is the founder and CTO of MailPass. Previously, Gregor was a CTO across cybersecurity, cyber insurance, and general software engineering companies. He has been based in Asia-Pacific for almost a decade and can be found via his profile at VAND.HK.

1:22Hi Scott, welcome to Software Engineering Daily. Hey Gregor, thanks for having me. It's super awesome to be here on Software Engineering Daily. Yeah, great to have you here. And you're here on behalf of StackHawk as a co-founder. And we're going to be hearing all about StackHawk and what the platform does. I mean, it's all sort of without any spoilers here. It's all about security. It's about API security. So this is a topic I love to dive into. And I think API security especially is always something that I've always wondered how best to do this. So we're going to be going into that today. But in true SE Daily fashion, it'd be great just to hear a little bit about yourself.

2:01Just a brief, what was your path to co-founding and you're also the CSO of Statcock. So what was life up until that point? Yeah, absolutely. Wow, so many questions. First of all, just a little bit about me. I'm Scott Gerlach. My background is in running security teams, security operations, security engineering. So I spent 10 years, almost 10 years running different security teams and functions at GoDaddy, which is essentially like 50 years of security because of, you know, give lots of random people access to your servers and see what goes wrong. That was super fun. I learned a ton. After that, I was the CISO at SendGrid email company.

2:45Hopefully you know SendGrid. Lots of devs know SendGrid because it's so easy to get it installed and up and running and send an email. After SendGrid and Twilio got combined is when I left. I took a little break looking for my next gig, next role. And I met Joni, our CEO and co-founder, and she was out digging around in the application security space. we had a pretty good conversation where she was asking questions about application security and why it's tough and i was kind of unloading a little bit because i was in that like freewheeling i don't have a job space so i kind of unloaded on appsec a little about how terrible it was and how underserved uh devs were in this process like literally i think one of the things that we talked about was literally the people that can fix this problem are the last ones to know about it.

3:41Like in almost every scenario we go, Hey, devs publish code and we'll get back to you in like a month or so. Talk to you about all the security problems in there. So after we were done talking about this app sec thing, I was feeling pretty full of myself, but I was also like, well, that was pretty opinionated. She probably is not going to talk to me anymore. like two weeks later we started stack off awesome that's a great co-founder story i love that yeah i mean the exactly the application security it's something i've spoken a bit about on on other episodes just how you know i think there's a misconception that developers are taught this you know if you do like cs or something that this is like part of the syllabus which is partly true but but it's kind of like an afterthought and you know when it comes to just general you know day-to-day work security for a developer again is sort of unfortunately the you know the business end isn't often thinking about it and they don't see it until it's gone wrong really so it's sort of been this slow burn to get to get businesses and developers kind of more more understanding of the problems but i think the problems are pretty well understood i think just to kind of set the scene i thought might be helpful just to kind of a couple of acronyms here you know d-a-s-t and s-a-s-t what are those and you know which one is stackhawk and why.

4:59Sure. DAST and SAST, acronym soup of information security. Dynamic application security testing, that's DAST. So testing what is a running application or running HTTP classic like web.1 web server or an API, HTTP API. So a REST API or a GraphQL API, gRPC or god forbid soap that's what DAST specializes at testing and testing that running version of the application is really beneficial because of a couple of different things mostly discoverability and exploitability so when you are testing with a DAST tool it does a really good job of going hey this is super important because it is discoverable someone can find it and can exploit it.

5:53And insofar as those two things are true, what's great about it is it will tell you, it will help you prioritize your time and keep you away from kind of what has historically been the noise generated by SAST or static. Application security. Static application security testing. This is like, so AST means two different things, two different contexts. Anyway, the headache application security testing. which is really great at pointing you at exactly the line of code that you want to go fix but it's also really great at and also here's 9 000 of them and has no context of what is discoverable what is exploitable and where stackhawk fits in is kind of we're trying to make our way to the middle so doing dast but doing it more of a gray box white box fashion so we are also aware of code so that we can point you to, hey, this is discoverable and exploitable, but also here's the code that you have to go fix.

6:56And with the advent of LLMs, maybe the code fixes itself, that kind of stuff. So really helping people find security vulnerabilities in HTTP APIs and fix them as quick as possible. So as engineering teams and as security teams, we can get back with to the to the business of helping the business run which is the ultimate goal of everything of what everyone's doing yeah absolutely so that's a really nice sort of framing so stackhawk kind of sits sits in that sort of semi-middle ground taking sort of the top topic here which is you know stackhawk is a is proactive it's a proactive approach to api security i guess question number one is sort of like what what does that even mean like to be proactive but and then And looking just sort of this in context, I think a lot of developers are probably pretty aware listening today, you work for a company and you've got API endpoints, but you've probably got a ton of API endpoints and some are actually being used and then you've probably got a bunch of endpoints that are sort of still there, but no one's actually taking them out and this kind of thing.

8:04So talk about that, like the proactive side of, or proactive approach of StatCock, as well as sort of this in context of API sprawl is kind of a nice way to turn it. Yeah, for sure. So the proactive side is one of the things that I've kind of lived my security life by worked at a hosting company. That is a super reactive environment, no matter how well you do at it. Being reactive, especially with vulnerabilities that you can find sucks. Like it's just not fun. You know what I mean? Like I was trying to think about this earlier as an analogy of like building your own airbag and putting it in your car and never ever testing it and the only time you get to test it is when you run it into another car and hopefully that airbag is really good or the protection some some kind of protection mechanism is really good putting untested software unsecurity tested software out on the internet is sort of a recipe for a fire drill later down the road.

9:08You're going to run into, whether it's now or later, you're more than likely going to run into, holy crap, someone's attacking us. Let's see if we can contain this before it turns into an incident. And if it does turn into an incident, now it's a whole big thing where you've got engineering teams and security teams all working, stopping their regular work and working together to do incident containment and remediation, which is going to happen. Hopefully you want it to happen as little as possible, like reduce the amount of risk that you're, they're putting out on the, putting out onto the internet or in production is one of the things that security teams are supposed to be doing?

9:53And the reactive nature of API security where publish the APIs, watch for attacks, and then react to them, that seems scary. It's super valuable in a, I think I've put out the best thing I can put out. Now let's see if and when we get attacked, we can respond and block attackers and recover. But doing it without tested software, holy moly, it seems really scary to me. I don't know how other people think about it in security teams or how engineering teams think about it. I think engineers generally want to put out the highest quality, most secure code that they can. The barrier to, is this secure?

10:40Was really, really high. And I think the way I that have often thought about it from like, why is it so scary? It's scary because, you know, APIs are literally just roads to your database. That's pretty much like the simplest way to put it, right? So they just happen to be roads with lots of rules. But if those rules can be figured out or so on and so forth, then you've just got a straight road to your database. I mean, that's just, that's why it's scary, right? Yeah. And API sprawl is not only sprawl, but the explosion of APIs, it's just compounding the problem so much. You've got the introduction of LLMs, which is making it easier for software engineers to write software.

11:19It's reducing their comprehensive cognition of what's actually happening in the code. So generally we understand what's going on, but we have less of that manual one key at a time authoring of some of this code. So we kind of naturally lose some of the what's happening in the code. And then the ability to CICD agile process, get everything out the door so fast. APIs are just insanely growing. So API, I don't know if you know this API traffic is approximately 80 % of web traffic today. Yeah. Well, like everything is API and that's crazy. And you're absolutely right. It's just like, that is the most riskiest place because it's basically direct connect to the database where all the risk is that's where all the data is stored that's where all the all the good stuff that threat actors want is is right there in the database and the api is the gateway to get there exactly so i mean could you maybe just speak a bit to you know how does how does stackhawk you know approach like the discovery and the management like of this complexity and then sort of going on from that like how does does StackHawk actually simulate like real world attacks basically?

12:34I mean, that's always the bit I've been fascinated by. I mean, I thought about this space a few years ago. I just assumed someone was doing it. So that's probably why I didn't go down this road. But the simulating of the attacks is kind of, I think, super interesting. But let's start with how do you even like go about the discovery bit? And then like, how do you simulate the attacks? Yeah, totally. So the discovery bit, we think of source code as the source of truth. for any company that's writing software for any value at all. You might have APIs that you didn't author running around in your organization, but you probably also can't fix them.

13:10So that's just a upgrade or a patch. But we think about API discovery insofar as I wrote code that makes this API work and that code is stored in my source code repository. So let's go look there. Let's start there. So one of the very first things, one of the newest things we built, one of the first things you experience as a StackHawk user is connect StackHawk to your source code repository. And what we will end up doing is running some StackHawk magic on that source code and going, hey, we think this repository builds a REST API. This repository builds a web 1.0 web service. We think this one builds a SOAP service, you know, those kinds of things to be able to go, here's all the source code that builds APIs.

13:56Now they might get stitched together later, or they might get, they might all end up in a gateway, but that's sort of how we think about it. And how we think about testing is like test that smallest bit of code that you can, because it makes it so much easier to correlate the problems with where to fix it. It makes the problem set a lot smaller and way more distributed. So getting that information to the teams that work on those things is super important. So that's, that's how we deal with kind of the discovery of APIs and being able to say, here's the attack surface that you have based on what code you've written.

14:31Nice. And then moving to the sort of, then, then what happens in terms of, okay, you've now discovered and then, and then what happens? Yeah. So the, the really great thing about APIs, the great and terrible thing about APIs is there's not like a classic web browser to go looking around at. And if there is that web browser, I always like to, I always like to say the Instagram analogy, like think about when you're looking at Instagram and you're scrolling and scrolling and scrolling, that never ends. It used to, but they fixed that problem. And so, and all that is, is just API call after API call on the backend, pulling more content and more feed.

15:09You can't ever complete the task of looking at the front end to be able to get to the back end. So thinking about testing the back end directly, whether those API, whatever flavor that API is, is the most efficient route to testing that kind of data flow. So being able to go, Hey, there's a REST API in here and there's a REST API or an open API spec that's either generated by the code or it's hand rolled or StackHawk helped you build it. And then ingest that and go, okay, I understand how this API works. I understand what kind of date, what kinds of data need to go into these URL paths or the post parameters or whatever that is to drive the URL correctly and also try to attack the API itself with valid and invalid data.

16:02Those are the keys to being able to really get into an API, test it pretty thoroughly, and find out whether there are or are not problems. The next part of that is really kind of business logic-y testing. That's a tricky problem. One of the things that we did for that was just be able to write custom tests. So if you've got some custom business logic in your application that says Gregor shouldn't be able to get to Scott's information about his address and his family information, you could write a test for that, right? So I'm Gregor, see if I can get Scott's information. If that works, now I can throw an alert.

16:40So because that tendency is so hard to generalize or those kinds of workflows are so hard to generalize being able to help write custom information about that a lot of our customers found that super helpful what are the differences complexity wise between rest and graphql because i mean i think graphql is where more at least just from where i sit that's where i was even more problems have come up recently from an api standpoint where those exposing that graphql API aren't fully aware of kind of all the traversals that can happen and what can come back. So how does that look? Is it sort of a different process or is it just the same thing?

17:25Yeah. The complexity and differences between REST and GraphQL are wide and vast. The very first difference in the two is how they document. So GraphQL is really good about self-documentation and rest is not. And that's the very first thing, which is a bonus and a drawback for GraphQL. Like you don't have to mentally think about documenting how the GraphQL API works. And so therefore it makes it really easy for us to go test it. But it also makes it really easy for anyone to go find that information and kind of start traversing their way through the graph, finding information that they shouldn't be able to get to with recursion, different, different recursion attacks, those kinds of things.

18:12That's not really a drawback of graph per se. It's just kind of how it works and how it's intended to work to power the applications that it's intended to power. So I wouldn't say it's really a drawback. It's just one of those differences you got to know about to make sure that you understand why it's a problem and you can test it and find it and fix it. yeah and and this is i think this is i guess what's to me great about stackhawk it's got it can cover both so it doesn't particularly matter which one you go for the api type that suits your business case without then having to overthink the security side side of that yeah or the language right so like one of the really great parts about dast and why we kind of picked dast was it's language agnostic so whatever language you decide to write in like next week you decide that Rust is where it's at and you're going to transform all your stuff to Rust and you've got terrible language support from static providers.

19:12Just when you're building an HTTP application, DAST has the ability to test it no matter what language it's written in. So that's, that's always a really great thing for me, like as a practitioner, being able to let the engineering team innovate and develop and try new and different things and be able to secure those same things as you're building them is super important to me. Yeah, that's a great call out. If we just sort of move on to, we have to these days always touch on Gen AI and AI generally, it is pretty pertinent here, right? Because I'm an absolute convert to Cursor. I love using Cursor to write code now.

19:52And yeah, it's turning out a ton of code. And so long as it looks kind of okay to me and it works, because I'm like, yep, great, we're moving on here. I'm not taking too much time over certain parts of the application now where I'm kind of confident that it's done its job. It doesn't look awful. It can be optimized a bit later on. And just to be clear, I would say I'm using a little bit more on the front-end side right now than the back-end side because it does help to be more aware of your logic. But this is the point. Developers are very much using Gen AI now to generate quite a lot of code.

20:25And on the other side, we've obviously got the possibilities that people can be using llms to generate code or just instructions on sort of how might one go about you know getting into this app through the api etc you know how is stackhawk thinking about this how is like the last two years kind of unfolded for evolving the platform to kind of meet this yeah so the gen ai problem the llm being able to write code doing it faster we covered that a little bit it's contributing to the explosion of APIs in exactly the fashion that you said, like generally this looks right. It's going in my code. I'm not sure exactly what it does.

21:03And that has been shown to date to introduce vulnerabilities. Like sometimes it does and hopefully, and I think it will in the future get better at not doing that. And I think where you're going to see what, what you're going to see happening is as you have co-pilot or whatever in your IDE and your writing code, I think you're going to start getting the Clippy experience of hopefully not the Clippy experience, but like the spell check version of, of Clippy in the IDE that goes, Hey, you just, you just introduced a security vulnerability here. Would you like to fix it? Those kinds of things. So instead of like running SAST on your, on your code, like doing it live while that's happening, I think, I think you're going to get a ton of, ton of value out of that in the near future and near future.

21:55I mean like a year or two as contextually aware LLMs continue to get better and better and better, especially around code. So that's, that's one thing. The thing that you still have to test for is things that you don't see in code, right? So the pattern that you don't see in code is a business logic problem or a authentication problem. Sometimes you can't see that information in code. So you kind of have to still test running applications for how the app responds to inputs and outputs, right? So there's a world there where those things, I think, get more efficient. Developers get more efficient.

22:35The LLMs will stop running kind of our junior devs who aren't really aware of what's going on. Like when you have an LLM write some code for you and it's awful, you have to be able to see that, which is unfortunately a thing. Like I run a Kubernetes server at my house here just because I like to, you know, torture myself. And sometimes I go, hey, ChatGPT, help me write a deployment manifest for this service and blah, blah, blah. And I want it to expose itself on the Nginx web server and have an SSL cert. and it completely makes it up and it's totally wrong. And you're just like, no. But the trick is you have to understand how that today, you have to understand what it's supposed to look like so that you can go, nope, that's completely wrong.

23:24I think that starts to go away as well. Like in the future, like that whole total hallucination of what a solution looks like goes away. So it's just gonna keep getting faster and faster and faster. And we as software engineers and security people are just going to become less and less intimately associated with this code. It's just going to be a thing that we're generating and getting out to the market to provide value for customers as fast as possible. And we're going to have less of that like pride. I pumped hours and hours and hours into this code like we kind of do today. I'm already there.

24:04you know i've i've been using you know cursor now for about six months i think maybe five months and it is it is that i just feel less less sort of precious i guess of of of the code and i'm curious i mean i think that maybe helps in this case because i think with tools like stackhawk at least it's it's not a person telling you you've done something wrong but it's still kind of annoying when something's like this thing you wrote you need to go fix it and And would you say that now if developers are able to kind of utilize more from the LLM generation side, there may be a bit less pressures. And actually, at the end of the day, it's just kind of robot versus robot sort of saying, well, that robot didn't write the right thing.

24:47And you're just kind of the pilot overseeing the whole thing going, OK, well, that thing just didn't write the right thing. And StatCock said it's not correct. And actually, it's maybe sort of that sort of dynamic is going to play out. I don't know what you think. It could. it could like human emotion is a thing and every time a human interacts with the human and goes you did it wrong no matter what words you use that's that's ultimately what ends up happening you take a little bit of offense to that you know what i mean like it's really hard to communicate man this thing you did is super valuable like i understand what you're doing i know that you put your heart and sweat and soul into it and it took you forever to do this one thing could be uh better There's a problem in here.

25:31Like everyone skips that first part and just goes, this part sucked. And it feels like you're terrible at your job to your point that could turn into a deflective. Like, no, I'm not terrible at my job. The robot was. And so now I'll just go fix that. Maybe. I think the more important part is like the more in context of what you're doing, you are aware of the problems, the more likely you are to just be like, yeah, yeah, no problem. We'll just fix it. Right. You don't get mad at unit tests when they fail. You go, oh, okay, well, I got to fix that. And linters and all that good stuff. It's annoying, but you go, all right, I got to fix it.

26:10Yeah. So, I mean, just kind of like wrapping up on the GNI side, at the end of the day, code is being generated much faster. And sort of that inevitably leads to code that maybe hasn't had a full check or a full kind of sanity check from a human. And as you've called out, Scott, like I've seen code come out and it's, as you say, it's just completely wrong. But it's only from having years of experience I can quickly look at it and go, that's completely wrong. And then just can it and then be like, OK, either this needs a different prompt approach or just actually, you know, I'll just start it myself and then see where we go from there.

26:48And I think clearly, you know, something like StatCock is able to sanity check that far faster than sort of humans. And as you call it, there's almost a catch 22 now between sort of junior developers coming in. Are they supposed to be using this from day one? Are they or are they not? You know, and sort of how are they going to learn to sanity check? And yeah, it's kind of interesting. And I guess could you almost look at StatCock in this context as sort of just like a, I don't want to say like checking the homework, if you know what I mean. but like it's able to kind of sit there and just sort of yeah be that sort of overarching checkpoint i mean you know we haven't even touched on this the classic phrase shift left which i think probably applies to this somewhat but you know you know we've talked to lots of lots of different people using kind of llm assistance code assistance and one of the things they often say is who's going to check this code the llm well like when would the llm go this is wrong or maybe even more importantly when would the llm go this is right and do you trust the llm to check the work that it produced because is it gonna go no what i wrote there is completely insecure you should uh rewrite that for me you know what i mean like having something else that can check it is super valuable yeah that's a great point yeah so that positive affirmation it doesn't it doesn't to my knowledge give that yeah it doesn't say i don't think i've seen anything especially from a security standpoint where it says, here's this code and I've definitely, definitely checked there is nothing wrong with this code.

28:17I mean, it doesn't do that. Yeah, especially if you're writing like a snippet in the context or the context of how that snippet is included in the rest of the app or the code base. Like that's all really hard problems. I think it's going to get solved over time, but right now it's like, okay, somebody, something has to check that this is being done the right way. So just moving on to, you know, DevX, you know, we've got quite a technical heavy listener base. We do have non-technical listeners as well. So just sort of keep that in context for sort of when we're talking about containers and so on and so forth.

28:57But just maybe from a high level, like what does that look like for a developer? Like, you know, someone on your business team or security team has come to you and said, hey, we're implementing StackHawk. Off you go. Like, what is, like, go figure it out. Like, what does that look like? Go figure it out. I think for the most part, when we started the company, the whole idea was how do we put the right information in the right hands at the right time? Because there's tons of security tools out there that do a great job at serving security teams. And they do a great job at kind of building these reports that you look at and you have these huge pie charts of unfixed things and they never move.

29:33so our goal was like how do we help the people that can actually fix the problem understand there is a problem so they can actually fix it we're staring at shrinking pie charts or some other kind of bar graph whatever that's how it started and so the way that we build the tool and how you use it and the interaction and the documentation that we built and all of that is really really dev friendly configuration is code we use yaml love or hate yaml we didn't do it in XML. So be happy about that. That whole process of like thinking about how automation works and how you're going to wire this up in CI and how you're going to continuously test and how you can test while on your laptop, while you're writing, you put your little snippet of code into debug mode and you could test it and know what's going on before you spend 10 minutes in the CI pipeline.

30:25And then it pukes and information architecture, like here's the information you need to know. and the rest of it is off to the side. Like if you want to know, you can know, but here's the information you need to know. That's how we really, really started with the tool and built the tool and people super love it. Like we've got a ton of customers who are classic security teams. They have had that kind of centralized security app sec force that is trying to keep up with the developer teams. I don't know if you know this ratio, But generally, the ratio of developers to security people or AppSec people is 100 to 1.

31:06So you've got like one security person trying to support 100 developers. Never going to keep up if they kind of can't get this information out in a more timely fashion. So almost every single time when we run into that central team was like, this is broken. I can't do it like this. I need to be able to do more automation, get more information distributed to more teams. they kind of take it on themselves and go start small and go, okay, yeah, does I see how this could work? And we recently had more than, more than 10 customers. We do this kind of part of our onboarding is a developer training thing.