AI-Powered Threats to the Software Supply Chain

26:46So several years ago, we built OctoSCS to solve this problem for us, right? It is a credential federation service for GitHub because GitHub does not operate one. I've heard one's coming, but I think it's on their public roadmap. But this would have prevented a great many of these attacks. And we operate it as a public good. Anyone can use OctoSCS and the one long-lived credential that sits underneath it lives in a KMS system that nobody can access without it basically paging our security team. But it gives you short-lived credentials. And I even flagged that to the TJ Actions maintainer when this happened, because like I said, we operate this as a public good.

27:27I think another good one that's happened recently is the Trivi breach. This built on a technique that Billy Lynch, who is one of our engineers, disclosed to GitHub three and a half years ago. We call it imposter commits, which is this technique where folks are pinning a GitHub action to a commit. That commit actually doesn't have to live in the repo that it looks like it comes from, from the rest of the action line. It can actually come from a fork. If you Google chain guard imposter commits, you'll find the blog post, but this is part of the technique that was used to compromise trivia. And so treat your build systems like production systems, harden them, use short lived credentials, practice least privilege.

28:10I make damn sure that every link in your supply chain is auditable, right? Just like you would in your production environments. So, you know, what came from where, what happened where, et cetera. Yeah, that's great, great advice. So yeah, if anyone takes away anything today, take that away so looking at like chain guard as a product that as we've been talking about has evolved over the last couple years i'm kind of interested in i guess a few things actually so the first thing i guess is actually just the number of images that you now actually maintain which i think early 23 it was about 400 then it was i think over a thousand by the end of 24 or an hour pushing over 2000.

28:54And I guess it's like, how do you guys think about just maintaining that consistency and quality? I mean, that's just a crazy amount of stuff to maintain. Yeah, there's a lot. I think we're adding two to 300 images every quarter. And we've got an amazing team that has been building a lot of those in some cases by hand, increasingly leveraging automation. But on the upkeep side, right? Like two to 300 isn't necessarily growing super rapidly, but maintaining those images keeps growing, right? It grows by two to 300 every quarter, right? So we're looking at automation really across the board. And a lot of people talk about human in the loop, right?

29:31If you think about the purpose of automation, it's generally getting humans out of the loop, right? We're slow, we sleep, and we too can make mistakes. And we've always emphasized automation long before things like agents arrived on the scene. And we would try and automate as much as we could. And then humans would get involved when things broke down. And when we did get involved, we would look at ways of avoiding repeating that breakage, as you might look at a postmortem, right? You always reflect and you're like, oh, how can we do better next time? And over time, those systems have improved.

30:03But agents have come onto the scene and we aren't ignoring them, right? We are looking at ways we can safely use them. But the way we look at them is they're sort of yet another tool for pushing our automation further, handling more of that without needing human intervention. But ultimately, just because our automation is becoming agentic, just because you stick that word in front, it doesn't change any of our commitments or our SLAs. So we're trying to find ways to adopt it responsibly and figure out those best practices. Because the way I see a lot of what we're doing is operationalizing best practices at extreme scale.

30:38One of the very first places we actually employed agents was to bolster our testing across our inventory, right? Like, can it generate more tests that exercise different behaviors of libraries or packages and do things that could help us catch regressions coming in through new updates? And so we still very much have humans in the loop, but the mindset shift is increasingly going from how do we solve a problem to how do we solve entire classes of problem? Yeah. And the average patch time, I believe, is still incredibly impressive. You know, like a critical is like less than 20 hours. and then Pi is two days, something like that.

31:17And we recently announced an SLA for the Kev list. So if a vulnerability is known to be exploited, we have a 24-hour SLA for getting that fixed and in folks' hands. That's amazing. I guess looking from a product perspective, but I guess more on the competitor landscape as well, I think maybe a few people might be thinking, hey, doesn't Docker now do this? So yes, they do. And I guess that must have been an interesting moment for you guys. It does happen, unfortunately, to very great ideas of companies. You know, the bigger guy comes along and says, that looks interesting. We can do that too. GitHub, I think, have done that.

31:53And GitHub Actions was basically a clone of other startups into GitHub. Made sense. So, yeah, how has that affected things? How you approach this problem? Docker has a free tier. That's, I think, interesting to talk about as well. Yeah. Speaking of things that have changed in the last two years since we last spoke, competition has really come out. And I mentioned we had to convince people that what we do is possible. Thankfully, that's actually a conversation I have to have a lot less now that other competitors have come out with products that at least claim to do the same thing that we do. I think one of the confusing things with DHI is that they claim two things, only one of which can be true.

32:33One is that they are building on top of existing distributions like Alpine and Debian. They also claim that they're building everything from source, right? But if they're building everything from source, then they're not Alpine or Debian, right? They can't carry patches. They can't build versions that Alpine or Debian aren't carrying and build those things from source and then claim it's Alpine or Debian, right? Those two things are sort of in conflict. I think the other thing is we are very familiar with the Alpine ecosystem and the Debian ecosystem, and we use them for many years. Our CEO, Dan, and I really built the first real hardened images back in 2017, 2018, which were called the Google DistroList images.

33:14And we actually did build that on top of an existing distribution. We built it on top of Debian packages. And it was because of this that we knew that to get to the results that ChainGuard delivers, that we couldn't build it on top of another distribution. If you take any of those images, which are running on the latest Debian, they're minimized and have the latest versions of all those packages. You take any one of those images and scan them with the exception of one, they all have unpatched CVEs because Debian is, you know, deciding what to backport, what not to backport, et cetera. If you scan our equivalents, none of those have any CVEs.

33:48And so, you know, I think I'm glad that more folks are starting to take this problem seriously and not just because I don't have to explain that what we do is actually possible. But I don't think all of the approaches are necessarily created equal. And I think our approach is informed by an extremely long history in this space, doing things like this to harden these images and figure out the path that we actually have to take in order to, you know, achieve the result that we are getting. Yeah, I think usually in this case, it is so that, you know, rising tide raises all boats. And, you know, I think, or just like a bigger pie, or whatever analogy you want to give it, but it should overall be positive.

34:27But yeah, it's obviously also interesting, yeah, when such a large company does come along claiming to do sort of a very similar thing. And I think that distinction that you've already made is very helpful. One other thing that was interesting, I think end of last year, you guys announced Emirates OSS. You know, this is like maintaining deprecated open source projects. Emeritus, it's a play on. Oh, Emeritus, yeah. I was like, Emirate, is that like to do with that? OSS at the end. Emeritus. There we go. Nice. I was going to ask what the name meant, but you just, yeah, there we go. Yeah. So this is, I think that the origin of this, you know, is that certain projects that are in some cases, you know, critically load bearing for folks' workloads stop getting maintained.

35:12And so there was one that was near and dear to us. So one of the many, many, many, many projects that Dan, I and others collaborated on Google is actually the starter project for one of our engineering managers. When she joined reporting to Dan at Google, Priya was a project called Kaneko. She has since left Google and just joined us and leads our container team actually. But so, you know, this project, like Google never really had, they never productized it, they never support it. But this project literally closed a three, four-year-old issue for how do I build a container with a Docker file on Kubernetes without mounting the host daemon socket, without running with privilege so that I can start my own Docker daemon, Docker and Docker, stuff like that.

35:59The kinds of things you don't want folks doing in your cluster if you're an SRE or your product security, like Conoco closed that issue. It showed folks like, this is how you can do it. And I think GitLab documented as one of the ways to build container images. I joke that I can't meet with a bank without them talking about how they use Conoco for image builds. And then, you know, one day Google stopped really merging any pull requests. And I think it took about six months before they openly acknowledged, like, they were basically archiving the repo. And so we had a lot of history with this project, obviously.

36:33And so we were like, and we knew how important it was to a lot of folks. And so we decided we would take it on and we would make sure that it was receiving patches for various security problems in the libraries that are in there. But we weren't going to do a lot of feature work. The nature of sort of, at some point, a lot of open source projects reached this quote-unquote done status. And it was pretty mature. It had a lot of the features that it needed from supporting the Docker file spec. And we wanted to give folks a stable and safe way of consuming that piece of software where we would keep patching the security stuff, but not do a lot of feature work that would potentially destabilize their workflows.

37:12And so that was really the seed from which this idea for Emeritus grew. And there have been a few other projects that we've taken on with a similar sort of philosophy, right? We will build images from them. We will patch any vulnerabilities that come in through dependencies. If things are disclosed in the package itself, we will look into that and try and remediate those things. but we're not going to do feature work, right? I think one of the great things about open source is if someone really wants to take on making substantial changes to a project, they can fork it and they can add their features to that.

37:48And if that fork becomes a successful alternative, then people can just start consuming that. And that's great. We're open source working as intended and that can become the source of truth that folks use if they find the features that are being added valuable. And so I think there's some great examples of like really successful forks that become bigger than the original project. I mean, I think a lot of folks don't appreciate these days that like Jenkins started as a fork of a project called Hudson because Hudson was owned by Sun and then Sun got bought by Oracle. And the person who started Hudson, Koseuke left Oracle and wanted to continue his source project.

38:31So he forked it as Jenkins. And now who talks about Hudson? I mean, I think it's probably a vanishingly small set of folks who even know what it is. I didn't know that story. I had not heard of Hudson. So that's, yeah, super interesting. I guess as we kind of like look forwards, I guess in the present and forwards, you have mentioned that Chain Guard is not just hardened containers anymore. So kind of using that framing you know i think looking at things like s bombs you know so this is software bill of materials we've covered these in past episodes as well this is more like runtime behavior like what is being fetched like what is the provenance this feeds into the fact that we do have regulatory frameworks coming in towards the end of 2027 one is the eu cyber resilience act which okay it's eu but it will no doubt frame still what happens on the US side.

39:22And no doubt you have EU customers, so they have to abide by that. And that is saying, you know, mandatory SBOMs, things like 24-hour vulnerability reporting. Like, are you kind of taking in that kind of thing into where the product's going as well? Or like, you're already thinking about that? I just, yeah. What are you guys thinking about all of this? I think it's a nice tailwind for us because I think we do a lot of the things that folks need in order to satisfy these things. All of our images carry SBOMs that we generate at build time, but we also go out of our way to make sure that SCA tooling that generates SBOMs works exceptionally well with our software and the binaries that we include in that software.

40:02And so we definitely do a ton of stuff to make it so that folks can either get the artifacts that they need from us or generate them from the images that we are producing. And I think one of the interesting things just to sort of circle back to the trivia breach, you know, I think the European Commission itself was actually compromised by that trivia breach, right? And so they disclosed that they were compromised by it because of that 24-hour reporting deadline. I think it was beginning of May. And something like 340 gigs of EU citizens' data was leaked as a result of that, right? And, you know, that's, you know, something that wouldn't have impacted them if they were using Chingard's images, right?

40:44And so I think that even if folks are seeking this really just to be compliant, there's a really nice security properties that you get out of adopting hardened images, adopting software that knows exactly what's going into it. I worry that when folks talk about things like SBOMs, that they aren't talking about the complete picture a lot of the time, though. And so everything that goes into our images is captured in our SBOMs. And I can assert that because everything that goes into our images comes from one of our packages. And all of our packages capture everything, like every file that ends up in the final images file system comes from one of our packages.

41:25And so we have this sort of chain of provenance for every single file. We know exactly where it came from and you can trace the builds all the way back. But that's because we own that full end to end. And so I think that folks who aren't doing this, right, you look at the way folks traditionally have built container images with Docker files, like Docker files, most of the official images are bootstrapped from a random tarball somewhere that has a rootFS. Hopefully that's a set of packages that that rootFS was built from. And then they start to add stuff up on top of that. Some of those come from the package manager.

41:59Some of those curl pipe to bash in the Docker file build. One of my favorite things to call out around SBOMs is what our CEO Dan calls the WordPress test, right? If I have an SCA tool, can the SCA tool tell me that WordPress is in the official WordPress image on Docker Hub? And I think we found one that can actually tell you that WordPress is in there. I think it's the BlackDuck scanner, which uses a sort of signature, like file signature based scanning. but because the official image like pulls down the source code and builds it in the container and like that's what you get there's no metadata telling you that WordPress is actually in that container right if you scan our WordPress images because it comes in through the system package manager it's represented in our package database every SCA tool on the planet can tell you that WordPress is in our images but it's also in the sbomb that you're getting from us so I think one One of the things that I worry about, it's not just mandating SBOMs that I think is important, is this notion of dark matter, right?

43:02How much coverage does the SBOM actually have of the files that are in your system? And like, I mean, can I give you an empty JSON document and be like, here's your SBOM, satisfies the schema, but there's nothing in it. It has 0 % coverage. Does it pass the mandatory SBOM check, right? And so, okay, that's an extreme example. But say I give you an SBOM that covers like 10 % of what's in the image in terms of, you know, packages that are installed. Does that count for mandatory SBOMs? Like this is one of the tensions that I think that I've always struggled with, with people like being like, yeah, mandatory SBOMs.

43:37It's like, this isn't a yes or no thing. This isn't a binary thing. We should look at it sort of like code coverage, right? What is the coverage your SBOM has of your image, right? Because an empty JSON document would obviously be zero. But I think the intent of it is to have 100 % coverage, but they're not saying that, right? And I think that's one of the things that I just take issue with in general when folks talk about SBOMs. They're not talking about the bar you should be clearing when you were generating that kind of metadata about images. That's super interesting. I guess these frameworks, and especially if they're trying to like implement them as mandatory at all, then, well, I guess it's this question, yeah, is it mandatory?

44:14So should it be mandatory or should it be there's a bar, but then do you say mandatory? And I guess they had to pick one. But yeah, I totally agree. I think this will initially lead to probably a lot of subpar SBOMs. I think at least getting the terminology and getting a lot more. I mean, I can think of so many developers if I was to say, just say SBOM, have no idea what I'm talking about. That's going to be helpful. Yeah. When the original executive order happened in the US, there was also this tension where like they didn't want to be prescriptive about what format you use. And there's two major formats, right?

44:43There's CycloneDX and there's SPDX, right? Which both have community backing, but they didn't want to pick a winner, right? And so there's no prescription around what format you use. And I think because of that, they also didn't really do too much around prescribing the structure because you'd have to do it twice and do all this other stuff. I would go so far as to argue that the package database that distributions include in their images is effectively an S-bomb, right? And so I think that you could arguably check the box. Every SCA tool on the planet can generate you a fantastic SBOM from a distribution's package database.

45:21It's the first thing any of them look at. It's all the other stuff that doesn't come in through the package manager where things start to struggle, like the WordPress test I was talking about. And so that, I think, is one of the things that there's some tension around and yet another gap in the like, OK, well, like what mandatory SBOM? We're not talking about coverage. What format? Like, what are the minimum elements? What is the depth of resolution? I could tell you, you know, at a package install level, I could go down into the file level within like a Go binary file. I could tell you the transitive dependencies with respect to every version of every Go package that got pulled into that binary.

46:00Then there's other fun things too. Go is really good about including that dependency information, but Rust doesn't do it by default, right? And so you have to enable auditable mode in Rust compiler builds in order to get that same metadata. And we do that by default in our tool chains because, like I said, we go out of our way to make it so that SCA tools work really well with Chaingart-produced images and binaries. But you can get images today from AWS. They built a lot of stuff in Rust, but they're not turning on auditable mode. And so there may be vulnerabilities in some of the libraries they're using, but you can't tell because that metadata is not there.

46:35And if that's not something an SCA tool can recover because the metadata is not there and it's not captured in the S-bomb, who knows? You don't know whether or not you were vulnerable to that. And so that's part of why we go out of our way to work really well with SCA tools in addition to capturing as much metadata as we can. But there's file coverage, package coverage. There's depth of resolution. There's how you actually structure all of that information as well. There's so many things that are not well scoped. And so like saying mandatory SBOMs without any of that other stuff means it's really hard to satisfy that in a way that is particularly meaningful.

47:17And so as a result of that, right, like we do produce SBOMs, but like they don't necessarily have the same shape everyone expects because everyone thinks, you know, you ask 10 different tools what the right shape to produce an SBOM is, you'll get 10 different answers. But if we work really well with SCA tools that can generate their own SBOMs, and you can verify the image came from us via our image signatures, you can generate your own SBOMs from our images that we know will work with your tool. And so that's why we do it sort of both ways, right? And we don't think that the SBOM space has really matured enough that like just generating an SBOM is enough.

47:54And it's because of all of these unanswered questions about shape, coverage, depth, etc. that I think make it a hard thing to satisfy, at least in a useful way. Yeah, absolutely. Kind of related, like as we're sort of cruising to the end of the episode, but there's a couple of things. First is, related to what you've just been saying, enterprise adoption, what friction points could you, I guess, speak to? That's not a friction point. It's just a perceived friction point. and actually, I guess, using chain guard is like, you should just talk to us. There's definitely some fun out there about stuff like migration and things like that.

48:30Fear of the unknown. It's like, I don't know how hard it is to migrate. So it must be impossible, right? And so, you know, this is some of what we deal with coming from other distributions, you know, like Debian, like a RHEL-based distribution. And so I think that, you know, we've gone out of our way to accommodate in various ways as broad a base of compatibility as we can, right? So we use the same package manager that Alpine does. But one of the big sources of incompatibility in Alpine-based images is the muscle LibSee. They use a different LibSee than virtually every commercial distribution on the planet, as well as many of the non-commercial but widely used distributions.

49:07So like Debian, Ubuntu, all the RPM-based distro, Amazon Linux, RHEL, et cetera, they all use GeolibSee. And so when we bootstrapped our distribution, even though we decided to use the APK package manager, we bootstrapped our distribution around GLibC to get that sort of maximally broad compatibility. And so I think there's definitely a lot of FUD around migration, but it's definitely a spectrum. And I think the main thing is, OK, depending on the package manager you're coming from, you know, if you're coming from Alpine, it'll look very familiar, APK ad package name. If you're coming from Debian, yeah, you'll have to change apt-get into APK ad.

49:44or if you're coming from the Red Hat ecosystem, you'll have to change Yum install into APK ad. But we have tools that help streamline this, make this easy. We have a new tool that we call the Gardener that helps drive a local migration for you, leveraging our agents to help you do some of that package mapping and stuff like that. So if you're coming from Debian, we can be like, oh, this package is called this in the Debian ecosystem, but it's this within the Chain Guard ecosystem. or if you're coming from the REL ecosystem, it may be called something else, right? And so mapping package names, doing those kinds of things and really helping you drive that further and further to completion.

50:23We have a bunch of other stuff that we're working on to sort of smooth migration. But in some cases, it's really just a matter of dropping one of our images in. So the vast majority of our images are application images, which most folks just take and run, right? Like something like an Envoy, you typically plug it in and run it as a proxy with some configuration and maybe you're running something like, you know, Istio, how many folks are modifying their Istio images? They're mostly just taking it and running it through a Helm chart or something like that. I think the one caveat to that is folks who want to like inject their own root certificates into it.

50:58Maybe they are using something like Z scale or something like that. But other than that, right? Like it's, it's actually relatively uncommon for folks to be modifying the application images. And so those will just drop it and run as the application. You don't have to worry about it having a different package manager. You're not installing packages. And then when it comes to migrating stuff like Docker file-based builds, we have tools that will help drive that migration and help you get as close to that finish line as possible, if not all the way there. And then with things like our library's products, a lot of times our customers' goal is to effectively just turn a switch.

51:36A lot of our customers will proxy our language packages through something like Artifactory or CloudSmith or Nexus, right? And they've already done that to sort of insulate their internal networks from developers just being able to pull whatever. And so when you have that kind of mature setup where you can regulate what packages your developers consume and it's coming in through Artifactory, maybe it's just passed through from the outside world, but you have that cut point, you can cut that over from pulling from say Maven Central to pulling those from Chain Guard. And your developers may not even know that you switched it to Chain Guard.

52:12So on the library side, it's much more transparent in terms of dropping in and replacing things. That's our goal to be as drop in as possible. But I think some of where that can deviate a little bit, it's the same migration problem you'd have. If you want to move to one of our images that doesn't have a package manager, it's the same as moving to a Google Distro list image, which half of this cloud native ecosystem is using these days. So it's clearly possible. But if I were to move from like a Debian based image to a RHEL based image, I would have to rewrite my package install lines anyways and figure out the package mappings and get it working again anyways.

52:49And so any sort of migration like that will have a little bit of work. But folks are already going through those kinds of migrations to like one LTS release to the next. And, you know, where a lot of system packages are changing. And so we try and give folks something where we aren't an LTS based distro. We are a rolling distro. So if they migrate to us once, which we can help them do as smoothly as possible, they shouldn't have to go through that every four years when there's a new LTS as well. Right. We can hopefully, you know, be the last migration that you ever do. And we, like I said, are going out of our way to make that as easy and painless as possible.

53:28Yeah, amazing. For anyone, I guess at that sort of enterprise level, but medium company, whatever you want to call it, like just that on-ramp isn't maybe as scary as it sounds. Very briefly, just we were talking about it before we started recording, Anthropic Mythos Fable model or Fable, which is derived from Mythos. Super curious, just like this is very bleeding edge because it's only been announced in the last 24, 48 hours. But how does that affect your day to day or like you've been, I think, using a little bit like what are your first thoughts? I think it's when it first came out, everyone freaked out.

53:59And then there was a whole group of folks who were like, this is just marketing. I've seen some of the findings. It is not just marketing. These are real vulnerabilities. It has very good capabilities. There's several different aspects of it. I think I mentioned the Firefox numbers, right? They found a whole slew of vulnerabilities that other agentic tools that they had been running weren't finding. And I think one of the things that makes it exceptionally interesting is its ability to chain together multiple vulnerabilities, which in real time, we're watching it close the skills gap that's needed, like this notion of sort of scarcity.

54:36Right now, the skills needed to find and weaponize vulnerabilities are scarce. It's hard. It takes time. right? And that time is dropping precipitously. And so I think the Fable model that they released today has all kinds of security controls. The first thing I threw at it mostly for grins was to audit security isolation tool that I've been, you know, having Opus just audit in a loop and find and fix vulnerabilities in this abstraction. I've been very impressed with how good Opus has been, you know, I've been trying four, seven, four, eight. So I threw this at Fable and I got this prompt back being like, it looks like you're trying to do security stuff.

55:14I'm downgrading to Opus 48. So it wouldn't let me do it. So they said in the announcement that they had put in security controls around keeping folks from doing this for vulnerability research. And it caught me. So I think there are controls that will keep folks from using it in that way. Folks always find a way around those controls though. And so I think the thing to bear in mind is this is not marketing. This is very real. This will find tons of vulnerabilities that other models probably haven't found yet. And folks need to be prepared to patch things at machine speed, because you could already weaponize vulnerabilities at machine speed.

55:52And now with Mythos coming into the picture, being able to find even more and more effectively weaponize vulnerabilities at machine speed, if you are prepared to patch things at machine speed, which is really what we've been building towards at Chain Guard since before we even realized this was a problem folks were going to have, you're going to be pretty unprepared for the world of hurt that I think is going to play out over the next six to 12 months as at least the initial tranche of Mythos vulnerabilities become known to the world. And then it's a muscle you need to build even if you get that initial tranche patched, because I would be very surprised if we didn't continue to see step function improvements in model capability, the quality of the harnesses for leveraging the models to find these vulnerabilities.

56:40And it's going to be an interesting next year. For sure. It really is. So on that note, thank you so much again for coming on again. learned a lot this time as well as i and i think the audience did last time so yeah who knows we'll maybe get you back on in another two years we'll hear because you know this landscape is just changing so much and so fast so thanks again and i'm sure we'll catch up again thanks for having me

57:19Thank you.