Sponsored: What npm 12 fixes… and what it doesn’t

0:04Hey everyone, this is Casey Ellis for the Risky Business Podcast. We're here with Feross Aboukhadijeh from Socket. We're talking today about some of the changes that have happened in the NPM ecosystem, where they've worked, where they haven't, and what Socket is seeing out there in the wild. Good to see you, Feross. Yeah, good to see you, Casey. So yeah, let's get started, man. What's been going on? Yeah, so NPM version 12 just went GA earlier this month. And the big headline is that install scripts are now off by default. And so quick context for anyone who doesn't live in the JavaScript world.

0:41When you run NPM install, a package can declare scripts that NPM executes automatically as part of the install. They are called pre-install, install, and post-install. And they do exist for some legitimate reasons, mostly compiling native code. But the security of them is just absolutely horrible. Because the moment that you install a package, it can run arbitrary code as a U on your laptop or your build runner. And this happens before a single line of your application has even loaded. You don't have to import the package before the attacker can run their code. And so, you know, kind of as of version 12 of NPM, they kind of flipped three of the settings.

1:21So now install scripts don't run unless you allow them. Get dependencies don't resolve unless you allow them. And finally, dependencies pulled from arbitrary remote URLs don't resolve unless you allow them. Okay, so that all sounds like good news. But some of what you guys have been saying is that there are some gaps in that strategy. Yeah, I mean, I think, you know, attackers are really quick to evolve. And so one of the things we saw right away was folks started relying on this implicit behavior where if a package ships with a file called binding.jip, then NPM will just kick off a native compile and it'll allow the attacker to run code through there.

2:00And so it's been kind of, I guess, interesting to see sort of how quickly folks have evolved and kind of, yeah, I guess, like worked around these changes. So while these changes are really good, and I'm glad to see them, I think it's the right direction. By the way, I think it's frankly been a little late. These changes have been a little late in coming. Socket's been kind of flagging these risks, install scripts, get dependencies, and remote URL dependencies in our product for three or four years and flagging those for all of our customers. I'm glad to at least see the default package manager finally taking that position.

2:34But I think the key thing to think about with this change is that all Intacr has to do now is move their payload from the install script into the package itself. And now the moment that the developer actually imports the package, the payload runs. And so the question I ask is kind of like, how often do developers install packages that they don't intend to later import? Typically, you install it because you want to use it, right? And so this really just kind of moves the timing of the payload just kind of a minute or two later, which is good. I definitely think it will help. But I think we've kind of already seen a number of attacks that have kind of made this shift already in real time in the middle of ongoing campaigns in order to kind of basically work around these mitigations.

3:20Yeah, let's talk about that a bit more because, you know, in the middle of all of this and, you know, you've got Socket's position in the ecosystem. You've got obviously your background is like the JavaScript supply chain, you know, guru in that sense. So you've obviously had a lot of experience and a lot of visibility on how this has all kind of evolved over the years. But then on top of that, there's a lot of growth that Socket's experienced. And that gives you a lot of insight into kind of what's happening in the wild. We were talking a little bit before we started recording on you actually seeing some of the campaigns shift like midstream around some of these changes.

3:57Can you go into that a bit more? Yeah, for sure. So let me talk about one campaign. This one started early in July. It was a package called J Scrambler, which is a commercial code protection product. And they have an NPM package that you can kind of wire into your build pipeline to use their product. and someone published a release of it with a install hook that never existed in any prior version. So Socket, you know, the thing we do, like our bread and butter is we look for this stuff. So we flagged it within six minutes of publication because we're doing this sort of broad ecosystem-wide, you know, monitoring and surveillance.

4:34So we can flag these things really quickly. But, you know, I think the, I guess the kind of issue with the, with, I guess, let me walk through kind of what the, what the package actually did just to give people some, some idea of what the, what the payload was. Yeah. So, so kind of, it was a, it was an info stealer. They had different binaries for each operating system. They hid it inside of a file, you know, and they, when it ran, it kind of searched the system for crypto wallets, you know, browser extensions like MetaMask and so on. But they also kind of targeted cloud providers. So any access to like the GCP metadata service to steal service account credentials, the ECS task metadata endpoint, secrets manager, all these kinds of things, as well as Discord, Slack, Telegram, every Chromium browser.

5:22And of course, the AI coding assistants are a really ripe source now. So they kind of look through Cloud Desktop, Cursor, Windsurf, Zed, all the different config files because a lot of folks put their API keys into those, you know, for their MCP server configurations into those files. And so, you know, what we saw and kind of how the attacker adapted to your question was basically after about three hours of us flagging this attack, we saw the same actor publish, you know, additional releases as they kept trying to evade the blocks that were going out. And the payload was exactly the same, but the kind of way that they triggered the, you know, the delivery of the payload just moved from the pre-install script to what's called a self-executing function in JavaScript, which just means, you know, basically something that kind of runs right away as soon as that file is imported.

6:12You don't have to call any functions. You don't have to, you know, necessarily use the package in, you know, by invoking methods in it or functions in it. You can just sort of import it and then boom, like the code is kind of run as part of like the setup of the of the import and so you know uh it sort of just kind of immediately moved moved around this this npm v12 um you know protection um and it kind of it also defeated scanners that um you know i'd say there's some like lesser scanners out there that only inspect install hooks and they're pretty naive and so those those scanners then started reporting that the package was safe um so i think you know it's just it was kind of interesting to see like mid campaign you know then just a few hours you know that that type of evolution of their strategy yeah no definitely i mean it's it's interesting to me the uh the game of kind of cat and mouse that goes on um in in ecosystems like this one uh and it's not you know too dissimilar from what you see in other other kind of versions of the same thing out there in the industry across the board but you know some of the stuff that uh that you guys have worked on to basically kind of bypass that problem and kind of approach this whole idea of package security from like a first principles standpoint you want to go into into that a little bit yeah yeah definitely so i mean this is kind of a uh i don't know like a i don't say pet peeve or just sort of a hobby horse of mine to talk about which is just you know this idea that um we can look like at the end of the day what our our position is you really have to know what the code is doing that's the real source of truth that's the fundamental kind of trust question is should we be using this code because what is it going to actually do when we run it?

7:46What data is it going to access? What endpoints is it going to connect to? And so maybe like kind of a way to kind of make the point of maybe tell a story of another attack that we tracked. Is it cool if I do that? Yeah, totally. Yeah, so there's this other package, a couple of days actually from the other one, earlier, again, earlier this month in July, called Async API. And this is a kind of, I guess, a tool for event-driven APIs and ships as an NPM package. By the way, I keep talking about NPM just because it's the fun one where all the noise and fires are. And we all love to kind of pay attention and look and point at them and say, look at how bad NPM is.

8:23But I just want to point out that we see this across all the ecosystems, right? I mean, we track Python,.NET, Java, Ruby, Rust. Like it's the same thing everywhere. But as you say, NPM is like the currently topical cluster of trash fires that are kind of drawing attention to the broader problem, right? Absolutely. Yeah. I also want to draw attention to the Chrome extensions and to the VS Code extensions and to the, you know, cursor extension, all the editors, you know, different, you know, kind of skills, right? Anything that a developer is bringing in, you know, that's all part of the supply chain.

8:57But anyway, I digress. Back to kind of the story about async API. So, you know, basically, you know, so this is a package that, you know, helps you do, you know, use this, use these event-driven APIs. And they inserted the malicious code directly into legitimate source files inside the package. They put it into one of the utility files, into an error handler. And so it would run right away as soon as Node loaded the module. So again, bypassing the new protections in NPM v12. But I think what was kind of interesting was, first of all, they spawned some hidden processes that used pretty much every decentralization technology that I've heard of.

9:39I used to develop a torrent app, so I know about a lot of the different DC web technologies. So this thing used like LibP2P, the BitTorrent DHT, IPFS, Ethereum smart contracts, Noster relays to do their C2. So it was really quite an impressive slew of basically every possible C2 channel. And it's kind of sad to me as a decentralized technology person to see kind of the killer app often being these C2 use cases. But I think the part that was actually most disturbing was that the attacker here didn't steal an NPM token. They actually just got a commit into the repository. and the project automatically built the project and then published it through GitHub Actions.

10:32So they have this kind of published workflow set up that automatically triggers publishes when certain conditions are met, triggered through GitHub Actions. And it uses the OIDC flow that the entire industry has kind of spent two, three years kind of telling all maintainers to move to so that we can do provenance and have integrity and do SLSA, provenance attestations, and all this kind of basically fancy cryptography to help determine that a package hasn't been tampered with. And in this case, it didn't matter because the final artifact was malware and it had all the signatures and all the cryptography signed off saying, hey, this was built in GitHub Actions.

11:08It was built in a trusted environment. It was built by the correct GitHub Actions runner and nothing was tampered. And so it didn't really matter in this case because someone got a commit into the release branch and the project's own release pipeline just did the rest. so it's just very well-dressed malware at that point yeah i mean look i don't want to be too harsh on it because i think that trusted publishing has a place but i think you know it never answered the question about whether the source code was clean and that is really um for the current supply chain wave we're dealing with is the question we want to answer and so i think um you know i think signatures all the way down cool great um cool to see that the commits are signed but ultimately you don't want to have a beautifully signed, fully attested, you know, verifiably reproducible, you know, piece of malware basically, right?

11:56It's interesting. So like, how do you, I mean, I guess there's, there's sort of two, two kind of angles on the problem here. One is this idea of like these kind of cat and mouse shifts being useful, but not necessarily complete in terms of how they kind of address the overall ecosystem problem and then this idea of things getting kind of dressed up you know even better um and looking more credible even though they're malicious as a result of that like how do you solve that problem i guess look i think that i don't want the takeaway to be that provenance is completely feeder because i think that would be the wrong lesson but i because i do think it worked in in some ways you know it did give us the workflow the branch, the commit, and the runner, where the package, basically where the attack took place.

12:43And so six months ago, I think that would have been a harder question to answer. How did this get published? It would have been kind of like, who knows, maybe the maintainer lost their token. We don't really have any other details. So it does give us a bit of a forensic trail, and that is real. But I think it still doesn't tell you whether the artifact is safe. And so I think we just need everything in security, multiple pieces working together. And I think from our perspective, the socket perspective, the really hard part is how do you take a massive pile of completely untrusted code, often in dynamic languages, which are notoriously hard to analyze and figure out, you know, do I let this into my application?

13:19Do I let this into my environment? Do I let the vibe coders in my company that are, you know, writing code on, in some cases, unknowingly as cowork, you know, kicks off some job and starts installing packages, like how do we decide whether we want to allow those artifacts into the environment or not. So I think it's really, it needs to be this multi-level approach. And I think, hopefully that doesn't sound like a cop-out. I just think every piece of the ecosystem kind of plays its role and has to kind of work together here to get us where we want to go. Yeah, no, totally. I mean, to me, that doesn't sound like a cop-out at all.

13:55It sounds like a fairly familiar story across a whole range of different problems that we've tried to solve in cybersecurity over the years. Like you've got these deterministic gates and things that you know that you can do that are going to improve kind of the situation from the outside in, but ultimately they don't necessarily address the first principle that you're trying to think through from a risk and a security standpoint. So to your point, like approaching it as a pincer move does seem like the best thing to do. It seems to be the thing that we've done in a lot of areas. Yeah, totally.

14:26And speaking of like multiple levels, I think there's another thing I want the listeners to know, which is that there's kind of a change coming down the pipe that anyone using JavaScript should definitely be aware of, which is by the time people hear this, the first kind of two big NPM token changes is kind of coming down the pike. So the first one, which is going to have already happened by the time people hear this, is that access tokens that bypass 2FA are going to lose a lot of powers. So you're not going to be able to use those tokens anymore to change maintainers, to edit org membership and things like that.

15:06So it'll kind of completely close the account takeover surface. But publishing from those tokens is still going to work until around January, 2027, when they're going to kind of shift it and that's the second stage to what's called a staged publish where you can use the API to stage a publish but ultimately you're going to need a human with a 2FA approval to actually ship a package which is another change I'm actually pretty excited about because I think that's going to that's going to probably meaningfully move the needle there. The implications of that are interesting because you could kind of reasonably expect the last hurrah over those six months as well right?

15:42Definitely I mean well not to mention all the stolen credentials that people have already collected for NPM maintainers over the last number of months that I think are going to keep hitting. But I think the one thing I want to flag too is that this change about blocking install scripts, it's sort of nuanced in that the new default behavior is that when a package has an install script, NPM just kind of ignores it. It's kind of a soft fail and just proceeds with the installation and it just reports in a console that an install script was skipped. But I think what's kind of nefarious about this is actually that oftentimes those scripts are legitimate.

16:26I mean, they are building your embedded databases, your image processing libraries, your fast password hashing libraries, which are built in C and that do need to be compiled. And so what's going to end up happening here, my fear is, I guess, is that... Those are going to get skipped and you're going to get production failures, right? Production errors. Those are going to be runtime errors when those calls happen and there's no compiled module for Node to call out to. And so I think a lot of teams are probably going to do there is just blanket allow all install scripts to make the CI go green, right?

16:59and so I actually recommend folks enable an opt-in setting that turns the soft failure into a hard failure so you have to make a kind of concrete decision for every one of those scripts at the time you bring that package and that's my recommendation for teams today for that very cool very cool well that's I mean that's a lot there's a lot happening in this ecosystem it's great to have you guys kind of you know wrapping your arms around the entire thing and helping us all stay secure everyone this has been Feroz, a booker DJ for Socket.dev. This is Casey Ellis for the Risky Business Podcast. Cheers.

17:34Thanks, Casey. I appreciate it.