npm under siege (what to do about it) (Friends)

30:25And then I believe the exact timeline was the package was published, malicious package was published, and seven more versions were published over the next like few hours. Then NPM took the packages down. It looks like the following like three hours later, NPM took them down. And then the NX maintainers like revoked the account, the compromised tokens from NPM another hour or two later. So overall, the whole thing was over in about six hours from beginning to end. So the team did a really good job responding and reacting to it, obviously. Six hours is pretty fast. What do you all do at Socket when you detect something new like this?

31:14Obviously, your customers need to know about it, etc. But do you have an open channel with NPM in terms of like, hey, you better look into this right away? How does that get moving? Yeah, unfortunately, we just use the same reporting mechanisms that everybody has, which is if people don't know this, you can go to any NPM package page and you can click report malware and just fill out a form and tell them that it's that it's malicious. So we just report through that mechanism. And sometimes they're really fast. Sometimes they're really, really slow. It just really depends on the impact. So the more impactful the package is, the more they tend to respond quickly in our experience.

31:50for some of the less popular packages that we report, we sometimes never get a response. And they just leave that malware up for, like there's some that's been up for over a year now. They just never took it down. GitHub's usually in the picture for that, right? I mean, if you're on NPM, the repo's usually on GitHub, right? There's usually a mirror of it. Why don't I also publish an issue or something like that as well? Oh, like on the repo? Yeah, yeah. Because I mean, that's two places. It's like public awareness and then maintainer awareness. sometimes you don't want public awareness yeah i guess it's probably true well i guess for the ones who are like not responding to you it's just hanging out there and there's no public awareness right right yeah so i mean so one thing that's different about finding um malicious attacks versus uh vulnerabilities is that with vulnerabilities you have to be pretty sensitive with how you talk about them publicly because you could by not giving time to the to the software creator to fix the problem you could be hurting users or you could be hurting you know companies or end users of the software.

32:50So it's, that's why we have responsible disclosure. We have 90 days usually that we give people before we go public with the information. And it's sort of trying to balance, like giving them a chance to fix it with also realizing that, well, you know, some vendors never fix things, some maintainers never fix things. And so you have to have a time limit on which you say, look, there's actually more harm being done by not telling the people who are using the vulnerable software about this so that they have a chance to protect themselves. Because the longer we wait, the more that other people could discover this.

33:22So 90 days, you try to work with them privately. And then at some point, if they don't fix it, then you go public with it. And that's sort of been called responsible disclosure. But with malware and these types of supply chain attacks, it's kind of different. So unlike the vulnerabilities, there's really no harm in us shouting from the rooftops that a package is malicious because it just helps everybody. It's already out there, yeah. It's out there, it's already hurting people, telling everybody that it's out there. There's no harm. The only person who's harmed by us telling everybody that a package is malicious is the attacker.

33:55Because people can defend themselves. So that's the nice thing, is we don't have to be too secretive about these things. We just can tell NPM, we can tell GitHub, we can try to get all these repos taken down. But to your point, Adam, I mean, the problem with the issue, I mean, that's an interesting idea. We never thought of just opening an issue. But the problem is a lot of these things are like if you're dealing with a typo squatted package and it's not been a takeover, then putting an issue on their GitHub isn't going to help because they can just delete it. Like they're they're the owner of the repo.

34:24So if you're kind of going to the attacker's turf and putting an issue there, like it's the real issue is just that the that the package is bad and that people are accidentally installing it. So, yeah, but for things like a big popular project, we would absolutely open an issue and like, or contact the team directly and say, hey, we found your code is compromised. You need to like take it down and take steps. We do that all the time. let's go back to the github action stuff because that seems like a really fertile ground for getting your malware out there and i'm curious if there's you guys found the malware in the nx package you are scanning all packages on npm i assume what about github actions do you have any sort of proactive steps or tools for people's github actions that they could run a tool against it or anything have a socket page that says your actions are secure or not it's funny you ask uh it's uh it's something that i can i'll give you a little preview of uh we're going to be announcing github action support uh later in october so oh wow yeah what that means is we're going to treat github actions like any other ecosystem so we're going to treat it just like npm where there's a bunch of packages out there that you're trusting and we got to scan them and believe it or not, there is a supply chain of these things, right?

35:45There is, you can have actions depend on other actions. They have, I think, like reusable actions is what they call them. And they basically, you can have dependency tree of actions. So it's important to treat them, treat them just like any other untrusted third-party code and to scan them. Yeah, because this would have pointed out to the NX team early on that the way they wrote their action was insecure. Having said that, they found it and fixed it. And so that's kind of freaky. is like you can go back into the past and execute old code via a branch pull request. I assume only GitHub can fix that.

36:20I mean, you have to be able to like disable old branches or something or how would you actually mitigate that particular thing? Because that sounds gnarly. I agree. It does sound gnarly. The design of GitHub Actions just has a lot of foot guns in it. I think it's really unintuitive to the user in a lot of ways. I mean, I know to be clear, they do, they do document this pretty, pretty clearly. Like there's a reason why the NXT probably fixed this, which is that get up to document this and they did try to raise awareness of this at some point. But, and then, and, but, but like who, who knew, I don't think at least I didn't, I didn't, it was not widely known.

36:58I believe, I think it's fair to say that, that you could go back and run old actions. And that therefore security fixes in those old actions, you know, are impossible are possible. Yeah. Yeah. I mean, they're just impossible. because those exist in perpetuity unless there's some sort of switch or toggle that says don't run, I don't know, actions on old branches. I don't know how you'd say it, but something has to be said because copycats copycat for multiple reasons. One of them is because it's effective. That was effective. And now it wasn't known to you, it wasn't known to me. It probably wasn't known to a whole bunch of malware authors and now it is.

37:36So something has to be done about that. but it's hard to i'm sure it's hard to fix yeah i i i'm curious what the what the if there's going to be a fix there because you know we we saw that get up did respond to all these attacks and they have a bunch of changes for improving npm security but i don't think get up actions was part of any of those announcements so what do they do in the case of like old secrets you know like i accidentally pushed a thing six months ago that had a secret in it i've since removed it from my repo and there's a way to go back and rewrite that history or something isn't there where you can say because that's in your git history also in perpetuity unless you can go expunge it somehow maybe it's a similar technique so there's um for that case i mean the the most important thing to know is like you as a developer who's leaked a secret need to go and absolutely revoke the secret even if you're able to quickly force push and get rid of that commit like once it's out there the safest thing to assume is that somebody's seen it so you have to absolutely revoke it But then on top of that, like you said, it's probably a good idea to also try to get rid of it out of the history just to be extra safe.

38:42You can do it like a rebase or something, right? Like there's Git tools that allow for that. Yeah, there's Git commands. I think GitHub actually has a good guide on this too, like how to expunge it from your history. It'll walk you through how to, in your local copy of Git, it'll actually go through and you can kill those commits from the history and then you can force push the whole repo. But there is a kind of a problem with the way GitHub does its caching, where basically even after you force push over a repo and you've gotten rid of certain commits, if somebody knows the commit hash for the commit that you deleted, they can still go and find that commit in perpetuity on GitHub forever.

39:20So you have to literally contact support and give them the list of commit hashes, and then they will go and manually expunge those for you. There's no automatic way to do it. So you can literally, like the only way to do it is to delete the entire GitHub repo and just create a new repo from scratch. You can't, and also if you have forks, if you have public forks of your repo, then you're scratch. You can do a lot locally before you push it, but once you push it, there's no stopping it really. there's caching there's it's out there man it's the nature of git so it works it's out there don't don't push your you know you can don't change you can change tokens but don't don't push your face or your fingerprints or your your things you can't change up don't push your face i like that so let's imagine this future world where sockets new github action support is out there let's say it's mid-november and your tool's out and your tool i'm your customer so you're scanning all my stuff you know proactively looking at all my github actions and you find that i've done the same thing that nx did and i have this vulnerable action uh what would your advice be to me because it sounds like it's impossible for me to get rid of it even if i know about it like obviously i can get rid of it off of my main repo now but i got those old branches in perpetuity there's no like what's your advice socket yeah well so the first version i like the way you frame that um you know in our first version um we're not going to be scanning like the code that you put into your own action workflow files, we're focusing more on the supply chain.

40:50So like the, the reusable actions that you're sharing ones, the shared ones. So things like, you know, the ones everybody uses to check out repos, to do caching, to do like these different, different things they do. So the way to think about it is we're not scanning like your package JSON for problems in the scripts that you're, we're scanning the dependencies field. So we're just looking at the, at the chain, you know, the supply chain part of it. So down the line, we might do that because obviously this is a problem. Uh, but, uh, so I don't have an answer to your question cause we have, that's not what we're trying to do with the, with the first version.

41:20We're just focused on the supply chain part and, uh, okay. Doing that for now. Well, everybody go out there and check your actions. Um, there's probably, here's some good socket content ideas. If it doesn't exist on a GitHub doc is like, here's the 75 ways you can shoot yourself in the foot security wise with a GitHub action. Like here, don't do these 10 things or whatever the number is. Um, certainly some of Some of that's probably out there somewhere, but a canonical source for that would be spectacular. Okay, so man, that one's gnarly in lots of different ways. The LLM thing still strikes me as like evil genius move.

41:55Like that was really smart. Are they just assuming that you have cloud code? I mean, that thing blows up if you're not running. You said it runs against Gemini as well. So does it just like detect for whatever CLI tools you have installed and then run against that list? or what happens if I don't have any? You know, I'm a Luddite and I'm like, I'm not using AI. You're safe. There are a lot of those still, but I think the, yeah, I mean, you would be safe, I think. I don't think you had a backup plan. There you go, friends. One more reason to stay in the dark, you know? The Luddites will definitely use this as a good reason to not, to put their list.

42:34Well, they'll use any reason they can get, you know? I mean, isn't anything you use a deployment strategy for some hacker? I mean, you can't, just because you don't use this doesn't mean I use something else. The more powerful the tool, the more powerful it can be abused, you know? And it's automated too. Those suckers are powerful, aren't they? They are very powerful. I mean, the other thing is they're using up your tokens too. They're not even paying. They're paying. How rude. You're paying to attack yourself. You're going to get the billboard at the end of the day. That's adding insult to injury right there.

43:02Imagine if they just used your tokens to like run some arbitrary, you know, cloud code lookups or something for their own use cases. They just want your LLM tokens. Those suckers are valuable too. Is the question you're asking, Jared, like immediate actions for certain types, like developer versus security team kind of thing? Is that what you're trying to get at? With regards to what? Well, if you're going to be, if we have Feras here and there's advice to be shared, what are some immediate actions for developers? With regard to NPM? Just these challenges so that you can do what you can to protect yourself at the dev, at the security, at the team level.

43:39I was not going there, but I love it. So for us, how can people do things that are smart to protect ourselves from these things? Well, so for the GitHub actions one, I mean, the most obvious thing to do to start with is make sure you're not using pull request target and you're using the safe version. And then I wish I had an answer prepared for what to do about the historical commits, because I do think that that is the that is the actually interesting part of this attack, I guess. I don't actually know off the top of my head what you can do. I suspect that you could probably delete the branches.

44:13Don't quote me on this, but I believe, I don't know, can you open, you can't open PRs against like an old commit hash. You can open PRs against an old branch. So if you have no branches pointing to the, you know, if you have no branches with the vulnerable action, then that might be the defense. But again, don't quote me on that. I haven't tested it, but that's what I suspect the defense is. So look into that. But I think the real steps that developers can take are more about just the broader NPM supply chain that we've been talking about. That publisher piece, yeah, that's interesting. But most developers are more worried about the NPM packages that they're using and the risks that are coming downstream.

44:54They're not publishing NX, they're a user of NX. So how do they protect themselves as a user of these tools? I'll go through a bunch of stuff. I'll start with the obvious stuff first, like that everybody should just be doing because like it's just easy and it's common sense stuff. So lock files, use lock files. If you use NPM these days or really any modern package manager, they will create a lock file. And don't turn that off. Some people turn that off for some reason or some people will frequently blow away the lock file. Like they'll just delete it and reinstall from scratch. And when you do that, you're really just rolling the dice about what's going to come in.

45:37So the lock file is nice because it pins down exactly what dependencies are going to be brought in. And that means that when other people on the team or when you at a future date try to install the packages, you're going to get that exact set of versions and not be just pulling in whatever was published five minutes ago on NPM. So that gives you alone quite a bit of not just security protection, but just like reproducibility in your software. You can build this software project two years from now without everything breaking. So that's an easy one. A newer thing you can do that is pretty powerful that PNPM just shipped.

46:14So you have to be using PNPM, although I think that Yarn and others may be considering it as well, is to do a package publish delay. So what this means is you basically tell your package manager not to bring in any packages that are newer than a certain timeframe. So you can say, I don't want to use anything published in the last seven days. Just don't ever give me anything like newer than seven days. And that the idea behind that is that a lot of the most like recent attacks we've seen have been caught within a few days because they're so noisy. They're so big. There's so many companies like Socket, like trying to find these things and others just it's just it's too like if you just look at the track record, the really big, nasty ones we've caught pretty fast.

47:01So the thinking is, oh, seven days will be enough time to kind of let things bake out there before we bring them into our project. And so it's just a config option. It's a one line option that you can add into PNPM and just tell it seven days. And then you'll just be living in the past for, you know, just be seven days behind everybody else and and and protected from at least the worst ones. It's not a perfect foolproof solution, but it's pretty good. It doesn't cost you that much if you can afford to be seven days behind the latest hotness. Right. I think that's a great option. I know there's people that will stay versions back for the same reason, even though that's a much worse solution, because a lot of times when your version's back, you have the old insecure version and a new, you know, locked down version of against particular vulnerability has shipped and you don't have it.

47:46And so that can backfire quite readily. but it also is kind of one of these I just don't want to be on the bleeding edge I just want to be a little bit back from the bleeding edge and I think a time delay is much better than a version delay to accomplish that same goal yeah you can also override it for specific packages so if there is a really bad vulnerability that you need to fix you can add in a line so the config is called minimum release age and they have a minimum release age exclude option as well that you can put in specific packages to bypass it. So it's a challenge because you have this trade-off.

48:21It's a direct trade-off of like, the faster you upgrade your packages, the more safe you are from software vulnerabilities. But then the faster you upgrade, the more vulnerable you are to supply chain attacks. So that would tell you you should upgrade slower. So there's some middle ground where you want to be like behind a little bit, but not too behind, especially when there's a vulnerability. And that's the art of it, is like figuring out how to... And do you think seven days is a pretty good sweet spot for that? I personally think seven days or even 24 hours will, will, will, uh, get you a lot actually, uh, because mostly because there are, uh, people out there like, uh, scanning for these things and trying to find them and taking them down to protect the whole community, even if you're not a customer.

49:04And so that will give you some just almost like herd immunity protection just from, from, you know, uh, uh, us doing that work. but I don't want to imply that it's perfect because there was a study done there was an academic study done on how long malware persists on package managers that was done back in 2021 and they found that on average malware persists for 200 plus days now that is looking at all malware so it includes the really unpopular packages that get 30 downloads, not the ones that get$2 billion. So if you look across all of those, right, then you see that a lot of the less popular ones are sticking around and driving that number up to over 200 days, right?

49:53So you still might get a typo, you might still might typo a package and install some malware that's just been sitting there on unfound for a year that you got unlucky and you hit it and now it ran on your machine with an install script or something like that. So seven days won't, won't, won't protect you from everything. But, um, but given that it's one line to add and it will do some good, I mean, I, I think people should just do it. Yeah. I think people should just do it and it'll help. You mentioned typo squatting. I remember when you first started socket for those who are new to the show for us and us go way back before socket even existed.

50:29And so we've been along for, uh, for the ride of your career to a certain extent. and I remember that typosquatting defense against typosquatting was one of the highlight features of Socket because nobody else was doing it you know how to do it right we talked about signal versus noise and false positives and all the treacherous things that you could fall into which happens so often with security tools, it's just way too many false positives little boy who cried wolf and eventually turned this thing off it's not valuable etc and that was the conversation then, I'm curious now that it's been a few years I noticed in the NPM list there is a typo squat in these recent attacks like how many typo squats have there been because I thought of it as a pretty rare thing but have you I'm sure you guys have detected and caught some over the years like could you guesstimate has it been like six has it been like 60 has it been like 6 ,000 like how many typo squat attacks have you guys found I mean I can pull the latest numbers uh right off the top here please do and just tell you so I'm going to go into our our back end here and i'm going to search typo squats that are uh let's see uh confirmed what do you think adam what number is going to pull back tens of thousands tens of thousands holy cow i'm gonna go with like 2500 all right so i just put the pagination size on 500 which is the maximum um okay and i'm getting uh i'm getting 500 so i need that what i need to do is I'm going to export this as a CSV real quick.

51:59500 pages or 500 squats? Oh, 500 squats. So I'm actually going to need to do a dump to actually get the real number because 500 is just what the UI is giving me. Gotcha. Yeah. Okay, here we go. It's 1 ,700 types of squats. 1 ,700. Dang. That's legit. And those are confirmed. So those weren't just like false positives or positive positives. Yeah, those are like human confirmed. So we have a security research team that looks at all these. And that's over the course of how many years? When did you ship that feature? Maybe three years ago, something like that? A couple years ago, yeah. So yeah, these are...

52:34So it's safe to say like 500 a year of these things happen, or found, and maybe more. That's crazy out there. It's a wild, it's a worldwide web of danger, you know? All I want is my packages so I can build my software, okay? Can you just give me my packages, please? Right, I just want to make my web app and just live in harmony, you know? Well, we decided as a community that open source, you know, was like the collaboration and the productivity that we get from working in this way was more important than security. Like that's just what we decided collectively, right? We decided that, and I'm not even arguing against that.

53:17I'm just saying what it seems like we decided. That's a trade-off we made. We made that trade-off. We said it's better that someone can run NPM publish without any vetting and just get their code up and share it with people. And that that's going to do more good for the world than the bad guys that are using the same NPM publish command. Right. Bad stuff. And we just decided that that's what we wanted to do. And honestly, that's why, I mean, one huge reason why NPM has been so successful. We have to absolutely give credit to like Isaac for coming up with, for just deciding like, we're going to democratize package publishing and we're just going to give it to the masses and making it so easy for people to publish and and uh so so frictionless um and that's why npm is the largest ecosystem by far and right part of part of reason why we've had this like flourishing of of like you know the javascript ecosystem um so so you've been living with that decision ever since and you've been fighting on the front lines of it in order to secure that freedom that isaac you know allowed and everybody agreed to i mean i think they're probably there are certainly dissidents.

54:22Not all communities have those trade-offs, like JavaScript slash web slash NPM community made that node, that trade-off and have been living with it. If you were to start fresh today, everything you know now, you know, for us, if you were like the benevolent dictator for NPM's future or something, or like the new one, would you make that trade-off again today? Or would you say, yeah, not worth it? What do you think? That's a really good question. um thank you i think i think that i think that i'd probably still make the same trade-off because there's there's so much good created when you can when you can just i don't know just when you trust people and you you hope for the best and and i don't know i'm an optimist i don't know i think yeah i just think i think that i think that like we can clean this up like we're doing like sockets like we're doing our part like we're gonna clean it up we're doing what we can um we're helping people and it's, I wouldn't want to slow down like the innovation.

55:23I wouldn't want to slow down the collaboration. I would want to just let, like, I think, I mean, you could, maybe there's some obvious stuff they could have done sooner. Like you could keep the, keep the frictionless publishing, but just have a couple of things that they, like if they had done 2FA sooner, like a lot of the attacks that happened, like if you, you know, they should have just turned on 2FA for everyone who, who, who, who, you know, has above a trivial amount of downloads from the very beginning. right um you know maybe at some point when you hit a certain level of popularity uh or you're added to a package with a certain level of popularity you might maybe you should have to like do some kind of real identification of yourself you know and like prove you know what i mean like if a new account is being added to to lodash with access to publish to lodash maybe we should know who that is like just as a community maybe we should right maybe there should be verified blue checks or something for for the you know for people that are that are on these big accounts.

56:13Like there's these kinds of things that I don't know if these are good ideas, but I'm sure that I'm sure that like, there are things you can do that wouldn't slow people down too much at the stage where they're getting involved or getting started in open source, but where you could layer it on as you get more popular. And yeah, I just have to say, I'm somewhat disappointed. I know that there are good people at GitHub and NPM trying to work on this, but overall, I just think they haven't invested nearly enough in this and for being the stewards of the most popular and most important open source ecosystem, it's quite disappointing actually just how little improvement we've seen.

56:50And that's not to say that people, I know there are people working on that stuff that are trying hard, but I just think it's at a company level, they haven't invested enough resources. It's not the fault of the individual contributors. It's like, they just don't care about it. It's an afterthought. And yeah, absolutely, this could be handled better. Yeah. That's a harsh reality. I mean, to put it plainly, GitHub is the owner of NPM. Mm-hmm. Right? Not the community, but GitHub, the corporation that's owned by Microsoft. Mm-hmm. And that's the target of all these attacks. And they could all be better worked on.

57:30It's so strange. Like, after they bought NPM, it's so strange. Like, they just seem like they never really had their heart in it like even from the moment they bought npm they didn't really like they had this remember get up packages they had this like separate thing like they weren't like all in on npm they were like oh we should people should use get up packages and it just seemed like it was never yeah it was never prioritized from the very very beginning yeah it's just it's we were seeing the implicate we're seeing the consequences of that now you kind of wonder why they bought it i mean i wonder why they bought it if they weren't gonna foster it like why would you i guess i guess that happens with things it's probably better they probably saw it as better than the alternative because npm needed to be bought like they were out of money and as far as i can tell they it was not like a good sellout it was a save us sellout um because and npm inc shouldered the brunt of cost for the npm community ecosystem of developers all around the world for many years.

58:30And so it makes sense why you eventually just can't keep doing that forever. And they just needed a sugar daddy. To that end, GitHub did publish a few things they've done or are doing. This was in what a week ago in light of all of this. And they have three things that they call roadmap for hardening package publication. The first one is local publishing with required two factor auth. Thank you. And so this is what you brought up earlier, like this could have been done much sooner. So it's kind of the horses out of the barn. Is that how do you know that? Something like that. Obviously, doing it now is better than never.

59:13And then granular tokens, they say, which have a limited lifetime of seven days, is the second thing they're doing. And then third is called trusted publishing, which I did not double-click on. So I'm not sure exactly what that means. But are you aware of these three things? Your thoughts on them? Are they better than nothing moving forward? Have you looked at their implementations or anything? Anything like that? I mean, so the granular access tokens is an improvement for sure. So what this means is that you can't generate a token anymore that has an unlimited expiration time. So the maximum you can set now is 90 days.

59:49So this means that, I mean, it affects CICD workflows, absolutely, because now you can't put those in there and have them last forever. Right, things expire, and so you can't have latent stuff back in the old days that is available. to use now yeah yeah so that's going to change a lot of people's workflows uh and then they recommend i mean it actually breaks a lot of people's workflows because now you can't have a token in there that that just you have to i mean who wants to go in and generate a new token for every package they publish every 90 days it's a pain in the butt even when let's encrypt started with their 90 day ssl you know thing before we had automated all out of that it was such a pain like i had reminders like hey gotta go back and run these six commands and thankfully all that's kind of been uh tooled around but and for security it's great but for usability it's that old trade off it's like oh gosh uh this sucks so so they they recommend moving to is trusted publishers which is which is basically a way to publish packages with some extra like cryptographic guarantees around how the package was produced and it uses like temporary credentials uh the way that it works is there's basically built-in support with a small number of CICD providers like GitHub Actions and GitLab.

1:01:07So what this means is you can literally, in order to participate in this, you have to build and publish your packages on GitHub Actions or GitLab CICD today. You can't use anything else. I think they're trying to add more support for others, but part of what this is trying to guarantee is that like you know that this package was built on a trusted machine so not like some random developer laptop that might have malware on it or something but it's built on like in a trusted environment and then that's obviously going to be a small list of like companies that they kind of you know approve and and then and then it also makes the it eliminates the long tokens by by um pulling down just a temporary token as part of that process and and like I think it's fine.

1:01:54I think it's an improvement. I just don't think this is going to solve all the problems. At the end of the day, the code that's being built, there's nothing. The signature on this code that is being produced isn't attesting to any actual facts about what the code does. There's no socket scan being run. There's no behavioral analysis being done. So someone could still get access to the GitHub repo and then just put malware in and then it'll be signed and be published through trusted publishing. And, you know, I mean, so there's still kind of a more fundamental problem, which is at the end of the day, what we're trying to do as developers here is we're trying to take code written by somebody we don't know who we don't necessarily trust.

1:02:30And we're trying to run it on our systems and hope that it doesn't do anything bad. And no level of like running it on a safe GitHub Action server, you know, and signing the code and using a temporary token is going to like is going to fix this fundamental problem of code can do whatever it wants to do. And if you take code from a random person on the internet and you try to run it, that may not end well for you. That is just fundamentally inherent in what we're trying to do every day when we use open source code. So that is not fixed by this. But I'd say it's still a step in the right direction.

1:03:03But it doesn't really solve it, despite the name and the intent and all that being positive, if that makes sense. this is what has always given me belief in what socket does and your original thesis which is look at the behaviors look at the changes you know if there's a new maintainer being added you know what are the circumstances like things that change as a result of either new inputs or outputs to the code base that to me seems like the the most logical way to do it versus which trusted server under which circumstances, not the underlying swap outs or an install script that goes rogue or just all these things that is part of your original thesis.

1:03:50To me, that seems like the right way. Why? Maybe this is speaking to a different level, but Socket is a company, so I can't imagine, oh, just buy Socket and install it into NPM and boom, you're done. But more like, why isn't there a more concerted effort to do what you've done or what you're doing across different package managers, which it's not only NPM, it's others that are exposed as well. We're only talking about NPM today because there's been such a lot of, so many activities and events that have happened. Why isn't this at the true front lines of like the way NPM works? What Socket does for the packages on NPM?

1:04:30Why isn't that fundamental to the infrastructure of NPM? That's a great question. I mean, I talked to some folks on the GitHub security team a few years ago when I was like a speaker at GitHub Universe. And they told me that they were using AI to look for malware on NPM and that they had built similar systems to Socket. And at the time when they told me that, I was like, oh, maybe this won't be a problem anymore. Maybe, you know, this is going to get solved at the registry level. And then it just never solved the problem. So I don't know whether that never rolled out. I don't know whether it did roll out, but it just wasn't good.

1:05:01Like, I just don't know what happened. But I know that there were people there that seemed very smart that I had faith in that were working on this problem. So I just don't know why it never actually solved the problem. Yeah, I don't know. I just don't know. I don't know the answer to that. It's a good question. I think part of the thing that has helped a lot in the last few years, I will say, is it's not just Socket. There's actually been a bunch of other companies that have popped up now that are doing similar things to us. I mean, I would call them copycats to some extent. Well, you've been here a while.

1:05:33You have the right to call them that. Yeah. Didn't I tell you I'm the CEO of a new, a new startup. It's called pocket. Yeah. Mine's called jock it. Sprocket rocket. Yeah. Faster than socket. It's like the better than grep, you know, better than socket.com. So, so there's, it's not just us now. There's actually others out there that find, uh, find these things too. So I think there actually is a pretty good, like kind of almost like third party scanning going on of like just a bunch. And the other thing is we're all incentivized to kind of compete with each other and try to be the first to find these things.

1:06:06So I think it's doing a pretty good job of actually cleaning things up. But the one downside is, of course, that the packages are published first and then we find it afterwards. It almost seems like we should have like a vetting period where like packages have to like bake for a while. Imagine if, like, hypothetically, every registry said, you know, for the next, you know, when you publish a package, you know, there's like a three-hour waiting period during which time, you know, like Socket could take a look at the package and using our systems and our security teams and stuff. If something really trips a filter, you know, we would have a time to kind of go in and say, don't let this one go live, you know.

1:06:44That's kind of like, obviously, there's some cost to the developer experience of having to wait three hours. But if I was in charge at NPM, I might try something like that. I might say, like, why don't we find a trusted partner that's been doing a good job of this and, like, work with them on implementing something like that, you know? I think that's a good idea. I think, could you get that down to, like, 30 minutes or, like, three hours seems like as a developer, publisher. I'm like, that's pretty lame. But 30 minutes, you know? Back to the immediacy. You don't even want to wait three hours to solve security on NPM.

1:07:22You're like, three hours is too long. I don't want to wait three hours if I can wait 30 minutes. How long does it take you guys to scan a thing? I know there's probably lots of them coming out. You can do it faster. It's so true. The initial scan is all automated, so it could be minutes for sure. It could totally be minutes. I think 30 minutes to secure the supply chain is like, we could get on board. I put my name on that petition. 30 days to secure the supply chain. And it reminds me of, what was it, like 30 days to stop the spread? Do you remember? I think it was less than that. I think they're telling us it's like 14 days to stop the spread or something.

1:07:55Oh, my gosh. Get out of here. There's no stopping that spread. And there may not be stopping this spread either. I was thinking about a slight behavior change. Would it be impossible to ask developers who are publishing packages to, one, delay that publish scenario so that you can filter it through a socket type thing prior to publishing. Is that, is that unreasonable to ask or unreasonable as a community? I know that you're a for-profit company, so this is sort of hard to sort of mandate in a way because you're, you know, filling your pockets, let's just say. I don't think that's the case, but it's a weird way to say it.

1:08:43I mean, it could be seen that way if you would be yelling, yeah, that's the way to do it, right? Yeah, that's the way to do it. Yeah, do what it takes to make Socket bigger and better, right? But the point is, is there enough societal, communal pressure on developers to delay that publish mechanism unless it's a reason of security? Like we're fixing a fix kind of thing. So we've got to get it out there. But if it's normal everyday package publishing, what if it was the way it was through a secure system that NPM does not have? And just even in NPM's case, what if there was a different target similar to the way you can just swap out one string and get up actions and use a different server to do your builds, for example?

1:09:28Like just the same kind of easy swap in the developer flow that says, okay, you're inheriting a delay and you're inheriting better security by doing this one thing. I think it's possible to opt into that type of thing. But the real challenge is just that at the end of the day, the registry is the best place to implement this type of measure. Because at some point, if an attacker gets hold of the NPM published token or they get added as a maintainer or they get access to the GitHub repo. Well, actually, let's just focus on the NPM part. If they get access to the NPM token or they get added as a trusted maintainer, then whatever opt-in process that the maintainer has taken to like scan things with a tool like Socket before publishing can just get bypassed because they have direct access to NPM.

1:10:12So the right way to do this, if we were, and I'm not even advocating for this, I just think it's an interesting idea. So nobody send me hate mail if you don't like the idea. But I think if NPM itself had like a staging area where, you know, packages sit for, let's say, 30 minutes and then anyone. But it's not like that they're basically the way I would implement it is it would sit there in sort of like a like a place where anyone could see it. So everyone is published, but it's not rolling out yet. Exactly. Exactly. It's published, but it's not rolling out. You could even explicitly install it if you really wanted to and you knew the exact version or whatever, but you just can't make it the latest and greatest version that people pull in.

1:10:51And so it sits there and bakes for 30 minutes and people can look at it and then security vendors like us could go in and try to assess it and then would have some channel in to tell NPM, hey, wait a minute, wait a minute, don't let this one through. And that way, you know, it's sort of like very transparent and it's universal. You know, you can even, you know, another, another version of this, you could do, you could have it where maybe the delay is only like five minutes or one minute. And then only if something's flagged by the automated system, then it goes to 30 minute delay. And it says, oh, you're, you had, you added something in this new version that is just.

1:11:27Yeah, it graduated to your makes sense. Yeah. You, you made, you did some really sketchy change. Like you're a new maintainer that published this or you're, you know, you added in like. A score, man. All you need is a score. Like severity score. That's what we do. Yeah. But imagine you added in a bunch of IP addresses or some weird obfuscated code. Like, oh, let's sit on that one for 30 minutes and make sure it's good. Maybe we should do a little bit of extra checking. A brand new prompt. You got a prompt in there all of a sudden. Yeah, suddenly you have a prompt. Exactly. Why is this library prompting?

1:12:03And I don't want to imply that this is the solution, that we figured it out on this podcast. you know just in an hour here we solved all the security but well i hope we did because i want credit you know that's you can line your pockets all you want i just want the credit for figuring it out remember that time when the changelog guys figured it all out we have honestly jared a few times we've figured it all out a few times i believe for sure for sure i mean i just think i just think having someone who's at the helm who's actually kind of trying to improve things and trying things in a measured way is really important and uh i'd like to see more of that from not that i want to call them out necessarily but like who is in charge from a personhood like what individual or individuals are in charge of mpm that sounds like pulling out yeah i mean can you name the person in charge i mean there we can we can search on the internet and find that answer so it's not calling them out and doxing them but like who is who has the ability to make this change because i mean one let's play with them let's let's talk with them we used to that person but now he doesn't work at github anymore i think it's i think it's just i think i think it's just github leadership has to prioritize npm and to and make it a priority i don't think it's i think the people working on it are all good and doing their very best i think it's just it's just not a priority as a company right people working on it they haven't put enough uh like resources into it and github leadership is now microsoft leadership so that's even more vague than it previously was because thomas domke is out and nobody's replacing them so I know there's people there.

1:13:32I just don't know any of those people. And maybe we can dig down deep and see if we can find the answer to that question, Adam. But we don't know who that is. We may never find out. We may never find out. Well, the point was not to attack that person. The point was not to attack those people or to see them going wrong. Yeah, just simply like, who is in charge? Because I'd love to open some... They're obviously aware of it, or at least they should be. let's just have a conversation with how to best manage this. Cause obviously we keep talking about it. And, you know, when you have this level of attacks, they're sophisticated and they copy each other and they were successful.

1:14:13I mean, marginally given, you know, the amount of type of squats that were out there and marginally given how much was actually in the wallets of those who were able to, to get the crypt delicious say, but there was successful. And this is an upset. We're having this podcast for this very reason. And so, you know, how do we get NPM to be more secure? Is the question, period. And so the other question that follows is, may I speak to the manager? You know, like that's what Karen's asked when they can't get the answer out of someone else. They're like, who's in charge here? That's Adam's question.

1:14:50Who's got the keys to the kingdom? Yeah. May I please speak to the manager? Yeah, please. All right. There's an open invite for the ChangeLog podcast out there, GitHub folks. We have plenty of friends at GitHub. We can probably see what we can do about that. Good people, everyone we meet. And hopefully we can get to the bottom and help out, help secure this supply chain ecosystem. For us, here's a wild alternative. Earlier, you said something like, we're living with the cost of trying to run other people's code. that we don't necessarily trust, right, in our own systems, paraphrasing. What if we just didn't do that anymore?

1:15:36So hear me out. You can live in a bad neighborhood and you can buy a gun and put up a fence and do all that kind of stuff, or you can move to an entirely different neighborhood. And so what if we're now reaching the age of language model code generation? What if we just said, why are we installing other people's code when we can just generate everything we need? Is that a feasible alternative lifestyle today or maybe tomorrow where it's like, you know what, I don't need NPM because if I need a left pad function, I'm going to have my LLM generate one for me anyways. I think it's a great question.

1:16:13And it's actually, I think that this is the world we're going to end up in if we don't get the security under control. Because at some point, if we have more attacks at the scale that we saw in the last two months with literally the most popular packages on NPM. Yeah, it's ramping up, right? Right. Yeah. Yeah. If it becomes rampant, like if that becomes the new normal, then I think you actually do see serious consideration, serious conversations happening in companies where they'll start to think, well, they'll do a serious assessment of exactly the question you're posing, Jared. Like people say, why don't we just generate this?

1:16:52Like, especially for the more trivial packages, like why bring in a dependency? I mean, people can add that into the AI prompt files and telling the AIs not to bring in third party dependencies and just instead write everything from scratch. Now, that obviously has its own set of problems. Like you won't get improvements. It might be buggy in its own way. It might not be as robust. You might, you know, have to go back and maintain that code. Now it's really, really truly your problem in a way that it isn't when it's in a dependency. So there's a lot of downsides. But yeah, I mean, that would be a very different world we live in where now suddenly I'm not installing Next.js.

1:17:36I'm like generating my own Next.js. You know what I mean? I'm not installing React. I'm creating my own React, right? So I think if this were to start happening, it would happen with, it wouldn't happen with these big frameworks, it would happen with the packages around the edges. Right. Probably still use, you know, we're not rewriting Linux from scratch anytime soon. We're not rewriting, you know, Node.js anytime soon. But maybe that's where we end up in like a long enough timeline if all open source becomes completely untrusted. But yeah, I think that's a bad world. I actually think that that would be really, really bad because we built this awesome thing with open source.

1:18:20We've proven it's like a good way to, it's a good way to innovate. It's a good way to collaborate. It's, it's a, it's an incredible thing for innovation. And then to now have people doubting it and going backwards, going back to proprietary software. Like we already fought this war and open source one, right? We shouldn't, we shouldn't be going backwards into, into, into proprietary software. Yeah. And certainly an isolated world versus what we have now, which is like a communal world where we're sharing and helping each other. And I'm just generating all my own code for my company over here. And you're probably generating the same code for your company over there.

1:18:56There's lack, there's economies of non-scale, right? There's so much repetition there. There's so much isolation, solving the same problems in different ways. There's so many ways where it's not optimal. So I tend to agree with you. But I certainly also think that maybe JavaScript's culture of small packages makes it a low-hanging fruit for those kind of moves because there are so many single purpose libraries, small purpose libraries, things that really might not have been smart to be a dependency ever in the first place. But because of amazing hackers like Sindresor, who's coding up everything that we need, you know, and each package is like a single line function.

1:19:37That whole culture, I think, while it backfired in the short term in terms of security problems and just a mountain of node modules, maybe it allows web developers to think twice about their small functions, their utilities, and what they actually need to have as a dependency and opt out of those while still using the backbone libraries like React and their frameworks and the big stuff where it's just smarter if we all collaborate and work together on those things. Yeah, I talked to an engineering leader at a company that you all know, and he told me that they're starting to vendor open source dependencies into their main repo instead of pointing to NPM.

1:20:25And they're starting to do it with dependencies that just haven't had updates in a long time that are like these sort of like fixture type dependencies that just never change. and he actually named a few of mine. He called me out and he said, well, you have run parallel and run series, which are like not ever, I haven't been updated in five years. They're not really going to change. And so they just started inlining those. And the team at first, he said they reacted like, wait, what are we doing? Like you can do that? Like they didn't even conceive of the idea that like you could own the code instead of installing it from NPM.

1:20:57So he's trying to bring this kind of like culture of like we don't need a dependency for everything. And so there's kind of like this change happening. This is a company that never conceived of the idea that you might not want to NPM install something, but rather like write it yourself or own it yourself. And now there's like a real conversation happening because of the last two months where they're actually deciding to shift to, you know, owning that stuff. Even if it's just inlining it and just bringing in the NPM code and keeping the license on it, but putting it in their own repo just to make the risk go away.

1:21:26Like they're doing that now. That's actually happening at companies now. So it's pretty crazy. That's a lot like the staging scenario, but just reversed that you suggested, right? It's putting the code, the third party code in a place where you can sort of examine it more closely or control it more closely. it's the reverse mechanism of that where you're saying earlier to do it as a pre i guess a pre-published scenario where this is more of a post-publish like hey you haven't changed in a while why keep pulling the dynamic version of you from the registry let me just cash or keep it on cold storage or whatever this unchanged package and one you can run some security diagnostics against it and then two you just know it hasn't changed so you have that luxury of just knowing that.

1:22:12So that's interesting. Is there anything that's like, is there any sort of like, I guess, fake registry that does that at scale in terms of like, we can have our own fake registry that is consumes or keeps a hold of our blessed packages. And that's our true push pull place as let's say developers, not so much publishers to packages. Yeah. There, there are some products out there from companies that do give you like a private registry that you can host internally. There's also an open source project called Verdashio that can let you do that as well. And it's basically a mirror of NPM. I don't know how easy it is to use it like the way you're talking about to sort of set policies about what can be brought into your mirror, like what meets the criteria for your mirror or not.

1:23:05But it does let you save copies of all packages for the future. So if like NPM goes down one day, you know, you can still do your builds and, and you can, you can kind of, you could treat that as potentially treat that as like your set of blessed packages. But actually the, the way I'd, and I might use this as an opportunity to shout out something that we're, we're shipping today, actually on Tuesday. So when, when people hear this, yeah, if people hear this on Friday, it'll, it'll be out for a few days, but we're shipping something kind of like what you just asked Adam, which is a. We call it socket firewall.

1:23:41And it's basically a command that you can prepend to all your NPM installs. It's called SFW, socket firewall. So NPM install SFW. And then you just run SFW NPM install. And then whatever NPM is about to do, all the packages that are being fetched are routed through a firewall that's on your local system. It's a local server that we spin up temporarily, and then it points to that as the registry. And then that local server goes out to NPM and gets the packages. But before it brings anything in, it makes sure that they're not malicious, make sure that there's no backdoors, there's no typo squats.

1:24:24And then it lets them through only if they meet that policy. So that basically you can just put that before all your NPM install commands, and it'll make sure that you're safe. And it works for Yarn. It works for PNPM. It works for cargo and it works for all the Python package managers too. Nice. And so on and so forth. Yeah. Is it pretty easy to go from like registry or I guess like cargo to NPM, for example, like how do you as a developer, you know, just pick it up and start using it? Let's say for cargo, for example. Yeah. You would just run the same command. So it'd be, you know, SFW cargo fetch.

1:25:04You just put SFW in front of any of the package manager commands, and it just automatically puts it through the firewall. So it's really easy to use from a developer experience perspective. And you can even alias it in your terminal. You could go in and make NPM, just run SFW NPM. What about policy development myself? If I wanted to run something like that, am I, this is early days, so you can tell me how you're developing it, but is that where you tell me the policies and I get what you give me? Or is that you give me some and I can add some later to say, as an example, you know, this firewall could do what we just said earlier, which is if the package hasn't changed in the last year, let's, let's just go ahead and automate icing that.

1:25:49Let's just keep that version of it because it hasn't changed in a year or at least ponied up into a list. So you have a human loop that says, OK, here's a list of known packages. Our socket firewall has consumed over the last whatever. Ten of these haven't changed in a year. Let's evaluate them from a human level. Should these be on ice or not kind of thing? Can I as the developer begin to orchestrate policy on this firewall? Yeah. So there's some stuff you can definitely do there. It doesn't do all the things you just talked through there in the first version, but we're going to be expanding it so that eventually you can set a really complex policy and say, I only want packages older than that have baked for seven days or et cetera, et cetera.

1:26:33But today, the free version that we just announced, all it does is block malicious dependencies and all the attacks that we've been talking about for this whole show. So that's what it's focused on. And then over time, we're going to evolve it into something closer to what you're talking about. we're going to have like support for more ecosystems. We're going to have like a telemetry on like what developers are installing. So if you're like a big company and you want to know not just that none of our software has malicious components in it, but actually that no one even even like, do you want to, because people want to confirm that no one installed it even on their developer machine, right?

1:27:11So the only way to do that is you have a firewall and you can see every package that came into every developer system. so we're going to be able to have some monitoring for that and be able to log all that so that people could find out like oh yeah we didn't fortunately ship any malware in our published package in our published software but we did have like three people who installed it locally and so now we might need to go and clean up their machines their local laptops you know so there's things like that that we can help with Could this be your next big thing? This firewall? I think so I hope so yeah It sounds like a big deal You just like dropped it at the end here it sounds like a big deal Yeah, maybe I should have dropped it sooner.

1:27:49I didn't market it right at the beginning. But no, I do think it's going to be a big deal. I think it's a really lightweight tool. It's free. There's no API key, no config required. It's really easy to use. You just npm install sfw-g and then you now have this sfw command. And we just want everybody to use it. We think it should be part of boilerplates. It should be part of everyone's de facto stack. and you'll get like the most valuable part of Socket, which is the malware protection for free. And we're just giving it away to everybody. It's like, we think it's, we should, everybody should have it.

1:28:21So that's what we're trying to do. And then we hope that people like use it and think, oh, this is cool. I want this for, you know, my enterprise Java. And then they'll contact us and we'll be able to sell them a more enterprise version that can work for Java and these other types of things. And if they want a more customizable like policy on what they allow and what they don't allow, then they can also contact us. So we're hoping we give away the most valuable part for free. And then we will get, you know, some enterprises will find the paid version of the school as well. That's what we're hoping.

1:28:49In case you said it already, refresh. What are the criteria for free for this new firewall? Yeah, so there's no rate limiting or anything like that. It's unlimited usage. There's no API key required. So people can just go hog wild with it and use it however they want. It is limited to only four ecosystems such as JS and TS, Python and Rust. So those are the ecosystems that you can use it with for free. If you want to use it for any of the other ecosystems we support like Java or Ruby or Go or yeah, for those right now, we're keeping those as part of the paid version, enterprise version. So we'll see about what we do with that.

1:29:32Maybe down the line, we may consider moving more to the free version, but today we're trying to just help with these three ecosystems. And, and yeah, and there's no configuration of the security policy. So it's just malware. It's just blocking malware. It's just blocking these attacks we've been talking about. If you want more customization around any of the other stuff we can block, like if you want to warn people about deprecated packages, or you want to block certain licenses or these kinds of things, then you got to get in touch with us for the paid version but um we think that that's probably only interesting to larger companies anyway i think you'd be surprised honestly uh i'd personally want to swap out like well if you're gonna give me four let me choose the four versus take the four and miss the one that i really want uh just candidly behind the scenes i'm just tinkering with some go and some some rust stuff and so i'd be a sad go developer in this case and a happy rust developer in the other case not a lot of JSTS stuff I'd swap that one if I could and just say give me go and cargo essentially and I'll be a happy camper not pressuring you to do that but that'd be kind of cool feedback taken yeah I think it's always easy to add more to the free version of things it's hard to take away so I think we wanted to start with a set that we felt was like I'm just glad you're doing a period man you know I think it's awesome to do that I think it does sound pretty interesting I'm really curious about how it works as a product a lot of thoughts swirling let's just say but yeah please give it a shot and let me know what you think of it SFW what a great name dude love it I assume that's a double acronym for safer work as well yeah we were joking we should register NSFW and it only lets you install malware or a redirect SFW says what are you crazy oh lord that's awesome well cool for us thanks for the deep dive on npm and all the things yeah i can always count you for a deep technical but also a really good take on how we should operate as a community what we should expect from the actors that are playing their roles in our community to act like npm as an example i think there's a lot of catching up there to do no harm against the folks that are actually doing the work but it is a serious place uh it is a serious thing for the community, check your responsibility, I guess.

1:31:58Check it. Let's go. Yeah. Thanks for having me, guys. It's always fun. Anything left unsaid? No. I mean, yeah. Please give Socket Fireball a try. Let me know if you guys have feedback. And anyone else who gives it a try, my DMs are open. You can contact me on X or Blue Sky or Mastodon. Everywhere. Guys everywhere. Email me. Awesome. Thanks, Ross. Until next time. Thanks, guys. Stay safe out there. Stay safe for work. That's right. SFW. Peace. Peace. All right. That's your changelog for this week. Thanks for hanging with us. Did you know we are playing with the idea of adding a classifieds section to the news?

1:32:43We'd max it out at five listings per week, and they'd appear both in the newsletter and in the audio. It'd be super brief, headlines only, and link to a URL of your choice if you'd like to put your startup, your passion project, your big idea, your event, your whatever in front of change logs, discerning, handsome audience of hackers. Fill out the form that's linked in your show notes and in your chapter data. Thanks for listening and thanks to our partners for sponsoring Fly.io and Depot.dev. We appreciate you. Next week on the pod, news on Monday, Evan Yu on Wednesday and Jose Valim on Friday.

1:33:17Have yourself a great weekend. Keep your heart with all diligence for out of its spring the issues of life and i'll talk to you again real soon

1:33:46finally the end of change logging friends with adam and sharing some of the rando. We love that you loved it and stayed until the end. But now it's over, it's time to go. We know your problem should be coding and your deadline is pretty foreboding. Your ticket backlog is an actual problem, so why don't you go inside? No more listening to Change lock and fence, a badminton chair, but it's still a calm valley. No one gave the gag, we'll come to an end. But honestly, that will probably be our finale.

1:34:35You'd best be slinging ones and zeros, and that makes you one of our heroes. Your list of to-dos is waiting for you So why don't you go inside No more listening to ChangeLogging Friends With Adam and Jerry and people you know ChangeLogging Friends, time to get back into the flow ChangeLogging Friends, ChangeLogging Friends It's your favorite ever show Favorite ever show

1:35:16Game on.