Blocking Software Supply Chain Attacks with Feross Aboukhadijeh

0:00Modern software relies heavily on open source dependencies, often pulling in thousands of packages maintained by developers all over the world. This accelerates innovation, but also creates serious supply chain risks as attackers increasingly compromise popular libraries to spread malware at scale. Firas Abu Khadija is the founder and CEO of Socket, which is a security platform designed to protect software projects from open-source supply chain attacks. In this episode, he joins Josh Goldberg to talk about his career in open source, open source supply chain attacks, practical security lessons, the expanding attack surface in software development, and more.

0:42This episode is hosted by Josh Goldberg, an independent full-time open source developer. Josh works on projects in the TypeScript ecosystem, most notably TypeScript ESLint, a powerful static analysis tool set for JavaScript and TypeScript. He is also the author of the O 'Reilly Learning TypeScript book, a Microsoft MVP for developer technologies, and a co-founder of SquiggleConf, a conference for excellent web developer tooling. Find Josh on Blue Sky, Fostadon, and.com as Joshua K. Goldberg.

1:27for us abukadijay welcome to software engineering daily thanks josh glad to be here we're excited to have you you have been in and around open source and general security practices for quite a while before we dive into you and socket can you tell us how did you get into coding yeah i got into coding when I was in high school, I wanted to build a website to collect my favorite flash animations. So I was kind of born in the era of Newgrounds and E-bombs World and Albino Black Sheep. And just all these kind of, I don't know if folks remember these or if they're too young, I don't know the audience of this show.

1:58But yeah, I always thought those things were fun. And I wanted to kind of collect them all and put them onto one website. So I did a lot of downloading of those SWF files from other people's sites and then rehosting them on my own page. And I had to learn PHP to do that and MySQL. And so that was kind of my first foray. And then you went into Stanford for computer science after that? I mean, yes, I did go to Stanford to study CS. My high school didn't have a CS class. So I kind of was just self-taught with PHP up until that point. But learning CS at Stanford was amazing. A lot of the other majors at Stanford, they don't necessarily emphasize teaching well, but that's one thing that the computer science department really stands out in.

2:36They have just like a ton of support, like other undergraduates actually are your TAs and like help teach you. And so I learned a ton. I remember my first class I took there, it was using C++. And I remember like my first reaction was, how does the computer know that these words are variables if they don't have dollar signs in front of them? Because in PHP, every variable has a dollar sign. And so my mind was blown. Like I almost spent too much time in PHP in high school. And like, it took like a little bit of unlearning to kind of like realize, oh wow CS and programming is this like really broad thing it's not just PHP and MySQL.

3:10Yeah do you ever go back and look at the PHP you wrote in high school? I have actually done that before and it's hilarious I didn't even know about functions even after like years of writing it I literally just pasted everything multiple times if I needed to do it in more than one place so it's it's horrible but but it worked it kind of didn't matter in some sense. Like I built a site that got, I think at its peak, it had like 600 ,000 annual visitors watching all those flash videos. And I don't know, it's kind of funny how I think the lesson from that is that you should just jump in and do stuff.

3:45And I mean, I obviously would never write code like that now, but like if you let doing things the right way, getting your way, it can kind of take the fun out of it and like stop you from just kind of, I don't know, catching the bug. And I caught the bug. What really got me excited and kept me going. And it was this idea that I can put code online. And while I'm sleeping, it's like working for me. It's like serving visitors, people are coming to the site, like having a great time. And I'm literally asleep. It's like that scene in Fantasia with the brooms and the buckets. And like, I don't know, it's like automation.

4:16I don't know. It's just sort of this cool idea of like, this thing is just out there. And I think at one point, I put Google AdSense on the site. And so like, I was like, oh, I'm sleeping. And it's like, making me money. You know, it's such a cool idea. Yeah, I don't know. That got me really into stuff in high school and college. Believe it or not, it's not the having excellent clean code that gets a lot of newcomers in. It's the results and being able to have that superpower. Yeah. Yeah. That said, like, I also think it's good to use functions. I wouldn't recommend anyone copy what I did in those early days.

4:45Good to know. So that was your first popular website. What was your second popular or hit project or website that got released? So while I was still in high school, I did another site that was around, like sharing my notes for my AP classes that I was taking with other students. So you could go there and like, you know, access outlines of different chapters in the book. It was kind of like a Sparknotes for AP courses. And that was great because a lot of high schoolers didn't want to read their textbook. And so that was something that got quite popular. That's also when I started learning about SEO and just like all the web search type stuff.

5:18And like, how does Google work? How do you like properly semantically build web pages that are going to be indexed by Google? that Google is going to love indexing and sending visitors to you. So that was a cool way to learn. And then like after that, when I went to college, there was one other site that I did that was maybe notable, which was called YouTube Instant. And this was kind of the result of a bet that I did with a college roommate of mine. So Google had just announced this thing called Google Instant, which was a way to kind of, it's kind of like autocomplete on steroids. So as you type like letters into the search box, instead of just showing you like, here are five possible searches that you might want to do, it would actually take you to the search results page of the first suggestion before you've even hit enter.

6:00So like you type A and then it's like, takes you to this actual search page with like the 10 links for Apple or whatever it thinks you were searching for. And so I saw that and I was like, oh, that's cool. I bet I could do that for YouTube and it would be even more cool because now you're typing letters in and you're getting like flashes of like different videos coming in and like it's almost like channel surfing like YouTube. And so I bet my friend I could build it in an hour, which was kind of completely ridiculous. But I just thought like, oh, let me just, I don't know why I did that, but I did.

6:27And then he was like, yeah, I'll take you up on that. And then I actually did build it in three hours. It was very simple. It was just using like the YouTube API and like putting a video embed into the page. No fancy web frameworks, no futzing with like getting TypeScript and React and all that stuff set up. It was just literally like one JS file and one HTML file. So very, very simple, pretty horrible code as well. but got that up and then I put a tweet out and a Facebook post out and went to sleep and I woke up the next morning and it was completely viral it had gone like across the internet overnight while I was sleeping and I don't really to this day really know why I think if I had to guess it was because it was I mean it was pretty fun like once you used it people wanted to share it but like it was also because it kind of like piggybacked on Google instant and they had spent two years engineering Google to support that level of new load on, you know, just in terms of number of search pages being loaded as literally everyone typing letters, it's like 10x or more of the search volume that they had ever had in the past.

7:27And so they like hyped up how much like engineering effort they put into this. And then like, lo and behold, some college sophomore comes along and is like, I built the same thing for YouTube in three hours. And then that somehow like, unintentionally like became this like media story. And then it was like fueled even further because the YouTube CEO tweeted to me and said like, hey, this is great work. Do you want a job at YouTube? And then like that caused a whole nother burst of attention. People were talking about how like the resume is dead and like building projects and putting them out there is the future of getting hired.

7:57And like all this thought pieces came out of it. And I was just like kind of swept up in this media storm, basically. So it was kind of a fun experience to go through as a sophomore and like kind of trying to not like say something stupid while getting interviewed by NBC or whatever, you know? So it was pretty cool. That sounds like a very thrilling ride. Are there particular lessons you took away from it that you still use now as a CEO on the other side of the hiring table? Huh, yeah. I mean, I think the big thing is like just launch, like just ship things. Don't think too much about it, like build stuff in public.

8:26I mean, if I had sat around and like, I don't know, tried to use the right framework or tried to, and it's funny, I say this, like I'm actually good at this, but the reality is I'm more saying this as advice to myself to remember because I also tend to like over-engineer things and like want to do things the right way. and like sometimes don't ship. And like what I think worked in this case was I just like put it out there. It didn't really matter that it was like pretty bad code. I like put it out there and then I improved it after I saw that people actually cared. And so that's the right way to do things.

8:52If you're interested in like finding something that works or that resonates with people, it's just, it's really about like the number of things you put out there, not like sitting around in a room by yourself, trying to like think of the perfect idea. And that's a lesson like I've had to learn again over and over in my career, including like literally when I started my first company and we kind of did the opposite of that where we literally just sat around and like built some cool technology for about eight or nine months without really putting it in front of any users and then at the end learning actually nobody wants this we just spent nine months like building it's a cool science project I learned a lot I'm glad I worked on it but like nobody wants to buy this or use this so I could have probably learned that really quickly if I just talked to people or had put something more minimal out there you are not the first ceo to mention this on this podcast yeah that's a really important learning yeah there was even like a guy on hacker news like when youtube incident was like on the home page it was actually on the home page a couple of times like multiple times at the same time and then one of the comments i remember this was this guy who i don't know who he was but he was like really upset that my site was on the home page because he said he's like i've been building the same site for three months and it has way more features and it supports like filtering and sorting the videos by views and by different things.

10:11And he kind of was explaining all the things that could do that mine couldn't, but like, then someone replied to him and was like, you should have just shipped. Like, and I saw that and I was like, you know, evidence of this lesson literally in that thread. And I mean, I obviously felt kind of bad for him. He put in a ton of time and like, but it sort of proves the point of he should have just shipped and no one really cares about all the advanced options he was adding, and he should have just put it out there sooner. Yeah, it's a pity. You've shipped quite a few things. We could take this either through your career or through your open source projects in general.

10:41Let's talk about the open source projects first, because they're quite visible. Is this a continuation of the same mindset of you've just shipped a bunch of NPM packages? How does that play out? Yeah, I mean, so I got into open source originally, because after that company I mentioned, it was called PureCDN that I worked on for about eight or nine months. It was, we basically were going to shut it down. And then we got really lucky and kind of out of nowhere, Yahoo offered to buy the company. And really, they wanted us to just join the team and like work on JavaScript and video players and stuff like that at Yahoo.

11:11So we took the offer because it was me and my two friends. And we were like, well, yeah, this sounds like a good way to wrap this up and make something of this, this like unfortunate situation that no one actually wants what we built. Maybe we can go to Yahoo and they'll use it. But they ended up not using the code really at all. Like they had bigger problems than reducing bandwidth costs of video. And so we ended up just kind of helping with more fundamental things like making their video player load quickly and work on mobile and stuff like that. But I did feel kind of sad that after all that work, you know, that science project I was talking about, like all this cool technology and stuff we had put into it, like didn't get to see the light of day, and would never get to see the light of day, because now they owned it.

11:47And it was all proprietary code. And so after I left, I worked there for about a year. And I'm proud of what I did there. But it was also one of those things where, you know, it was a really big company and it was like hard to ship things. I felt like I was fighting the company to like ship things, which is crazy. At one point, my manager said to me, you need to do less. Just like, stop trying to do all this stuff. Like you're making my life hard. Thinking like, this is not why we required literally, the CEO of Yahoo, Marissa Meyer at the time told us and she was on a startup acquisition spree during this time.

12:18She told us the reason we're buying you is because we want you to inject startup energy and like just this attitude of getting things done and like shipping stuff into the company. And like, this is why we're bringing you on. But we were just too few when we were on a much bigger team with the old school folks that were there before. And so like we did some good stuff and there were some really smart people on the team, but ultimately like, I think the organization couldn't really handle it. But like the old blood one rather than the new blood, if that makes sense. So I left and then I wanted to play with some of the same ideas of like what the original company was doing, which was about building a peer-to-peer CDN.

12:52So think like a content delivery network like Cloudflare or Akamai, but powered by end user devices. Like your laptop is in a network, kind of like BitTorrent. And if you're watching a video, you're going to serve that video to other folks that are coming to the same site. And it's sort of like a big peer-to-peer network. And that could speed up sites in some cases, and it could also massively reduce the cost to host sites, which before Cloudflare was really popular, CDNs were actually quite expensive before they started giving everybody bandwidth for free. And so this was where the idea came from.

13:23And so after I left Yahoo, I just decided, oh, it'd be cool to like build that again, but in an open source way so that no one can ever like take it away from me or from the community. I want this to be open. I want anyone to be able to use this. And so that was like why I got into open source. I mean, I had always thought of open source maintainers as this sort of crazy, awesome level of like programmer that like I one day wanted to aspire to be. You know, when I was learning the code, I was almost afraid to look at open source packages because it was like all this crazy advanced code that how could you understand what it's doing?

13:54So I kind of always thought it'd be cool to do that and like give back in that way. And then I had this opportunity to do it with WebTorrent. And so that's kind of what came out of all this was I ended up building a torrent app that could work in your web browser and do peer-to-peer, like from my browser tab to your browser tab without any software installed on the computer. And so you could really get like BitTorrent experience like on a website. Which is amusing now because BitTorrent has BitTorrent web. Yeah. And actually they use WebTorrent under the hood. The protocol that we developed, like all the major torrent clients actually support WebTorrent now, finally, after like 10 years of trying to get adoption of this standard.

14:30So it's pretty cool to see finally that it's actually like the vision has actually kind of happened where any torrent client running on a desktop can talk to a web browser user across that bridge, which is pretty crazy. I mean, when you think about it, that that's actually possible with WebRTC. As an open source maintainer, how do you feel when one of the projects you lead or maintain or have worked on becomes successful in this way? That's a fun one. So at first, it's like awesome, right? I mean, you get to talk about it, people are interested in it, you get invited to speak at NodeCon for JSConf or whatever.

15:04And I remember at that time in my life, I was my early 20s. And it was super fun getting to go travel the world. Like I went to so many countries I've never been to before, met people. And it was, it was funny, too, because people, they would use me and a few other early node folks as the mad science track of the conference. So they'd have all these talks on like react and kind of practical things like, you know, how do you deploy microservice and this and that and the other thing. And then they wanted to have like, certain talks that were like, you're never going to use this at work, but it's just cool.

15:33And like, that's what they would invite me to be the entertainment, I guess, in that sense. And so I always I think that's why I was invited to those. And so it was just fun to like be able to show people what I was building. And it was also very in public because honestly, for the first six months of talking about WebTorrent, it didn't work. It was literally like in development. And I was talking about it and explaining how WebRTC worked and like how this was all going to work and like doing some cool demos of WebRTC for people during the talks, some live demos and interactive things with people all like interacting on a whiteboard from their phones and on my screen and like over peer-to-peer peer network.

16:07And it was really cool, but it was, it wasn't web torrent. And so I was definitely doing it in the open and it was really fun. But then when you get to a certain point in your project, like what ends up happening, I've seen this happen so many times, not just with me, but with like every, basically almost every maintainer, but not all, but, but almost every maintainer that I know is that they eventually just get burned out of doing it because at least for me, what happened was I felt this incredible responsibility to like fix every issue that was reported on GitHub. And at first the volume was really low.

16:34It was like someone would open an issue and you'd be like, Oh my God, this is so cool. I have a user using this. And you're like, so happy. And then eventually you get to a place where you wake up every morning and it's like 40 issues opened and you're like, Oh my God, this is not sustainable. And then they were all issues too. Like they weren't even, it was like, I kind of forgot why I got into it. And almost by the end, cause I was like, someone's reporting about, Oh, it doesn't work on Arch Linux version, whatever. And I'm like, okay, I don't even care. I should have just said to them, like, you submit a PR if you care about this and I'll review it and merge it.

17:04But instead I took it Like, oh, it's my job to go and fix it. And so I very much burned myself out by the end and like basically needed to walk away and be like, add other folks as maintainers and just kind of like go do something else for a while. There's a great blog post that you've, I think alluded to in the past from Nolan Lawson, what it feels like to be an open source maintainer, talking about how it's like you have this giant stretch of people in front of you and there's absolutely no way you can address all of them. And it feels awful on both ends. Yeah, it definitely feels like that.

17:33You wanna do the right thing and help them and you feel this obligation because it's like your code and they found a bug and they're right. There is a bug and like you're correct. And so therefore I must fix it. But like, actually, there's not a contract saying that you have to do that. It's like I put this code online as a gift to the world. I didn't promise it would never have a defect. And so it's actually OK. It's actually enough to give a one time gift. You don't have to give a permanent SaaS subscription of your time as a gift to the whole world. So I think that was the realization that I like finally got to.

18:03but after getting super burned out. Is your AI model taking weeks to train or is it too slow for real-time inference? Fixstar's AI Booster is the acceleration platform that solves both. AI Booster automatically analyzes and optimizes your entire AI pipeline. The result? Dramatically faster training, up to five times faster, and compute costs slashed by up to 80%. Trusted by major companies, including Sony Honda Mobility, Stop waiting on your hardware. Visit FixedStars.com to learn how. Let's talk about that dark side of open source because it transitions really well into what you've been doing recently.

18:44Specifically, let's talk about security. What is Socket.dev and what's the open source problem that it's trying to address? So open source, when it works, is amazing, right? You can just go out and grab any code from the Internet that's written by somebody you don't know. You don't know who they are. You probably didn't even read the code. and you can just pull it in and run it on your computer and it saves you a ton of time, right? It's this amazing buffet of unlimited, all you can eat code that you can just pull from. But there are bad guys, there are criminals that have realized that actually getting access to an open source project and putting malicious code into the package is a great way to attack a bunch of people at once at a really large scale.

19:25And so we've seen a lot of attacks of this nature over the last several years for all kinds of motives. Sometimes it's politically motivated, it's countries attacking other countries or trying to attack US companies or things like that. Other times, it's just basic cryptocurrency criminals that want to just steal people's crypto. There's even been cases where people have like, they just, they were a good maintainer at one point in time, and then they just something just snaps in them and they want to just they want to watch the world burn. And so they'll put malicious code into their own package that has been trusted for super long.

19:57And so it made me start wondering, like, why is there no way to stop this? Why is there no way to catch this? Especially after like the fifth or sixth time I saw this happen. You know, I mean, the first time I saw this happen is in 2017, with EventStream getting compromised. This was a package where a new maintainer was added, who basically said, Hey, can I help maintain this package? I noticed it hasn't gotten any updates in a while. And they got access. And then after about 30 days of doing some good changes, they put in a malicious change that stole a bunch of cryptocurrency out of a specific application that was targeted.

20:29So they obfuscated the code, and then it only triggered in one specific electron app. So everyone else who had the code, it just sort of did a no-op, but in this one app, it would trigger and steal all the users' money. So this kept happening. And then I looked around and I was like, why is no one doing anything about this? Is this just like the way the world works? I was also confused. Why is nobody reading the code that they're using? And then I realized I'm really weird. And like, I read every one of my dependencies before I use it. I mean like maybe not anymore but like when I was working on web torrent certainly it was a small enough surface area that I could actually do that and then I realized like no one else really does this like it's a pretty rare thing like some people do but it's pretty rare and so that's why like there's really not much of an immune system against this usually what happens is like the attacker makes a mistake that ends up showing in a log file or something and then the community just accidentally discovers it and that's what kept happening it was always like oh we accidentally noticed that like this package has been stealing everyone's information for the last two weeks.

21:25And that's when I was like, really worried, because I'm like, how many more of these do we not know about if like, we're just accidentally finding these. So in the event stream example, let's give you an example of how we found it as a community. It turns out the attacker, they obfuscated their code, and they were using the crypto module in Node.js to decrypt the attack code. But they didn't know that like in literally a day or two after they backdoor the package, that node decided to deprecate the function that they used. And so everybody who was using the package started getting this deprecation warning.

21:59And then they traced it back to this chunk of code that was added into this library. And so like, that's such a random accident. If that hadn't happened, it might've gone for weeks or months without being discovered. And this pattern just repeats in every one of these attacks. And so that's kind of like when I was like, dang, this is a problem. And like, no one seems to know how to solve it. So what are the techniques that we can use as application developers, separate from, say, Socket, just to kind of harden ourselves against these before we go into specific services? Yeah, yeah, for sure. I mean, the number one most important thing that everyone should be doing, and everyone is probably already doing, but if not, you should use a lock file.

22:34So your package manager, probably, depending on the language, but almost all of them support a lock file of some kind. And this is very important because it locks down the specific versions of every package in your dependency tree so that when a new person on your team clones the code down and runs the install command to install those packages, they're going to get the exact same set of dependencies that you had when you built the application. And so there's no randomness or there's no new versions being brought in unintentionally. It's very explicit. There's also a record of what versions of every package were in use in the Git history.

23:10So everyone knows what was in use at every point in time in the application. That's the number one thing you can do that obviously doesn't stop you from installing a bad package, but at least you're not just pulling in whatever happens to be the latest version at the time you run the install command. And I want to emphasize too, it's important that you're using a lock file and not just pinning your direct dependencies to a specific version. If you pin your direct dependencies, that's a good start, but that doesn't lock down the transitives, because those can also have loose ranges. And then the package manager will go ahead and you'll install your direct dependency at that exact version.