pnpm with Zoltan Kochan

0:00Traditional package management systems for JavaScript have faced several inefficiencies related to dependency storage, resolution, and project performance. PNPM is a fast, disk-efficient package manager for JavaScript and TypeScript projects, serving as an alternative to NPM and Yarn. Due to its efficiency and reliability, PNPM is increasingly popular for managing monorepos and large-scale applications. Zoltan Kokan is a full-stack web developer and the creator of PNPM. He joins the show with Josh Goldberg to talk about his background and package management in the web ecosystem. This episode is hosted by Josh Goldberg, an independent full-time open-source developer.

0:44Josh works on projects in the TypeScript ecosystem, most notably TypeScript ES Slint, the tooling that enables ES Slint and Prettier to run on TypeScript code. Josh is also the author of the O 'Reilly Learning TypeScript book, a Microsoft MVP for developer technologies, and a live code streamer on Twitch. Find Josh on Blue Sky, Mastodon, Twitter, Twitch, YouTube, and.com as Joshua K. Goldberg.

1:24Zoltan, welcome to Software Engineering Daily. Hi. Hi. I'm happy to be here. As a consistent and now long-time user of PMPM, I'm excited to talk to you. We've got a lot of great stuff to talk about, like architectures and modern repo management. Before we get into all that, though, how did you get into coding? I started coding at school, like, at 2004. I didn't have a computer back then, or maybe it was even a little bit earlier. So I kind of had the whole journey from the invention of internet to the invention of AI. The first couple of years, I was coding without internet. I got internet only in 2008.

2:09So I was using just old school books and pirate content, I guess, in Ukraine. Actually, early on, I was planning to be a painter. I was doing a lot of drawing. I didn't want to join the university or institute it was for painters, but I missed the entry days. I mixed up the dates. So accidentally, I failed that gym, and later on I entered the math faculty to become an engineer. That's, I guess, most of it. I started with Pascal at school, and then I used C-sharp, and I used C-sharp at my first job. So you're saying there's an alternate universe where instead of creating PNPM, you became an artist, a painter?

3:04Yeah, but I don't think I would be very successful at that. I was always painting real stuff, you know, not some imaginative abstract stuff, which I think is more exciting for a painter. So you got your first job in C-sharp. What was that? So the first job was at an outsource company. It's called Svan. And they have a little office in my city, Ushgorod, which is in Ukraine. And they had different projects for American companies. I started as a junior programmer there. And I was only doing like unit tests, which is interesting because people didn't use like test geo and development people were just coding and i was writing the tests for them which now seems weird but that's how it worked and i was so happy when after a few months i started doing real tasks that are not related to tests so at that company i worked almost two years And then I joined JustAnswer, which is a Q &A website where verified experts answer questions.

4:20And there I worked as a full stack developer on a.NET stack. And after four years, maybe at JustAnswer, I started really to look into Node.js and frontend. And that's when I found PNPM. I have to ask, as someone who was in.NET before JavaScript, how did you find that transition going from the strongly typed, strong foundation of.NET and C Sharp to the, at the time, even more so wild and wacky and loose world of Node.js? So there are pros and cons. What I really loved about JavaScript, that's before I used TypeScript, is that it's so faster to prototype and to get the working little CLI or tool.

5:08like for scripting it's so much easier to use JavaScript because you don't need to deal with all that boilerplate with C Sharp like even to write the hello world CLI app with C Sharp you need to declare a class declare a function and stuff like that maybe it changed since then I don't know because now there is more than 10 years have passed since then and I know that C Sharp has improved a lot. Now you can even use C Sharp on all platforms, stuff like that. But that was my experience back then. And also you don't need all that IDE, which is actually really expensive to buy a Visual Studio license.

5:54You can just use it as an editor for JavaScript. But maybe with C Sharp, it's also possible, but it's not so common. Yeah, these days you can use VS Code for C Sharp, but it is not quite the primary path. Yeah, it is kind of amazing to think of the trajectory that the.NET area and ethos has gone through. TypeScript recently announced that they are rewriting in Go, which at that time, 10 years ago, would have been completely unimaginable. Microsoft using a Google-oriented or sponsored language for TypeScript development for anything other than.NET. It's an amazing change but okay moving on node.js so node.js typically ships with npm which is their package manager of the node package manager as it occasionally is called why would you want to create something like pnpm a different package manager than the default so how it happened actually at just answer we had a huge monorepo it's about okay it's not a huge monorepo by today's standard, but at that time it was like 140 components.

7:05And the way it worked, there was a simple script, which in a for loop ran npm install in each directory. And maybe they were linked together through Simlinks or not, I don't remember. And we had like a Sinopia, which is the old name of Verdaccio, running in San Francisco. And we were using it from Ukraine. And this installation was terribly slow. In San Francisco, it was faster, maybe 10, 15 minutes. I don't remember, but in Ukraine, it was 30 minutes just to run and install in this monorepo. And also it used a lot of disk space. So like dozens of gigabytes because of all these duplications in NodeModules folders like 100 times.

7:54I'm not sure how I found PNPM, but I found it and I tried it instead of NPM in this monorepo and it immediately improved speed drastically. So back then NPM was a lot slower. Not everyone remembers it because it was like in 2015, 16. It was NPM 3. It was before Yarn came out. and even on small like projects npm was like it was slower than the current npm probably 10 times like the current npm is super fast in comparison but pnpm was really revolutionary at that time and i decided to look into it and at that time pnpm was not really maintained it was created by Rico Stagruz. I think he's from South Asia.

8:54Now he lives in Australia, I think. And it was like a fork or like a clone, I don't know, of another package manager IED created by Alexander Google. And the reason these package managers were created was because in NPM3, the structure of node modules has changed. If you remember, I don't know. In NPM2, node modules used like a nested structure. So every dependency had its own dependencies in a subdirectory inside the node modules in the subdirectory. So you could have really deep tree structure in a node modules. And on Windows, this was causing constant issues because on Windows, there's a limit to the file path.

9:41And also the other problem with this structure was that dependencies were duplicated many times inside NodeModules. So they wanted to fix these issues. And in NPM3, they changed the NodeModules layout to a flat layout, where now dependencies are hoisted to the root. And as an alternative to this approach, this IED package manager was developed by Alexander Google. And the idea was to use Simlinks to avoid both nesting real directories and to avoid hoisting everything to the root. So that's the main reason why PNPM was created. You used a few terms there. There's an old joke among developers, a famous cartoon on the internet about how in increasing order of size, there's the planets, the stars, a black hole, and then node modules.

10:35So it's nice to see so much work has gone into reducing that size and complexity. What exactly does hoisting mean in this context? And how does that change or not change how Node.js resolves packages? So by hoisting, we mean that every dependency in the dependency graph is moved to the top of your node modules directory. So even if you only have like Webpack in your dependencies, Webpack itself has like hundreds maybe of dependencies of its own. And if you install Webpack with npm and you open your node modules directory, you'll see hundreds of directories there. Even though in your package.asone, you see only one dependency, which is Webpack.

11:21This is a simple explanation of hoisting. If in the dependency graph, there are multiple versions of a package, then this hoisting algorithm will pick one of these versions and put it to the root of node modules. And the other ones will be nested in one of the dependencies sub node modules directory. Okay. And then you also mentioned Simplinks. Now, as a user of PNPM, what we experience is that you can install pretty darn quickly. And oftentimes, if it's the same installation you've done on a previous run on your device, you can just do it offline or even on an airplane. But how does simlinking play into all of this?

12:01And what even is simlinking? I'm not sure the simlinks have a role in the speed. It just allows us to use an alternative layout for structuring NodeModules. Actually, you also asked about Node.js resolution in the previous question, but I did not answer, so I can mention it now. So the way Node.js resolves dependencies is that it searches for Node.js modules directories in the current directory of the JS file that has the required statement, and then also looks in every parent directory as well. and when it finds this dependency, it resolves it. And an interesting, important point here is that for this resolution, it uses the real location of the module, not the simlink location.

12:59So it means that we can have a simlink at one place, but we can have all the dependencies at a different place where the sim link points. So with PNPM, if you run PNPM install with Webpack, for instance, you'll see only Webpack in the root of the NodeModules directory, and it will be actually a sim link. And you'll also see a hidden directory,.pnpm. And if you open up the.pnpm directory, you'll see lots of directories there, like in the form of a package name, add, and version. So this is a flat directory structure, even flatter than the npm one with the hoisted node modules because it has the version also of the package.

13:53So the webpack in the root of node modules points to a directory in this.pnpm directory. And if you open any directory inside this.pnpm, you'll see another node module there, and you'll see all the direct dependencies of each dependency. So if you open webpack in.pnpm, you'll see sim links to the direct dependencies of webpack. And because when you require webpack from your project, Node.js will resolve the sim link to that hidden directory inside.pnpm, and it will actually look for dependencies of webpack in that isolated directory inside.pnpm. That's fascinating. Yeah, it's really, I think, a great example of doing less work to achieve just a really clean solution to a common problem.

14:46And as a result, to users, it just feels seamless. Like you run pnpm install and it feels the same as an npm installation, just faster. But under the hood, it is kind of shockingly different. Yeah, the speed difference is actually achieved by something else. It was also a part of the initial PNPM version when I found it. So if you open the PNPM website and go to, I think, the motivation page, you'll see there a nice illustration of what makes PNPM so performant. So instead of having separate installation stages, where in the first stage all the dependencies are resolved, then a separate stage when all dependencies are fetched, And the third stage, when all dependencies are written to the node modules directory, in PNPM, there are separate independent pipelines for each package.

15:40So one of the package will be resolved, fetched, and written to node modules at the same time when others are still being resolved and independently at different stages of the installation. actually even yarn has this different installation stages but still it's fast enough i think your team is generating more code than ever but you're still stuck with rigid legacy tools inflexible workflows manual updates and siloed communication are slowing you down just as your engineers juggle more pull requests and context switches than ever that's why there's monday.com's dev platform with fully customizable workflows you can ship faster no admin bottlenecks no clunky add-ons let your developers work in monday dev or right from their ide with ai-powered integrations that keep every task in context get full visibility into progress performance and risk all in real time fully synced with github and your entire ecosystem and with business connectivity built in monday dev keeps engineering priorities aligned with the impact that matters most no more admin bottlenecks.

16:49Visit monday.com slash dev to learn more. Before we move on to the higher level features like workspaces, it would be great to get your opinions on the current state of package managers. As you noted, Node and NPM themselves have changed a lot. And also Yaren has had quite a few versions. Do you see any particular pros, cons, feature comparisons between the current versions of those three package managers? So I believe currently, if you check, PNPM is most popular for using in monorepos. I don't know actually how good are the other package managers at monorepos. I really stopped comparing at some point, but people say that it currently has the best feature set for Monorepos.

17:38You can see also on our website on the workspace page, as at the end of the page, we have like usage examples, I think. And there are some big names there, like GitHub projects with thousands of stars like Nuxt or Vite, some well-known projects use PNPM for their Monorepos. There are some features that are available only in one of the three package managers. Like Yarn has the plug and play feature, which we kind of have also, but it's not really well maintained. So we use Yarn's libraries for that. So if you need plug and play, I would recommend using Yarn. BNPM has some features that others don't have.

18:27like recently we have added configurational dependencies which is basically a new type of dependencies that run before everything else and you can load settings from there or use it for plugins it's really new so i think it will be very powerful but it needs more work to be done but i think yarn also has some plugin system i don't know how it works but i think they have like core plugins which are installed by default and they have maybe some community plugins in pnpm we have hooks like in pnpm file cjs you can hook into different parts of pnpm it's always lovely talking to open source maintainers very rarely does anyone who's not an open source maintainer in one answer mention things like we use one of our direct equivalent packages or projects as a dependency, such as you do with Yarn.

19:24Or for this particular feature, although we have it, you should go use our other equivalent project for certain scenarios. It's really lovely to see that kind of cooperation in the ecosystem. Yeah, well, I've worked on PNPM for like nine years, maybe. So many things have changed since then. But Yarn's core contributor was Myel. he has also worked on yarn for six years maybe or more so we have talked a lot and we had you know when there is some hate in the community you can empathize with that like i wouldn't talk about yarn i know how much work goes there yeah there is an unfortunate amount of community vitriol but you're all working towards a better future and it's really lovely to see And speaking of that feature, let's talk about some of the features of PNPM that you mentioned.

20:20So just to take what you said about hooks or the configurational dependencies as an example, why would I want, let's say, a configurational dependency if I'm managing my team's package or projects? Yeah, I think I was a little bit inspired with Bit for configurational dependencies. I worked for Bit for like three years, I think, or more, three, four years. If you haven't heard about Bit, it's like a GitHub for component-based development. It has its own version control system, package manager, hosting, CI, everything to make component-based development easier. What makes Bit interesting is that you don't have to choose between a monorepo and a single-package repo because you have dynamic workspaces where you can fetch only those components which you need for a particular feature.

21:21So it solves the scaling issue of monorepo but has all the pros of the monorepo. And one of the ways that Bit makes it possible is that it has this environment for components and the environment has all the necessary configuration, like not only the dev dependencies, but also any rules for the linter, for the bundler. Think about it like all the.files that you commit to your repo. And then you have... So this is one of the disadvantages of a multi-repo approach that you need somehow to synchronize all these.files. But with Bit, you have your environment component, and this way you can share all your configs by using the environment.

22:16I wanted to make it easier with PNPM also, so to be able to publish a package that will contain all your settings. For instance, maybe you use some patches to fix dependencies. If you haven't heard about it, you can use patch files with PNPM. It's also supported in Yarn and in NPM, you can use a package for that patch packages, I think. So you can, if you have some fix for maybe an unmaintained dependency in your node modules, and it's very deep in your dependency graph, so you can't just fork it. I mean, actually you can fork it and use an override, but with the patch it's even easier because you don't need to publish your fork to the registry.

23:09So you generate a patch file and you commit it to your repository and you tell PNPM to use this patch file for the dependency. But if you have many repositories and each repository uses this dependency, you'd have to copy the patch to each repository and make the configuration change in each repository. With configurational dependencies, you can add this patch file to the configurational dependency and then just use the configurational dependency in each of your repositories and it makes it easier. Also, you can, for instance, have some standard overrides in configuration of dependencies. Or you can have the list of package names that you want to hoist to the root of NodeMoyles, because sometimes you need that with PNPM.

24:05We also have a feature which other package managers don't have currently, which is called catalogs. It's a way to centralize, to synchronize versions of dependencies in a monorepo. So you have these catalogs declared in the pnpm workspace YAML file at the root of the monorepo. And then inside your projects, you can just put catalog column in the version field instead of the version. This way, it's easier to control each project in the workspace. We'll use the same versions, which is otherwise possible only using some LinkedIn. I think Yarn has constraints, it's called, which allows to create LinkedIn rules for this.

24:55And it's a third-party package for this, I should think, called LinkSync. But with catalogs, it's a bit nicer because it's centralized in one place. And it's nicer to see on the change in one file, not in many packages on files throughout the repository. For the future, that we could move these catalogs to these configurational dependencies. And maybe companies could have like catalogs of recommended versions throughout their repositories, stuff like that. Or even like an allow list of dependencies. Maybe I know that some companies have limits to what packages they can use for their codebase. So it would be easier to ship this.

25:48Yeah, there's one developer behavior that's well understood in the web ecosystem. It's that people prefer and enjoy configuration files and avoid and hate linting. So it's nice to see that being moved to a first class supported feature. A bit back brought up your company and we should talk a little bit about it because it's fascinating and also very relevant to this area of repo and component management. So you work at BitDev and BitDev does use things like PNPM. But why would, let's say I have a monorepo with a design system and a bunch of apps and some front ends. Why would I want to use something like BitDev or its Harmony platform to be able to manage all those components and front ends and so on?

26:29From my point of view, the main advantage is actually this possibility to create your own workspace. So your company won't have to decide whether to use multi-package repositories or to use single package repositories or have multiple monorepositories for different things and then somehow link them together with bits. Every component is independent. It's shipped. Actually, you can think about it as a single package repository. So each component has its own version control system. But unlike with a Git-hosted approach, you don't have to... Like Bit creates a much better user experience. Even like in the UI, the website, BitCloud.

27:21So there is BitCloud and BitDev. bit dev is i think mostly about the open source aspects of bit because you can actually use bit for free and host it on servers because bit is a an open source project but then it has bit cloud which has additional proprietary code and paid features but like with bit like the code you use, like when you make changes in multiple components in the PR, you will see all the changes to all the components in one place. So that's actually not possible currently with GitHub. Maybe some other companies allow it, I don't know. And it's very well-optimized currently for UI components too.

28:10So it's like it like has the how it's this project called for UI libraries, I don't remember, where you can visually see the components and even like click on them. Storybook. Yeah. So it's kind of like a mixture of Storybook and GitHub, I think, visually. So the problem with Monorepos is that you can't scale it. No matter what you come up with, at some point it will break. You can scale it to that amount of components that you will need in today's world. I know there are like Turbo and NX, which improve things a lot for Monorepos, but still at some point it can't handle that amount. So I think it is a really unique approach because nobody else tries to solve it this way.

29:03Nobody wants to create a new version control system and a new GitHub. So yeah, just check it out, I guess. Yeah, creating a new Git does sound rather daunting. There have been some things built on top of Git parentheses as a service that have sprung up, but it's nice to see the innovation here. Have you tried building a text-to-SQL chatbot? If your AI agents don't understand your data, its definitions, queries, and lineage, they're forced to guess. And bad guesses mean risky assumptions. That's where SelectStar comes in. SelectStar automatically builds an always up-to-date knowledge graph of your data, capturing metadata like lineage, usage, and example queries.

29:48So whether you're training an AI model or deploying an agent, your AI can answer with facts, not assumptions. Stop the wrong SQL queries before they happen. Learn more at SelectStar.com.

30:05Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge and is simplifying car shopping. Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade-in value. Advanced, intuitive, and deployed, that's how they stack. That's technology at Capital One. We don't have too much time left for questions, but real quick towards the end, I do want to ask you, you've talked about monorepos versus multi repos or single you've talked about different forms of package management why and how newer innovations in pnpm and yarn have happened for the next say five years are there any more upcoming trends or new areas you're excited about being innovated in well the ceo of entropic said that developers will not be needed in one year so i don't know things that are actually going to happen i should clarify to be honest the problem with a popular project is that you have so much stuff to do that you can't really make a long term like you can't think about big stuff we had many actually like you know ban came out right there is also origin which is created by uh cat marchan I hope I pronounced her name correctly, but she was a core contributor to NPM CLI.

31:31There is also Vault, which is created by ex-NPM maintainers and founders. So sometimes we look at these competitors and think about what could we do differently. So, for instance, we spent a lot of time on thinking about rewriting PNPM in Rust or some other language because a ban came out and it claimed that it's 10 times faster than PNPM. So to be honest, for a long time, I thought that maybe the long-term plan would be to rewrite PNPM in Rust. But then we created a proof of concept, and it turned out that the Rust rewrite didn't make it faster. Maybe we just failed, I don't know. But it looked like most of the time is spent on network requests which are not influenced by the text that we use.

32:28So I think now my big idea for the next few years would be to create some registry integration because we have, I think, optimized almost everything we could on the CLI part. But if we would have our own registry, for instance, I don't know if we would do it or we would use some existing registry. But if we could, for instance, move resolution to the registry side, it could make installation faster. Or use some different archiving algorithms for the tarballs that are faster. or make some partial, I don't know. Or maybe we could fetch not the whole packages, but just the files that are required from code.

33:23So I think it would be interesting to look at the registry side. Could be also a way to get some additional funds if we would have our registry. But the competition is really big currently because I know there is JSR from Dino and Vault. is coming out and something else. I don't know. Maybe BAM will have its own registry. I don't know. They are very wide in their plans. It would be unsurprising. We have at Software Engineering Daily, an episode from a few months back into 2024, an interview with Luca from Dino that touches on how, yes, once you push a lot of these concerns into the registry, as you said, you can do a lot of optimizations.

34:07For them, a big focus is TypeScript supports and documentation generation. That'd be fascinating. Sultan, I have one last question for you before we wrap up. What is your take on spicy food? And where have you had your favorite pieces of hot food? Thanks. Yeah, I really like spicy food. I'm Hungarian. And in Hungary, there is a famous soup goulash, which should be consumed spicy. And there are some sauces, Hungarian sauce called eryšpisto. but then I discovered Asian food which is even more spicy and Mexican food and I really really love Indian food I have many friends in India from work, from my previous work for one week I visited Bengaluru and there it was heaven heaven, food heaven for me yeah a lot of folks from North America, from the West might not know that there are quite a lot of fantastic, very flavorful Eastern European foods, such as goulash.