FreeBSD with John Baldwin

24:51And we would rather use something that was off the shelf as our starting point. And when they're evaluating alternatives, one of the things that also happened around the time of the development of the PS4 was the FSF released version 3 of the GPL license, which included some clauses related to granting of patent rights and so forth, that for various, not all, but some companies, that gives their lawyers a bit more heartburn than previous versions of the GPL did, for example. And I believe that was true for the case for Sony and that when they were evaluating what platform to use, one of the big reasons they chose FreeBSD was FreeBSD was BSD license and that they did not have to worry about dealing with GPLv3 and what possible implications that might have if they were to use GPLv3 software and the PlayStation OS.

25:36That's my understanding of how they ended up landing on FreeBSD for the PS4. We have gotten some contributions from Sony on the PS4 that came out of the PlayStation effort. Some of it is direct to us. For example, some support for things like AVX and our kernel, I think, came from Sony originally. But they've also contributed in other ways that have really benefited the project. Another thing that was happening kind of in the, I guess that's about the right time frame, mid-2010s or 2000s, was the rise of LVM as another alternative open source compiler suite. And LVM was very attractive to Sony, and they spent a lot of work contributing to the linker side of the story, which at the time wasn't nearly as well developed as Clang as a C compiler.

26:15So they contributed a lot of effort into the LLD linker. And one of the ways FreeBSD has benefited from that is that now we use Clang and LLD as our default toolchain for all our platforms. So I think with FreeBSD 15, for example, I believe we have one binary left in the base system as DPL, which is like diff3 or something. But the rest of our system now is fully BSD licensed. So that's been one big benefit that we've gotten from effort that Sony has done. Nice. So then Netflix is obviously quite a big story for FreeBSD. And I think for you personally as well, the work that you've done. So let's get into that.

26:50Where does FreeBSD turn up in Netflix's infrastructure? And then what kind of happened there? Sure. So to be clear, I've done some contract work from Netflix, but I'm not an employee of Netflix. Yes, yes. I should have clarified that. disclaimer of there may be bits that I don't know. I'll say it that way. But Netflix uses FreeBSD in their CDN. So if you're watching a movie from Netflix and the bits are being streamed to your device, probably it is coming from a FreeBSD box that's at your ISP. And typically they're trying to avoid sending lots of bits across the actual internet and only sending the bits locally within your kind of local WAN for your ISP or so forth.

27:27And they do that with a distributed set of boxes to build a CDN. And the boxes in the CDN are running FreeBSD. And that's where they're playing with pushing a lot of bits out of boxes, hundreds of gigabits of TLS encrypted traffic to all sorts of customers. And so that workload is a very high performance workload in terms of the raw throughput. The individual connections, my understanding is they're not aiming for like 100 gigabits on a single connection. Instead, you need thousands of clients connected to a box. many of them at the end of very slow links with maybe terrible latency and packet loss and drop and so being efficient about pushing out as many bits as you can while tolerating a very mixed quality of the links that you have that's kind of what they are focused on and working on the things that changes they make in particular to free bsd around a lot of work in the tcp stack and dealing with tls offload and things like that yeah so there was some collaboration with chelcio I believe, and that's who you were contracting for?

28:26So one of the things that Netflix was very interested in early on was the ability to encrypt all the traffic that was going by. And they had a couple of different reasons for doing this that I probably need to get into. But this presented a bit of a technical problem. Traditionally, when you were just sending web traffic, because that's effectively what serving movies is, it's just web requests, like fetching a couple of megabytes at a time of these backend movie files. Traditionally, OSs, both Linux and FreeBSD, had an optimization for a web server that if you're sending a chunk of a file over a socket, you can make a single system call called send file.

28:59And the kernel would take care of running the state machinery between as interrupts are coming in from the network device and interrupts are coming in from the storage device. And when a block shows up, you can send it straight out the socket to the neck and never involve user land at all. And the kernel can do this all asynchronously and have driven. It's very efficient. Well, the minute that you throw TLS into the equation, all that goes out the window because somebody has to encrypt the traffic. And the old way of doing this was you did all the encryption in user land. So now you have to go get bits from the disk using a blocking system call, wait for the bits to come from the disk into user land, copy them out to user land, do the encryption, copy that data back into the kernel into a separate buffer to be sent out on the NIC, on the socket.

29:40And unlike send file, where if you're not using TLS and you're sending the same file to like 20 different clients, you only need one copy of that file's data in memory. Inside the kernel, we can share the pages that hold that data among multiple open network connections. That's really easy to do. You just share the same page and send it down multiple network connections. With TLS, it's not the same data because every different connection has its own session keys. You have to encrypt the data differently for every session. And so no longer can you share the data. You use much more memory. you start using more bandwidth on the PCI bus and so forth, and it kind of all cascades down into horribleness.

30:16So one of the solutions that Netflix pursued to help address this was to move the TLS processing out of user spaces into the kernel. It allows you to get back to using SIN file and have the kernel manage all the workflow. And initially, the way they did this was to, you still end up with multiple copies of the data, but you're able to do the bulk encryption, like the AES, GCM, what would most commonly be used nowadays, to actually encrypt the data and send it out on each connection. And they had an internal version of that. One of my first projects that I worked on with Netflix was helping to clean it up a bit so that it could be upstreamed into FreeBSD because they didn't consider this to be their secret sauce.

30:51Their secret sauce is making movies. Their secret sauce is not TLS encryption. So they were happy to have that be something that we could put into stock FreeBSD so that other people could use it and also contribute to it. Then there's a follow-on project to that. Another of my clients is a company called Chelseao that makes some smart NICs that have more than just kind of Ethernet inside their NIC that can do various things like TCP offload and whatnot. And they have the ability to do the actual encryption of individual TLS packets on the NIC. So then I extended the framework that had come initially from Netflix to allow for some network connections.

31:25We may not need to do the encryption and software in the kernel, but we can send the raw, unencrypted data all the way down to the NIC and it can encrypt it on the wire on the way out. And one of the advantages of that approach is you no longer need separate copies of the data. You're back completely to the original SIM file version where you have one copy of the data for the raw file on disk of what the movie is. And that's the only copy you need inside memory of the host. And only the NIC is actually dealing with encrypting the data on the fly. So that was kind of an interesting project because I got to work with, it ended up being effectively a joint project of what does Netflix need as a customer?

31:59What can help Chelsea kind of sell their NICs? And what's the design that works for both of those to allow us to glue things together? Yeah, as you call out, it was a sort of collaboration where Chelsea, they had to open up their hardware specs effectively to make this possible. Netflix kind of funded the whole thing. Is that sort of, I mean, just in broad brush terms, how it worked? So I'm a contractor for Chelsea, so I can access their docs under the NDA that I have as a contractor. So the Chelsea specific bits I wrote as on the Chelsea side, but I collaborated with Netflix on the design of how this would plug in and how the framework would work.

32:36Ah, okay. Yeah. No, it's very interesting. I think, again, for our listeners, sort of understanding how these things come together is often helpful as well. Equally, I work on projects that have these kind of same dynamics where sort of it's three parties and party A doesn't necessarily directly talk to party C, but because you've got B in the middle, the whole thing can work basically. So, yeah, it makes a lot of sense. And then maybe just touching briefly on macOS again, like what was maybe the history there for why FreeBSD ended up in macOS? So I think my understanding is after macOS 9, I think is what it would be, Apple started working on macOS 10.

33:12They borrowed a bunch from, I believe, Next. That's kind of where the mocky bits came from, because there's bits of macOS that are very mock derived. But there's also a fair bit that is BSD derived. In particular, I believe the initial network stack in macOS 10.0. This largely came from FreeBSD or at least NetBSD, but BSD bits in general. And a lot of their LibC, their equivalent of GLibC in userspace is largely derived from FreeBSD. And in particular, one of the things that also happened is several of the folks that I originally worked with in my first job out of college ended up going to Apple and kind of taking part of the FreeBSD culture with them, I guess you might say, or at least like the mindshare a little bit.

33:50And working on whether it was userspace bits or kernel-y bits and all sorts of Apple products that came down the line. So there's a lot of kind of friendship between the projects, in part because there's a lot of friends between the different communities. Yeah. OK, makes sense. So let's move beyond where FreeBSD has sort of ended up and actually a little bit to maybe a key part of the journey of FreeBSD itself. I guess it's what we call SMP, so Symmetric Multiprocessing. I think this was something that sort of had to evolve over a few years even. But yeah, could you maybe just talk us through like what was the problem and why did this have to be solved and how was it solved?

34:35The problem is physics at its root. But I mean, SMP is the problem that at some point CPUs, we can no longer scale them individually to get performance. We had to scale horizontally instead of vertically. So we start having to add multiple cores into systems. And dealing with multithreading is a complex problem in any environment, much less an OS kernel. FreeBSD first started supporting S &P systems like dual Pentium systems or dual Pentium Pro back in that kind of era, very late 90s. A lot of parallelism. And that was kind of the point when I got this first job working on FreeBSD stuff. that I was supposed to, fresh out of college, work on like a bootloader for Itanium.

35:19I think some changes to the installer to have a better user interface in the installer was one of the things I was supposed to work on. But one of the things that happened is right around this time of 2000, the company I was at was called Wanna Creek CD-ROM and it merged with the company who had the 1-800-ITS-UNIX telephone number to form a newer company called BSD-I who had this commercial BSD operating system and to help bootstrap FreeBSD's effort at having a more mature support for multiprocessors, they kind of gave us a code dump and said, hey, you can borrow stuff from us, like an initial version of kind of their way of doing locking and so forth.

35:55And so we had some developers in the community who are working on that. And I started helping out on IRC during the nights and during the days at my new job. And somehow a couple of months in, I'm actually doing a lot of this work, even though I'm fresh out of school and wet behind the ears and probably shouldn't be doing this work and having to learn a lot of things about atomics and so forth, which actually the Itanium manuals were very useful for me because I learned about things like acquire and release semantics of memory that's now like how you talk about concurrency in C and so forth. But I started working on this project, which internally in FreeBSD we called it SMP Next Generation.

36:31So SMPNG is what we called it. And it's a long-running project. Initially, FreeBSD, when they were just trying to get this to work on plain x86, like the dual pinions, they did the most naive thing you can do, which is you have one giant spin lock around the entire kernel. And anytime a user process would go into the kernel, it would have to wait for this one giant spin lock. There's a wonderful book I read when I've got out of college, or maybe it was my senior year or so called Unix Systems for Modern Architectures, which modern meant late 90s. And it talked about the way that systems like System 5, Release 4, and a few other systems, how they had done SMP.

37:09And the model that FreeBSD had is the one that I believe the book describes as about the worst possible thing you could do in terms of it doesn't scale at all, once you start having more numbers, of course. So the effort around SMP and G was how do we not have one giant spin lock? And how do we have a system where multiple things can happen, not just in user space, but in the kernel concurrently on different CPUs. And FreeBSD chose to go with a design that modeled more the way that Solaris and perhaps Irix kind of and other commercial Unix did it, where we did things like we created dedicated threads in the kernel where interrupt handlers would run instead of running interrupt handlers on your kind of borrowed context.

37:48Linux still actually does the kind of borrowed context thing. I believe Windows does too as well, in effect. Although a DPC in Windows is kind of like kicking things off to a thread, sort of, that we do in FreeBSD. But we had this, that was one of the first big changes was how we move our interpellers into threads. That was actually the first thing that was kind of helping to stabilize and get landed into the tree. And then from there, I started working on things like, well, we've got this code dump from these nice generous folks, but it's kind of very x86 specific. And we had some kind of early patches for an alpha port that we had to try to bring up SMP on alpha.

38:21And I kind of atom smashed those things together and tried to clean up the SMP code to be more portable and not very X86 specific, which meant refactoring the mutex code we've inherited and defining extensions to our atomic operations to deal with memory barriers and various things to kind of get us to the point where we're a little bit cleaner and kind of can expand over time. In terms of like the project itself though, multithreaded a kernel and like, you might think of it just like performance scaling across multiple cores as a never-ending project. Like you're constantly just finding new things.

38:53Like Netflix continues to push the boundary in certain areas of the system. We have mostly gotten rid of kind of this legacy giant lock that we had as our holdover. And I think it's now effectively around the keyboard driver and a few other things that just kind of aren't worth fixing almost at this point. So it's mostly done, even though like the variable still exists. But even so that we have a multithreaded kernel, like performance scaling continues to be a thing, right? We continue to grow sideways. When we were first doing S &P and G, we were worried about scaling well on four core or maybe like six core, eight core systems.

39:26And now we have to scale on 512 core systems. So it's kind of a never ending problem because physics, like we just don't get to have 40 gigahertz processors. So oh well. I mean, just briefly going back to kind of how you described the project being set up overall, is that still almost like a certain set of the contributors that work on like that scaling side? Or is that again, just like a distributed problem across all of the pieces. Much more distributed problem. You know, developers scratching individual itches, right? For example, they're very worried about network scalability or virtual memory system scalability.

40:03So when they are looking at scaling problems and things like SendFile, that's the areas they look at. They're not necessarily trying to make sure that some little timer device driver can scale across cores for some reason. That's not the problem they're trying to solve. So they focus on the part that they're trying to solve. And that's still how things in general work is people are working on what's relevant to them. Yeah. So as FreeBSD has evolved, modernized storage, I believe, has sort of been an interesting area that things have had to develop, you know, as I think a lot of our listeners will understand just this evolution of how storage operates going from HDDs to SSDs.

40:41And then now we've got NVMEs. We can probably do a whole episode on that in the future. I think it'd be interesting just to understand how FreeBSD has had to adapt to this evolution. And then we're also going to just look at how a major release came out, I believe, just the end of last year, version 15. It'd be very interesting just to hear what leads up to that. And then again, perhaps like what is 15 as well. So that's a lot of questions. Let's go back to storage. Like how has FreeBSD and storage had to evolve? Well, I think in general, storage has evolved in the industry. if you look at old spinning Rust and the command sets we had with things like ATA or SCSI, and SCSI in particular has very complex state machines.

41:21And if you go look at the spec for, I say the spec, there's actually several specs. If you look at the specs for SCSI, it's very, very complicated. And if you go now to look at the spec for NVMe, NVMe is very simple. And part of what's happened is as we've moved away from spinning Rust, we moved to flash memory. It turns out that a lot of the complexity you needed to deal with controller timeouts or seek latency and so forth, those are no longer relevant in a flash storage world. And so NVMe arose a bit out of, well, can we make something that is a lot simpler and cuts away a lot of the complexity?

41:56And that complexity, though, gets mirrored in the software stack too, right? So in FreeBSD, we have a most unified kind of storage framework, something that I think was a standard developed outside of FreeBSD, but FreeBSD adopted called CAM. like a common access method, I think is what it stands for, which is how we deal with SCSI and ATA and how we deal with NVMe. But if you look at the bits, like the state machine handling for NVMe inside of CAM is much simpler than the state machine handling for SCSI and ATA and so forth. We've been able to make that model work fine, but certainly moving forward, life gets a little less complicated.

42:30Like you don't have to worry about disk scheduling in the same way. But now with NVMe or flash storage, you do have to think about things like trim, which you didn't have to think about before in the same way on spinning rust. But you don't have to think about like the elevator algorithm and trying to sort and minimize seek head, the amount of time you're moving the head around and what the seek time of moving your heads is. That's gone. So seek latency is basically gone. But you do have to kind of think about when do I trim? How much do I trim? That's kind of my big bottleneck of how much IO can I schedule in between trims?

43:00And so it's a different kind of problem, I guess, to think about. And then there's also network attached storage, things like iSCSI and NVMe over fabrics, which is you want to present the same command set, but over a network connection. And there, you don't like definitely seek latency is not quite the same thing at all, because it's all this kind of virtualized notion of a storage device. And that all kind of hooks up in the same way, all hooks up into our storage layer in the same way. The APIs had to modernize as well alongside storage, not necessarily because of storage, but in terms of the modernization?

43:34Well, I guess one way to look at it is that as a long-running source project, gosh, I'm 30 years old. It's crazy to think I've been around it for 25 of those 30 years. Over time, you accrete a bunch of technical debt and you have APIs that looked good in the past. And then as you've worked with them for several years, suddenly they're not quite as useful maybe or they're a little crufty. And so one of the things that I also do, this is more my volunteer time, is I will try to find some things that are kind of crufty and clean them up. It is a balance. You don't want to do churn for churn's sake because that doesn't do anybody any good.

44:11And people who are carrying downstream forks like Netflix, you don't want to cause them undue suffering by making them merge a bunch of conflicting changes. But in particular, in our device driver frameworks, I had found a couple of things over the year that annoyed me where due to some legacy reasons, like in every device driver, we required you to declare a global variable that was used in a macro that no one ever used. And so I did some work over the past couple of years to transition us to where you could use the macro both with and without the variable. And then once all the tree had been converted, then I kind of dropped the compatibility shims.

44:45I've done another one this year. The compatibility shims are in FreeBSD 15, but I haven't finished the transition. The finish will be in 16, where when you're dealing with IO resources in a device driver, things like marine mapped IO or an IO port for like a PCI bar, but also other things like PCI bus numbers and so forth that a driver might need to kind of allocate and hold on to or map and then use it to read and write registers. The way our API worked is there was various functions you had to call both to allocate the resource when you wanted to use it, maybe to map it and then to release it when you were done.

45:17You had to pass several redundant arguments kind of to every step along the path. And so even though like in a true object-oriented design, like the very first thing you call to allocate a resource, it gives you an opaque object back. And the opaque object like knows things about itself. And it should know all the parameters you pass to it. So one of the changes I made was to make it like learn the one missing parameter that it didn't already know about itself. So that for all the following calls, like releasing a resource or mapping ones into like a CPU address space, you just pass the resource now instead of having to say, is it memory versus IO?

45:52And what identifier did I use? which PCI bar was it, so that it just makes that a little bit simpler and less for the programmer to think about. And those seem like worthy cleanups, and they're ones where using either evil or magic, depending on your perspective, and the CEP processor, you can allow both forms to work for a while so that you can, across multiple versions, you can allow there to be compatibility so that device driver developers don't have to worry about it. If they adopt a new one, they can still use it fine when they merge changes to older branches. And that's kind of the strategy I've taken in that case.

46:22And those are nice cleanups to do. It's healthy, I think, in code bases to clean up things that get crusty after a while. Because if you avoid dealing with technical debt, it just gets bigger. And at some point it becomes a bigger mountain to have to shovel through. Yeah, absolutely. I really like that framing because at the end of the day, this is helping developers use it. And yeah, I think often other projects, usually just because of time of the maintainers, it's just almost impossible for them to come back to these things, even though they fully admit or agree that something is crufty or just out of date by the standards of how a developer might want to interact with that framework.

46:59And also, one of the things that, especially early on, we would kind of talk about in the FreeBSD community is that in a corporate environment, when you're writing software, you're under a time crunch in a schedule. And it's ship it fast, not ship it well. And often you write code that you know is kind of crap or prototype, and that's going out the door. And it's like, there's nothing you can do about it at the engineer level. And one of the things that I think some people really enjoy about working in open source in general, and in FreeBSD in particular, is we have a place where we can do things right.

47:30And you can take time and not, you're not under a time crunch of the schedule in the same way to push something out the door. And you can sit down and think, well, what is a good design? And work through the good design and like take your time to do it well and like maintain it and keep it clean kind of in this platform. And that's one of the reasons I think people are attracted to doing this kind of work in an open source space is without that time pressure of an employer, you get to do it right and kind of do it well. And I like to have peace about what you did instead of knowing that some product shipped and you wrote it and you would rather disown it.

48:02Yeah. Well, talking about shipping, version 15 came out at the end of last year. I believe that was roughly a kind of two year cycle versus when 14 came out. Is that kind of a normal cycle? And I guess what contributes to the stage, the thought around a major version bump on FreeBSD? So it is now kind of a normal cycle. Historically, we've aimed for a kind of cadence of roughly two and a half years or so between major releases. Before that, it was a bit more up in the air. One of the changes that's happened in the last couple of years is we had a different person become our lead release engineer.

48:40And this individual, he's a really great guy. His name's Colin Percival, very smart. He wanted to have a very fixed kind of release schedule, which is what some folks in the community have been calling out for years, but he sat down and done it. And so he's got us on a pretty fixed schedule where we're now, and 15 was our first major release that was kind of on this new schedule that he proposed. We're doing a major release now in Q4 every other year. So every two years we'll have a new, that's like, so 16.0 will be two years from last December. Like that's already kind of set in stone for us.

49:10And our minor releases are now once a quarter. The only difference being when we're doing a major release, we take a whole half year to focus on that. So we don't have a Q3 release on those years. But, you know, so now we have, I think, 14.4 is going to be Q1 this year and 15.1 will be Q2. So now we're actually on a better cadence, which I think is healthy for us, of having releases at a fixed interval. And this means they're not gated on a feature. And so developers still got to develop. That's the sauce that goes into releases. is the itches they scratch, the things they work on. And so it's a coordination of developers working on stuff.

49:43But then being respectful of the timelines too. I think one of the things sometimes we would see in the past when we didn't have a good fixed schedule is when you would announce a release because there was uncertainty about when the next minor release would be. Then you'd get this last minute story of like a ton of stuff that would land in the tree all at once. Not always the most stable of things being merged last minute into the tree. Whereas having a fixed schedule means that we can, and our release engineer is empowered to say no, if he needs to say no, or she needs to say no. And developers know, well, if I miss this one, well, there'll be another one on this branch in six months.

50:18So it's not the end of the world. I know it's coming. I know my bits will get out and it takes a little bit of the stress off. It also allows companies who support drivers in the tree, for example, they have wanted to have a schedule to know how to schedule kind of their internal resourcing. And now they have that. And so they can figure out when they want to put bits into the stable branches in advance enough of what a release would call out. Yeah, that makes a lot of sense. And yeah, that predictable cadence, I think, as you say, sort of helps literally everybody who's part of the process and the end consumer, if you like, i.e.

50:48the company's taking which bits they would like to as well. So as we kind of start cruising towards the end of the episode, I do want to touch on Sherry. Oh, that's C-H-E-R-I for those looking this one up. What is Sherry? So it's Sherry BSD, I believe. And like, what is that project? And yeah, that's a sort of interesting evolution. Sherry is a research project that's kind of come or been developed at the University of Cambridge in the UK. That's where most of the effort is centered. There are a few of us, like myself, who are contractors who can help with the project, who are scattered in the US and other places, but it's mostly a team of folks in the UK at Cambridge.

51:27And Sherry is trying to leverage some ideas from an older set of computer systems called capability systems. And I won't dive too much into that, but folks who are a bit more gray hair than I do might be familiar with capability systems, which were a different way of thinking about software. But Sherry aims to take some of the ideas, not exactly the same capability systems from the past, but some of the ideas and see if we can use them to make existing real-world C and C++ code more memory safe. Because existing C and C++ code is very memory unsafe today. So if you're familiar with buffer overflows, the reason we have them is that at a core level, at the ISAs of contemporary CPUs like x86 and ARM and RISC-V, the way we think about regions of memory is we have a pointer that has the address, the starting address, and it knows nothing else.

52:18It doesn't know how big the region is. it doesn't know if you've moved partway into the region or partway out or if you've kind of jumped in front of the region. The pointer has no idea. It's just a number. It's just the address. And so this means that it is up to software and software engineers to maintain all the metadata correctly about how big it, when I called mallet to get memory, how much did I ask for and keeping track of how much I asked for and making sure I don't get that wrong. And if history has shown us anything over the last many, several decades, it's that we as engineers get that wrong all the time.

52:49So kind of one of the big changes Cherry makes is that Cherry proposes a different register type down in the hardware that is something called a capability. And this part is kind of similar to capability systems where you can have a register that has not just an address, kind of in the low 64 bits or 32 bits, but you have another word of data that goes along with that that includes other information about a region of memory. So it can include information like bounds and permissions. and the bounds are encoded using kind of a floating point scheme to avoid having really big pointers. They're only twice as big, not like four times as big, which was the starting point.

53:24But we're able to store this information about pointers down inside the ISA. And in addition to having this metadata, we also have this one-bit tag on the side that we use to maintain whether or not pointers are valid. And part of the reason the tag is important is it allows us to constrain the operations you can do on a capability. So you can reduce permissions on a capability by reducing the bounds or handing out some of the permissions you already have, but you can never increase your permissions. You can't gain broader rights than the capability that you already have. And in particular, if you try to do an operation that might do that, what will happen is maybe we'll actually modify the metadata word of the capability, but we'll also clear the tag to say that the thing you got as a result, you can no longer use.

54:06Then the other change we make in the ISA is we make load and store operations. now use these capability registers as your base address register instead of a plain integer. So now all your loads and stores and memory accesses have to be authorized by a valid capability. And we can verify when your access and memory isn't in bounds. Are you doing a load against a capability that has read permission? Are you doing instruction fetch against a capability in your program counter that has execute permission? And if you break those rules, you get an exception instead of wandering off into undefined magic machine land where bad things happen.

54:38Then on top of that, So we've modified the ISA to now have this new type, and we modify all the general purpose registers in your CPU to use this new register type. And we modify the memory subsystem to allow us to store tags in memory and make sure it's all coherent. So, for example, if you do atomics, we'll make sure the compare and swap does the right thing with tags all the way through. Or if you do just random memory writes to something, you'll actually clear tags. You can only preserve tags if you go out of your way to do the right thing correctly. Then on top of that, we have extended LLVM to support Cherry architectures.

55:12And when you're compiling for a Cherry architecture, all the pointers in your C and C++ application now become these minor capabilities with metadata. And that includes both explicit pointers, so things you declare in your source code, but also there's lots of things that your runtime creates kind of inside your process that you as a developer never see. Things like got tables, which are how we kind of indirect to get to global variables inside of other parts of your program, or like in C++, if you have a class with virtual methods and you have an array of these pointers to all your function pointers, a Vtable, and all those things become capabilities in Sherry C++.

55:47And it turns out that for some code, things like operating system kernels or maybe your C library or C runtime that implements malloc, you do have to make some changes because, for example, in malloc, you need to make sure that if someone asks for 16 bytes that you get a chunk of memory from the OS. You need to narrow the bounds to make sure that the pointer you return to the caller can only access 16 bytes. So you have to make some changes in places like that. But a lot of application level code, it turns out you don't have to make so many changes. So for example, we've been able to run most of KDE and QT on top of a system I'll talk about in a second without having to make hardly any changes inside of KDE or QT itself.

56:23Because it's pretty clean, well-disciplined C++ code and that just compiles for this alternate ABI out of the box. In terms of places you can run Cherry, when I started with Cherry, it was kind of definitely a very, and still is a research system. It's all sorts of research we're still doing, but it was focused on MIPS. And so we had an extension to MIPS, which is a lovely little architecture, one way to describe it. But other folks have also shown interest in particular arm has found cherry very appealing. And they actually built a custom CPU and sock called Morello around that implements kind of cherry extensions on top of the arm V architecture.

56:59And it's like, I have one sitting over here. It's like a quad core 2.4 gigahertz processor. And that's the thing that like it can run KDE. It can run kind of a full loan Unix. Cherry BSD is a port that we have a FreeBSD to run on top of these type of systems and to support these architectures. And that's so that's my box is running Cherry BSD. Run KDE, run a web browser. We actually even have an early port of Chrome, which is an incredibly amount of work for people to do. But it is a real thing and it can run real software. There's also an interest in the RISC-V community. There's currently an ongoing effort to standardize a new extension called RVY.

57:35Hopefully later this year, perhaps, will finally be ratified as an alternate base ISA to RISC-5 to support Cherry systems. So it's a very interesting bit of work. And it kind of complements other work going on the kind of memory safety space. So if you look at other things like Rust or other languages like that, Rust still needs bits of itself and its runtime and so forth that are written in C. And so Cherry gives you the opportunity to enforce the memory model all the way down into even the bits of C to interact. We had an earlier research project that dealt with this in Java and dealing with JNI where you could enforce the memory model from Java down inside the JNI code that was running because it had to use Cherry capabilities to access anything inside of Java's heap.

58:18And so all the restrictions that Java wanted to enforce are things like unsafe Rust code could still have the same restrictions because it's in the ISA. It's not something that can be bypassed. It's not purely maintained kind of in software. Yeah, I was going to touch on the Rust piece because we do have a lot of Rust listeners, I believe. And so probably some of them are saying, well, why not just use Rust? So, yeah, and you've given very good explanation of at the end of the day, having a much lower level implementation of this memory safety can be very advantageous. Well, I think memory safety is a big issue, right?

58:49I think Microsoft did a study that something like 70 % of the critical vulnerabilities where they had to ship a patch for boiled down to memory safety in some form or fashion. And we need different toolkits to address different parts of the problem. It doesn't seem to me realistic that we're going to rewrite everything that's currently in C in some other language. There's billions, if not trillions, of existing C code in the world, for better or for worse. And, you know, we're kind of stuck with it. And some of that stuff will get rewritten in REST, but some of it may not. So it may be very hard to rewrite.

59:19so we need different tools for different options and so i think churi is a good alternative to give us more tools to attack the problem so we're coming to the end here you've imparted a lot of knowledge and wisdom today i think a lot of our listeners will find that they've learned i'm not going to say something today i learned a lot today i sometimes ask this question to guests just to sort of round out really just what is something that you know now that you would maybe you like to tell yourself coming out of college or high school, whichever, what do you know now? I mean, related to perhaps software development or a career in software development, like what would you tell yourself then that you know now?

59:57One of the things I've had the privilege of doing a few years ago was teaching on the Greg course for four semesters, OSs, as you can imagine. And so I'd actually did get to talk to my students a few times along these lines. I guess some of the things I would tell them are like, have fun. That's definitely true. Things like when you're doing a job interview, recognize an interview is a two-way street. It's not solely you as the candidate who are trying to do something. You should be paying attention to the culture of where you're interviewing and decide, do I like these people? Can I work with these people?

1:00:26That's a valid thing. You should not purely think about that. It's only a view of being serviced to the company. I always recommended a couple of books. Some books that really I enjoyed when I first got out of college were The Mythical Man Month. It's probably my favorite book on It's sadly far too relevant today. And I think Frederick Brooks said that on the 25th anniversary, which is what I was reading in the 90s, that it was still sadly too relevant and still very relevant today. Another book I really enjoyed is something called Peopleware, which talked about some of the – neither one of these is about like fixing a bug.

1:01:02Neither one of these is about like writing four lists or anything. They're about thinking about the art side of software engineering and what you're doing. I guess one of the things I would tell my students is that your job is not to just grab random things off of Stack Overflow and throw them in because you won't earn your salary if that's all you're doing. Like your job is to like engage your brain and think about solving the unique problems that your employer has. And lots of times that does mean assembling things from other pieces that you find, things you find on the Internet. But there's going to be some wrinkle where they just can't use something off the shelf because the off the shelf thing will be a lot cheaper.

1:01:37So to justify your existence, there's got to be some wrinkle to it. Like lean into that. Like in this case, I was usually told at the beginning, do your homework assignments. Not because I need copies of your homework assignments, because you need to practice. You need to get your 10 ,000 hours in to get competent. Yeah. And I think that's more relevant than ever. We've managed to really not even talk about AI whatsoever on this episode, which is great. I think we seem to touch on it almost in every episode. But I think all the things you've just called out, I was talking to one of the other presenters The other day, we have a regular SED News episode where we just talk about kind of more just very current.

1:02:13And we were talking about what does the education landscape look like for people today? I argued that a CS degree, and disclaimer, I don't have a CS degree, but a CS degree is still very powerful because it's about how you think about solving problems. And everything you've just said, John, I think holds true. It is just about leaning into solving problems and thinking about how to solve problems and so forth. Someone's got to make the AI machines work. That's also true. Yeah. For the record, my son is a first year. He's a freshman in CS. So that's actually a lot of fun because I'm perhaps assisting his teaching and making him learn the painful little bits that maybe he's not getting in class.

1:02:52Well, I think you'll have probably a very good career as a result, assuming he wants to continue with software development. so yeah well john again thank you so much for the time i've learned a lot i'm sure the audience has learned a lot and managed to sort of peek into the world of free bsd which is very interesting and as we've learned powers a lot of what we basically use daily without even realizing so thank you very much thank you for having me