Exploring the Capabilities of eBPF | Author Liz Rice

0:00So the slightly ironic thing is I don't really think very many people will need to write their own eBPF code so I've kind of written a book that I kind of think nobody needs to read but a lot of the time you do things not because you need to but because you you're interested and you want to you know and I'm the sort of person who doesn't I don't learn very well from you know pictures and diagrams I need to feel the thing working I need to you know try it out for myself and I need to see the code and that's what I've tried to do in the book so explain for people who are interested in what is this eBPF thing that people are talking about and how does it achieve all these amazing superpowers that people are talking about.

0:45At DevInrupted, we work to give engineering leaders actionable ways to improve their teams. That's why we're producing a three-part summer workshop series with Linear B. Each of the three workshops will explore the processes that elite software engineering organizations and executives use to deliver better business outcomes and reduce cycle time by 47 % on average in just 120 days. You'll learn how to assess your current performance and benchmark it against industry averages, streamline processes through automation, and improve business outcomes through resource allocation. Learn from the best and take your team to the next level.

1:20Visit our website to learn more and secure your spot today. Hey, everyone. Welcome to Dev Interrupted. This is your host, Dan Lyons, co-founder and COO at Linear B. And today we're joined by Liz Rice. author, and chief open source officer at ISOVALENT. Liz, welcome to the show. Hi, thanks for having me, Dan. Yeah, super great to have you on with us today. You have a really interesting, great technical skill set with a background in system software engineering. You're also a regular speaker at conferences like reInvent, Velocity, DockerCon. You've earned this reputation for making kind of these complex concepts more accessible and more understandable, which is really great because today we're going to talk about your most recent work as an author.

2:17You have a book titled Learning EBPF, Programming the Linux Kernel for Enhanced Observability, Networking, and Security. And in the opening line of the book, you describe eBPF as one of the hottest technical topics of recent years in the cloud native community and beyond. So, you know, we're really excited to dive into your book and what is eBPF and all of that kind of stuff. But first, we want to start out with you and your background. I looked you up on LinkedIn before we talked, and you had a really, I think, interesting career path because I see all these different titles, and you're involved with these different organizations and companies.

3:04So can you give our audience a little bit about who you are, what your career path looked like, and how you got to where you are today? Yeah. So I always knew I wanted to work with computers. Since I was a kid, it was always going to be something in software. My first job out of university was for a company that it's taken a very long and circuitous path, but it's now part of Microsoft. But we were doing networking stacks. So my kind of, you know, first foray into professional programming was all about portable networking stacks. And that made for really boring party conversation in my 20s. So after a while doing that, I moved into, I moved to Skype for a bit.

3:53And, you know, that was something that everybody had heard of. So that was brilliant. And then I moved from there to, I spent a little bit of time working on recommendation systems. I worked for a company called Last.fm, which did music recommendations. And then I did some startups around TV and movie recommendations. And then, and this was all kind of taking me further and further away from like the nuts and bolts of the technology. You know, it's a long way from a network protocol, you know, from a network packet to a movie recommendation. And at some point I was having a conversation, actually in a startup accelerator, somebody mentioned Docker to me and I thought, well, sounds interesting, maybe I should look into that.

4:35And I ended up co-founding a startup that did sort of container scaling way before its time. Terrible business idea, but got me back into this, you know, much lower into the weeds of the technology. got me involved in the container space and then the cloud native space. And I guess as part of that, I learned, I was doing a lot of presentations. I think, you know, if you're working for a startup, you spend a lot of time pitching your products and talking about what you do. And I somehow learned that I was kind of quite good at explaining these complicated things. So it was really nice of you to mention that in the introduction, because I think it's something that it took me a long time to learn about myself.

5:19But I think I am quite good at understanding how to break something complicated into like a storyline that people can build up and follow and, you know, take people on a journey from knowing nothing to actually having a pretty good understanding of how something quite complicated works. Whether that's a container or now EBPF, I hope. Yeah, absolutely. I mean, one of the things that I, I mean, you mentioned your startup and, you know, a lot of startups are before their time, right? You have to have like the perfect timing and a lot of things need to fall into place to get like a huge success. But one of the things I've learned about startups is you really do need to hone your storytelling skills.

6:04That's what people understand, especially us engineers, like coming from that background, at least for me, I was never trained in storytelling or something like that. But I do find that skill to be very, very useful. And yeah, thanks for sharing that wonderful background. Now, the book that we're going to be talking about, so it's called Learning EBPF. Now, is that your second book? It is, yeah. It's my second kind of proper book. I've written a couple of other little report things for O 'Reilly, but this is the second book that you can buy at your bookstore. The first one was about container security.

6:43I mentioned I got quite into containers and the container security book. I really wanted to explore not just like, in fact, not really the tools that you use, much more about how do containers work and what about them is potentially insecure and how could you attack? and what are the different mechanisms that people could use to attack a container and what are the different mechanisms you can use to defend your container. And it was through thinking about container security that I really got interested in eBPF and actually seen my now colleague Thomas Graff back in like 2017 talking about the Cilion project.

7:26We were actually both talking at the same DockerCon and he was talking about how Cineum's built on this amazing technology called eBPF. And I just thought that's really interesting. And at the time, it was a little bit felt like this is cutting edge kernel. You know, you practically need to build your own kernel to use this. But over the years, I've kind of kept, you know, interested in eBPF. We started using it on a project for container security. And then a couple of years ago, I ended up joining Thomas at ISOVALENT and really focusing on EBPF. That's kind of what my main focus is now. That's great.

8:10Well, let's actually define EBPF. What is it? What does that mean? What does it stand for? So, yeah, we have to, because it's a set of letters, we have to say what it stands for. And it stands for Extended Barclay Packet Filter. that. But now you can erase those words from your mind because although it has its roots in packet filtering, it is capable of so much more that the acronym is pretty much meaningless now. So we just think of it as a term, eBPF. And what it allows us to do is run custom programs inside the kernel. I nearly said the Linux kernel. It is also possible these days to run eBPF in Windows as well.

8:54So it's becoming adopted by other operating systems, but the sort of history of it, and it's certainly most widely adopted in Linux, and that's where I'm most familiar as well. Yeah, so it allows us to write custom programs, load them dynamically into the kernel, attach them to any event on the system, and then our program could be collecting information about the event. So we might use it for observability. We can even influence the way the kernel behaves. So we can use it to do things like modify network packets or redirect network packets to build networking functionality, or potentially make policy decisions for security reasons, whether that's dropping network packets or permitting or denying certain activities to happen from a security perspective.

9:48that having the ability to modify how the kernel behaves and modify it dynamically gives us superpowers. And that's kind of why I'm really excited about it. Yeah, no, that sounds amazing. In terms of the type of person that would be interested in this or who you would recommend to catch up on your book, what types of developers or any person, who is the audience really here? So the slightly ironic thing is I don't really think very many people will need to write their own eBPF code. So I've kind of written a book that I kind of think nobody needs to read. But a lot of the time you do things not because you need to, but because you're interested and you want to, you know.

10:36And I'm the sort of person who doesn't, I don't learn very well from, you know, pictures and diagrams. I need to feel the thing working. I need to, you know, try it out for myself and i need to see the code and that's what i've tried to do in the book so explain for people who are interested in what is this ebpf thing that people are talking about and how does it achieve all these amazing superpowers that we're talking about i in the book i'm trying to show you you know and sort of build that up and i think the best way to to explain it is through code. So if you do then subsequently want to go and write eBPF code, hopefully it's a good starting point to do that.

11:20But I think for most of us, it will be, okay, this gives me a feel for how the concepts work. It gives me a mental model, understand how other tools that are built on eBPF, how they're working, and hopefully get some interest from that. So it sounds like for the book, if I get it right, I can really get my hands dirty. Like maybe I am I able to actually write my own EBPF thing? Like I'm making an assumption here. But is this the type of thing where, you know, companies are going to like standardize this? So it's like more out of the box for me. Like I'm not going to have to write my own thing.

12:01Or is it still something like if I want to use it, I'm on my own. Like where does it live in the world of like maturity right now? Yeah. So the book absolutely gives you tons of examples that you can try out for yourself. And I've also put together a Lima VM configuration, because one of the problems with eBPF, and in many sort of environments, is you need to have the tool chain set up and you need all the right things in place. So I'm trying to make that as easy as I can so that people can spin up a virtual machine that has all the right things in the right place. And then you can follow through all the different examples in the book.

12:41Now, whether or not everyone will need to do that, I think most people's experience of eBPF will be through other projects and products. you know whether that's i mean i'm very involved in the psyllium project and psyllium uses ebpf extensively and i think you know we see a lot of psyllium users who are you know that they're interested to understand how ebpf works and they're interested to kind of you know kick the tires and play with it a bit and and you know you'd have this the knowledge of like okay how can I expect my system to see what eBPF programs Cidium is using and, you know, just try and get maybe a bit more knowledge of the eBPF tools that you're using.

13:28But yeah, I think most people will find themselves using, and there are plenty of tools out there that are based on eBPF, you know, a ton of command line tools, TCP dump. If you've used set comp, that's using a sort of a form of BPF. So a lot of us have been using eBPF to some extent, even if we didn't realize it. I mean, that's how it usually goes, right? If you're listening to this pod, it's depending up to you. How deep do you want to go? Do you really want to understand how it works? Or do you want to be more on the surface level? That's usually where things go, more the surface level. I'm trying to achieve maybe something in security, whatever it is.

14:10we're going to dive into all of that because I do want to get to some of the more like specific takeaways. But before we go there, one thing that is really cool about you is that you have written these two books now. And so for our audience of engineers, if they are in a situation where they're like passionate about a particular topic or they feel like they have like a cool technology, Can you give us like pull the curtain back at all on what it takes to write a book or get your work out there if that's something we want to do? Yeah. So I have actually written myself a note to say, Liz, never write another book while you are also doing a full-time job because it's a lot.

14:57And I think both times I've written a book, I've had, I felt like I had quite a lot of material already from doing conference talks. And the nice thing about a book is you get this ability to take, you know, a talk that maybe is half an hour or 40 minutes and really dive in a bit more into the details and the things that you kind of have to cover at a pretty superficial level you can really dive into and you know it in a book if people find that detail boring they can just skip over it and move to the next section so you know you you have that kind of as a reader you have that choose your own adventure aspect of reading a book.

15:35So I felt like I had quite a few talks that I could use as a basis for, and I had a rough idea what the structure of the book was going to be. What actually happened was I kept uncovering more and more interesting information and things that I thought I understood and, you know, it turned out that while I was writing my examples, I'd learn things for myself, which you know it is all part of the fun but and I think that's it's one of the nice things about you know whether it's a conference talk or a blog post or a book or whatever it is that you're creating if you're trying to teach something to somebody else you know in your heart whether you really understood it as well and I find that part really satisfying.

16:19How long did it take? it was probably not quite a year between me first writing the proposal and getting the book physically in my hand but it was definitely in chunks like I spent you know a couple of weeks in the summer where I just took a couple of weeks off and pretty much wrote like three or four of the chapters and I had another chunk of time in the autumn I was like okay I'm gonna have to get get some chapters churned out, take some time. Okay, it kind of gives us like an estimate of what it really takes. I mean, that's like at least a year project commitment to get that going. Yeah, really it is.

16:58And if you work backwards from publishing, prior to publishing there's various editing phases and feedback cycle, and I got some incredibly really good feedback from people involved in EBPF, both on sort of technical aspects and also just helping sort of structure my thoughts. It's something that you do when you're writing a book that you don't maybe do so much if you're just writing a blog post, is really getting that in-depth feedback and that sort of sense of somebody saying, well, I felt like I would have preferred you to introduce this concept earlier. And that's really, really helpful. That's amazing.

17:40Thanks for sharing that. That's super useful information for anyone that wants to go down that path. Now, if we go back to eBPF, and you've mentioned Cilium a few times here, and you've also mentioned, hey, eBPF, like this technology is actually used, I don't know, in these projects, or you might already be using it. What is Cilium? Catch us up on that, and how is eBPF used there? Yeah, so Cilium is probably best known in the Kubernetes world. It's a networking plugin for Kubernetes, although we do also have some people using it in other networking environments, standalone from Kubernetes, but probably 90 % of the users right now are using it with Kubernetes or is in Kubernetes.

18:29And it provides that networking layer. It also provides network security. so i mentioned before you know this ability to use ebpf to get network packets at various points in the stack cilium uses that ability to optimize the way that we connect and pass packets between different entities and kubernetes do things like encryption enforce network policies by using ebpf to compare packets against policies and decide whether or not they need to be forwarded or dropped if they're out of policy. Yeah, so it's a very powerful platform for connecting your cloud native workloads. We also have the observability aspect.

19:15So it's one thing to have networking. It's another thing to understand, to be able to debug your networking if something goes wrong. And so the Hubble component, which gives observability, is a really important part of the project and you get this really amazing sort of flow of network packets you can get really cool metrics we've got a bunch of really nice dashboards that you can put in grafana for seeing how how different aspects of your networking are performing you know different latency characteristics how many packets are being dropped by network policy all kinds of different aspects that you can visualize and all of this is kind of built on top of the fact that we can extract this information from the kernel using eBPF.

20:01Yeah. And in a really performant way. Well, we're like a few minutes into our conversation, and I already know I'm not as smart as you are. So I'll try to ask a few questions that will help orient me a bit here. So one of the things that I usually ask myself when there's kind of this new amazing technology, sometimes if you look back in history, there's amazing technologies and then it takes like a while to get them to be something that's like practical for consumers or like practical for the world. And it seems to me like some of the things that you're focusing on here is like security, right?

20:43So security is one of them. And observability usually goes along with that security component. But what's the difference between, because there's a lot of observability companies out there, there's a lot of security companies out there. What would be the difference if I do not have this eBPF technology versus if I do have it in the world of security? I think there are a couple of things that I would highlight. So one is, because eBPF is sitting in the kernel, however many processes you have running on that machine, whether they're in containers or not in containers whatever is running on that virtual machine or physical machine there is one kernel and if we can instrument that kernel we get visibility over everything that's happening on that machine which is really powerful particularly in a cloud native environment where traditionally we've had to kind of instrument our applications by using sidecar containers so the whole point of containers is to isolate them from each other so that kind of by design means one container can't really observe or interfere with another container and so we have to have this sidecar injecting sidecars into the same pod as a container so that they can observe and interact with each other whereas if we can use the kernel for the instrumentation, then we just automatically have this ability to see and influence what's happening across all the applications on that machine.

22:25Oh, that's cool. So it's a more like centralized location where everything is happening. That's what it sounds like to me. Containers by design, they're designed to be exclusive from each other. That's like one of the pros of it. But you're saying, okay, now with the kernel, we can do like our, I don't know, security inspection there and everything flows through it. Is that like a, okay. Exactly. So whenever your application is doing anything interesting, whether it's, you know, sending a network message, reading something from a file, you know, writing something to a screen, even accessing memory or whenever permissions get changed, all of these things require assistance from the kernel.

23:09The kernel is involved when you're doing basically anything interesting. And so that's a really good opportunity to look at things from a security perspective. You know, should this application be allowed to access a file? Well, having the control, and this is nothing new, having the ability to control this from the kernel. We've had things like, you know, AppArmor and SE Linux and SetCom that I mentioned before, you know, that are using interfaces in the kernel. But the difference with eBPF is we can customize that and have very dynamic policies and we can extract information and pass it to user space.

23:49We can build innovative interfaces. We can extract that information and send it to a SIM or send it to a Grafana dashboard or whatever it is that we want to, however we want to format the information. We have so much flexibility by being able to program the kernel. That's really cool. Is there anything around performance that comes along with that? Or is performance still the same? How do you think about that? The nice thing is that it is typically a really performant way to gather information. Because you don't have to worry about the transition between user space and kernel. Which is interesting.

24:30I actually tried to find some data on like, you know, it's a sort of known thing that this transition between kernel and user space is expensive. How expensive? Well, it depends. So I just have to kind of gloss over that and say it is expensive to transition between user space and kernel space. The fact that you can, with eBPF, see something happening in the kernel, store some metrics about, store whatever information you want about it without making that transition makes it much more sort of performant, really, than anything that requires you to sort of access it. user space. So typically it's going to be an efficient, I mean, like anything, you can then write code that's really inefficient or you can do something to kind of make things not perform well.

25:22But at least in theory, eBPF programs are typically really lightweight. They're avoiding these transitions, so they're not going to have a huge effect on performance. Yeah, that's really cool. Going back to the security side of things, totally understand that now it's in a centralized location, so it's easier. It's a great area to do all of your inspection. Sometimes when, let's say, a new security company would come out, they would say, okay, email security is now a new thing back in the day, and we're going to provide more email security or like against phishing attacks or against this type of attack or that type of attack.

26:09Is there anything specific with eBPF that there's like a new type of security detection or is it more like doing it easier, more performant, that type of thing? I'm going to say it is an opportunity to build a whole new approach to security. so quite often in the security world we think about things in terms of network security and then like runtime security so we're very used to the idea that firewalls should drop packets you know if you see traffic that you don't want to handle you drop the packets that's normal behavior for runtime security we typically see people being a bit more cautious about that You know, okay, if I see something that looks like malicious activity, now, you know, there are sort of ranges of this.

27:05Like, you know, we probably say we're going to have permissions on a set of files and we're going to lock down permissions to, you know, what files people can access. But we typically see, and I used to work for a security company which had the ability to do both audit and enforcement of runtime security. Nearly all customers just wanted the audit, and then they wanted to get all the information about events into a sim and then analyze. After something's happened, they've got all the data and they can do the forensics. And I think some of the reasons around that are performance, and some of them I think are more to do with how hard it is to write a meaningful policy that doesn't break your application.

27:48So you might say, well, I don't want, you know, I want to stop my application from, it's only going to access a certain set of files. But if I got that set of files wrong, then maybe my application would break and then everything would be awful and we don't want security to get in the way of the application. I think a lot of that is because the profiles are really hard to write. If people are writing SC Linux profiles or AppHarma profiles, it's just too hard for people to get them correct. I think there's an opportunity for a much more sort of expressive and meaningful way of talking about these profiles.

28:30And the reason why I think eBPF is going to be key to this is we can express policies, and this isn't going to be an easy problem to solve, but we can do the sort of filtering in the kernel. There's actually some project in Cilium called Tetragon. It allows you to observe security events and you can also prevent events that violate a policy. You can prevent them from happening. And the fact that we can filter those events in the kernel gives us this opportunity to be really high performance, to not stand in the way of the application. But I think there's still this aspect of being able to, as a developer, say, here is my app.

29:21It talks to this service, that service, and the other service. It expects connections from here. It writes to these files. It doesn't do anything crazy about escalating privileges. So if we see anything like it suddenly wants to get Capsys admin or it suddenly wants to start initiating network connections to some unknown destination, those things should be blocked. And I think being able to express those rules in a way that really makes sense to developers is going to be sort of the next green field for security applications. Thank you for explaining that. That sounds amazing. The last question that I wanted to ask you in this area of the show is, we have dived in to what I would consider the practical nature of EBPF.

30:12Is there anything more futuristic that comes to mind for you? It's not being used that way now. Maybe you found it in your book. I don't know, like any like future dreams you have for it? Well, it's something that, I mean, is already in progress, but I think it speaks a little bit to the breadth of things that we can do with eBPF. There's a project called Kepler that's being used to measure the energy efficiency of software. So it's hooking into various points in the kernel to sort of measure essentially how much CPU time, how much I actually interviewed one of the maintainers of the project and he was explaining how every time you retrieve information from cache or if it's not in cache and you have to go and actually hit memory, then that is more energy intense.