ByteDance’s Container Networking Stack with Chen Tang

0:00ByteDance is a global technology company operating a wide range of content platforms around the world, and is best known for creating TikTok. The company operates at a massive scale, which naturally presents challenges in ensuring performance and stability across its data centers. It has over a million servers running containerized applications, and this required the company to find a networking solution that could handle high throughput while maintaining stability. eBPF is a technology for dynamically and safely reprogramming the Linux kernel. ByteDance leveraged EBPF to successfully implement a decentralized networking solution that improved efficiency, scalability, and performance.

0:43Chen Tang is an engineer at ByteDance, where he worked on redesigning the company's container networking stack using EBPF. In this episode, Chen joins the show with Kevin Ball to talk about EBPF, the problems it solves, and how it was used at ByteDance. Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

1:35Chen, welcome to the show. Hi, Kevin. I'm excited to get to talk to you. Yeah, me too. Let's maybe start with, you can introduce yourself. Just tell me a little bit about your background and how you got involved doing cloud native stuff and networking. Okay. My name is Chen, and I'm currently a software engineer in Python. So my job is focusing on networking parts in our data center to keep every service in our data centers, especially deployed in containers, rounding, and to make sure their connectivity and stability. And in this field, we use a lot of different technologies. We are using kernel technology and hardware technologies.

2:19But in the kernel part, we use eBPF. It's a customized program you wrote and you can somehow put it inside the kernel without loading a kernel module. I think for us, the eBPF technology has been developed rapidly in the recent decades and now it's very popular. You can use it in not just networking, but basically you can do everything with eBPF. And you can just write your program and you want them to run inside the kernel. and you don't have to be afraid that your code might jeopardize the entire system. I think that's worth digging into, especially for older developers like me. I remember to get anything into the kernel when I started, it was this months or years long process where you would go back and forth on email and all these different things.

3:10But eBPF, as I understand it, lets you essentially run sandboxed code similar to how you might run JavaScript in a browser. Is that a fair analogy? I'm not quite familiar with JavaScript, but yes, I think you are right. You run virtual machines inside the kernel, and you have the environment prepared for you, and you just wrap your code, your program, and you get them run inside the system. That is really cool. So before we dive into some of the specific ways that you're using it, maybe we could talk a little bit more about eBPF. So what does the programming environment look like for these virtual machines?

3:49First, BPF means Berkeley Packet Filter. It is a technology developed by the UC Berkeley, and it was first to filter packet inside the kernel. And then when people find that this mechanism to do something inside the kernel, it can be used in a different way. So the people just expand this entire technology and we call it the eBPF. The e means extend, extend packet, Berkeley Packet Filter. But we can do more than just filtering packets. And we can run different code inside the kernel. You can filter the packets. You can monitor the entire systems. And you can trigger your custom functions when somebody loads a file.

4:34You can do basically a lot of things. And the interesting thing is that when we talk about kernel, we know it's a big system and it's fragile. If you do something wrong and you probably blow up the entire systems, but EVPF provides you with a mechanism. They check every line of your code. They make sure your code can run safely inside the kernel. So this will help provide developers with powerful tools for them to, they don't need to care about the check, the verifier, everything. They just run their code inside the kernel. If the verifier checks your code is not safe, that might blow up the system and they will tell you you cannot load it inside the kernel.

5:14But once you pass the check and you don't need to be afraid of that. So basically, this is what eBPF is about. So to make sure that I understand, it essentially provides a set of APIs or hooks into the kernel that are more stable than kernel internals. So you can hook into networking stack, you can hook into system calls, things like that. And on top of that, it does a static analysis verification up front to make sure that this is going to be safe to run. Yes, exactly. That makes a ton of sense. So I think this is super cool because there's a lot of times when you need to get down into the kernel.

5:57and I know writing application code, crossing the kernel interface, doing a system call, it's expensive. Yes. What was the motivator for you to start working in eBPF? Yes, let's go back to the networking part. Because why we need eBPF in networking? Because it's simple, because we come back to the container, cloud native things. We run multiple containers on the server and in each container, they own a unique network namespace means the network environment of the container is isolated from the host. So each container, they have its own IP address. They probably have its own network interface card.

6:35And that's caused a problem. So if your application inside the container and you want to send a packet to the outside world, and there is a wall between the container and the physical nick is the host, and you need something to connect between the container and the host interface. And that's we use eBPF. And the mechanism is simple. Let's assume we run a customer eBPF program in our kernel stack. When we receive a packet and the eBPF program will capture the packet when we analysis the packet and we see, okay, this packet belongs to this container and we send them to the container. That's simple.

7:19That's why we need eBPF in container networking. just for a very simple target. Got it. So to make sure that I understand when you're in this container environment, inside the container, it doesn't know it's in a container. It wants to treat its network like a network stack. But outside, you can look at the kernel layer that's running all of these containers. You can say, hey, this packet actually is going to another machine or a container inside of my same virtual machine. There's no need to go through an expensive interrupt stack to go through an actual physical device. I know in software, I can just route it straight to that other container.

7:54Exactly. It's a lightweight virtual machine. You can think container is just a lightweight virtual machine. Yeah, that's really interesting. So what were you doing before and how did you kind of realize you needed to do something like that? Because let's go back to the network virtualized things. Before we run eBPF for container networking, what we use, We still need network virtualization for virtual machines, right? We have virtual switch. I'm not sure you're familiar with virtual switch. It's kind of like a switch running in a software road switch on your node. And when you receive a packet, that virtual switch will capture the package and determine where to send it out.

8:41So basically, these virtual machines, they need dedicated cores to run them because to achieve the maximum performance. So it's heavy. Virtual Switch is heavy and the virtual machine is heavy. And this is for the, well, for most cloud provider because virtual machine is a technology probably like 20 years ago. And container is something new. And the feature of container is lightweight. We don't want those heavy things, but we want to achieve the same purpose. We want isolation. We want security. And we want resource designation to make sure the container is running in a safe environment. And we need two cores, probably four gigabytes of RAMs.

9:26And we make sure the connectivity between the container and outside work. But we don't want that virtual switch. And what we need to do next, and let's go inside the kernel to see what we have. what the kernel can provide. And the kernel provides us with eBPF. But at the very beginning, the eBPF is still not functioning as what we have today. And there is still a lot of work to do. So there's many develops. We get together to think how to make eBPF more powerful for container networking. So it's been developed for five years. And now I think it's mature technology for container networking. So basically you have a very many popular projects, open source projects, aiming to use eBPF to serve for container networking in each aspect.

10:14For example, not just connectivity, but like security, and you bring a lot of cloud-native concepts inside eBPF. That's interesting. So actually, let's maybe walk through that journey a little bit. When you started working on this and you said, hey, we have these heavy virtual switches. We want to get rid of those. And you looked at eBPF. You said it was not ready. What was missing and what were the things that you needed to add or develop to get this to work for you? Yeah. So the thing is you use eBPX, but fundamentally you are leveraging the ability from the kernel. And eBPX, they help you to expose some of the key functions from the kernel for the user.

10:59But at the very beginning, that function exposed is not enough. So you will have these problems. The performance is not good enough, and you can write a very simple program. If you write the program more complicated, like I said before, we have a verifier. And the verifier will reject the program because it's too complicated for you to recognize if it's safe or not. So you will encounter many problems. So the people keep working on this, keep optimizing the entire system. So basically, this is what all the eBPF programs have been doing in these five years. Yeah. To make sure I understand, they were, one, extending the surface area of what you could do, what the hooks were, things around that.

11:48And two, improving the static analysis verifier to be able to handle more complex cases and still prove them to be safe. Exactly. Got it. Okay. So you mentioned pulling a lot of different cloud-native tooling into eBPF. What are some of the different ways in which you are using this in your stack today? Yeah. So basically what we are doing now, focusing on two parts. The first part is networking. We have packet direction, and we have to enforce different networking security policies. It works as a data plan. We receive policies from the remote controller, and the eBPF help you to implement those policies.

12:30And the second different part is the kernel tracing. Like we said before, the eBPF, it provides a lot of hook points inside the kernel, and we call them trace points. And each trace point, you can, like, when the trace point is triggered, and you can run your custom eBPF program to help you to understand what is going on inside the kernel. Like, for example, when the functions is called and you assume some bugs has happened and you want to know what is going on inside these functions, and you can bury your eBPF program inside this hook point and when the event is triggered, eBPF will help you to print everything you want, especially the context at this moment, and to deeply understand what is going on inside the kernel.

13:16Because most of the time, it's a black box. But with the help of eBPF program, you can really understand the kernel. It's almost like you can insert your observability stack down into the kernel and say, oh, we think there's something going on with this function. Print me the context on entry. Print me the context on exit. How much overhead is there in this tracing? The kernel tracing, actually, the cost is very high. So we don't use them often. We use them only when there's a bug is reported. We need to analyze the kernel. But for packets filtering, because there are different kinds of trace points inside the kernel, some trace points, the cost is low and some is high, especially when you want to bury your customized trace points inside the kernel, the cost is very high.

14:02But for most, especially in the networking part, all the trace points, that is not a trace point. The hook point, the trigger of those hook points, the cost is low. So we use it for networking. We direct tons of millions of packets and the costs can be limited. Let's see, the cost is acceptable. Yeah. Now, what does it take to roll out? Like, say you want to turn on tracing. Can you do that live on a running container? Do you need to restart the server? What does that lifecycle look like? Yes, and this is the attracting part. You can run the eBPF, the kernel tracing in the live container. You don't need to restart anything.

14:47You just inject your code inside the running system, and you get everything you need. And once it's done and you just cancel it, you can remove the program, and the system will become back to normal. That's wild to me that you can inject observability code down into your kernel on a live system with knowledge of safety, run it as long as you need it and just pull it out. Yes, exactly. That's why eBPF becomes so popular today. That's cool. So diving in maybe a little bit, you mentioned also there's a set of open source projects that have been building and bringing these cloud native technologies into eBPF.

15:27Are you working on any of those or are those areas that you're connected to? Yes. For now, I've been working on some of the open source projects, but I'm mainly a user because I work for a company. So my purpose is first to serve what the company wants me to do. And I will use a lot of open source tools. And when I found there's things I can modify, and yes, I will like to, it's kind of my way to, how to say this is what community do, right? Yes, absolutely. Which projects are you using? Since I first worked for the company, so my top priority is help the company to get things done, to get the job done.

16:09And during this process, I use a lot of open source projects and I get help from the communities. And especially there's an eBPF library called eBPF Go. Now it is the most popular eBPF library for Golan developer. And I use this library to help me to load eBPF program inside the kernel. I think this is the tools I use the most, the open source project. That brings up kind of an interesting area, which is what does the development environment for eBPF look like? Because you're compiling down, as I understand it, to essentially a bytecode that is what gets analyzed and loaded. So what is the environment that eBPF Go, for example, exposes to you?

16:56Okay, so for most developers like me, we wrote the C program. We want the C program to run inside the kernel. And once the C program is done, first we need to compile them with bytecode. and we load them into the kernel. And before we load them into the kernel, we need a verifier to make sure all the code can be run safe inside the kernel, and to run a syscall to the kernel to load the program to the bytecode inside the kernel. And after that, once the program is inside the kernel, it's still not functioning. You just somehow, this kernel helps you to store the program. And if you want them to run, you have to attach the program to a hook point.

17:39And this is what those libraries help you. You develop your program in a C code, but the library helps you to do the load, to do the verify, to do the attachment. Everything relates it like, yeah. Got it. So the core eBPF program is developed in C, but all of the different sort of attach, detach, load, all of this stuff is what you're using eBPF Go to do. Exactly. Awesome. Awesome. I'd love to go back a little bit to the use you used in the network. What was the impact of moving from this big virtual switch approach to the eBPF networking stack? Okay, so that's a very interesting story because it is related to the company's development.

18:23So let's take a look at the companies like Meta or Amazon. Those companies, when they run their data centers, it's probably like 20 years ago. So by the time the technologies they had for virtualization is virtual machines and the virtual switches. So this is what they used at the very beginning. And now we have to know one thing. If you migrate from one technology to the others, especially you have to upgrade the entire data center, the cost is huge. So probably they will keep using virtual machines. They keep using virtual switches for their data centers. But I work for Bythons. I don't know if you know ByteDance, but I think you know TikTok is one of the apps developed by our companies.

19:11ByteDance is founded in 2012. So it's only 10 years ago. So when ByteDance is growing and we need to build our data centers, cloud native technology emerged. So at this time, we have a better choice to manage our data center. That's why we use cloud-native technologies with Kubernetes to run everything as a container in our data centers. So I think this is luck. I don't know, because at this time we have better options and we still have the chance to make a choice. And we choose Kubernetes and container. That's really an interesting point of how coming later, you can often jump over the learnings that the earlier companies had to do.

19:59Maybe we can expand a little bit now into some of these other cloud native areas. What other places do you feel like going cloud native first has made a difference for ByteDance relative to, say, Amazon AWS or something like it? Okay, so now most companies, when they embrace cloud native, first let's make a statement. For companies like Meta or Amazon, they run their data centers in a different way. So when cloud-native technologies emerge, of course, everyone wants to use this, but we use them in a different purpose. For Amazon or Meta, they use cloud-native technology as a service, especially for Amazon.

20:43They are a cloud provider. They use cloud-native technologies mainly as a service for their customers. They don't use themselves. They provide them. But for us, we use ourselves. So the difference is since we have the Kubernetes running inside our clusters, and I think the biggest difference is the problem of scalability. For Amazon, they use Cloud Native as a service for their customers. And their customers, they mainly have a small scale of cluster because they cannot afford to build their own data center. That's why they want to buy machines from Amazon. So the scale of their clusters, let's assume 1 ,000 machines, I think that's a lot.

21:27But for us, we have over a million. So the scalability is the major problem. And we found that the cloud-native technology seems it's very powerful, but it has a fatal problem is the scalability. If you run a Kubernetes on a cluster with 1 ,000 machines, that's enough. and you can have a powerful cluster management tool with basically everything you want. But if we have 100 ,000 machines and you have the Kubernetes, it becomes a bottleneck of performance. So by the time, if you still want to use Kubernetes, you have to do a lot of modifications. Some of the concepts from cloud native ecosystem is powerful, but it's just not so efficient.

22:16we have to optimize them. So I think this is a major difference from how Python uses cloud-native technologies from the others. Yeah, there's a scale factor there that is really, blows my mind. You mentioned finding progressive bottlenecks in Kubernetes and cloud-native abstractions as you scaled up to 100 ,000 and beyond. Where were those? and what have you replaced or improved? Yes. For example, there is a very important concept in container networking. We call it service. Because once you have a client container, you want to talk to your server container. You cannot just ask for the IP address of your destination and you send the message to the destination IP address.

23:04In Kubernetes, they build a concept called service. You ask for the service and the service will return to you a target and they say you can connect to this destiny. But the service, the total cost for you to run a service is huge because a service, you have to store all those backend containers. You have to choose from one of the real server. If you have 100 servers, that's okay. But if you have 100 ,000 and the cost to run this service concept is huge, it's not acceptable. So we have to remove them. So we use a different framework. is like a service discovery. We develop our own service discovery framework to help the client container to discover their target.

23:51So this one difference. And also because for us, because we run our own data centers and the cost things is in recent years, all those IT companies, they're not running so well. You see there's layoff everywhere. so we can see the cost becomes a big problem for all those companies. They want to save money. They want to buy less servers because the server is the biggest cost for our company. So let's see if you have 1 % of improvement for the total cost of the machine and we have over a million servers, the 1 % would be a lot. So this is what drives us to seek for new solutions to optimize the entire system.

24:40We know Kubernetes is powerful. The cloud native concept is very useful, but sometimes it's just not what we want the most at this very moment. Yeah. So let me make sure I understand the service example. So in Kubernetes, you have the service abstraction, and it kind of assumes a global view. is going to index all of the different servers or containers that might respond for this service type. And when a client asks, it looks up in its index, who's free, sends them the target. That kind of assumes a bounded set that is small enough to perform quickly in that type of index. And when you scaled up large enough, you needed to say, oh, we actually need a much more contained version of service discovery that's not going to be trying to index 100 ,000 machines.

25:35Exactly. That makes a lot of sense, I think, and points towards a potentially even just class of problems where Kubernetes is providing this nice abstraction where it's going to do all the management for you of all these different things, package it up in a single location. And that's going to run into scalability concerns as you go far enough. Maybe you bump out of a cache or a memory or something like that. Were there other areas that you found needing to kind of almost scope down the abstraction exposed to be able to deal with that level of scale? Actually, I'm not expert in this field because we have a different team that manages the resource orchestration.

26:21And they are the teams to build the Kubernetes systems. I'm focusing on the networking part. What I know about the service things reaches my knowledge boundary. Okay. Looking within networking then, again, we talked a bit about using ePPF to make a much sort of more efficient networking stack within the physical machine. So you connect to other areas of the networking stack that being cloud native from the beginning has really made a difference. Yes, we have IP tables. That is old technologies being first used when cloud native. Actually, at the very beginning, when container networking becomes a problem to be solved, people use IP tables at first.

27:11But IP tables, it's slow. and then we use eBPF to replace IP tables. So now eBPF becomes the majority and IP tables just emerged from at the very beginning and then it gets replaced because of its bad performance. So I understood the eBPF approach to network device, replacing the virtual switch. Instead, you just drop into this kernel hook that looks in different places. What does it look like for IP tables? Iptables is also a function inside the kernel stack, but Iptables works in a chain format. For example, if you can write a chain of rules, you have the match and the actions, the packet match and hit these rules and will proceed with the action, it is a chain.

28:02And you can pass the packet to another chain. So the entire Iptables system is just a bunch of chain, it's a chain system. And we can say it is not efficient. And especially once the rules expand and the cost will not be acceptable. But the eBPF is a single program. You write your own program and one program will be enough. So that's why people replace IP tables and choose to use eBPF. And when it comes to the virtual switch things, I remember you also asked how eBPF replaced the virtual switch. And no, So EBPF doesn't replace virtual switch. They just work in different scenarios for different purposes.

28:45Got it. Let's maybe go in on there because I misunderstood. Which scenarios are you using the EBPF networking direction versus the virtual switch? So why we need a virtual switch and EBPF? Because they serve for a different purpose. EBPF focusing on container networking. Why eBPF can be used in container networking? Because it's a lightweight, isolated environment. There is still only one system, one kernel. Even containers run in an isolated environment, but actually the kernel is the same as the host one. So the eBPF just play the tricks on the host kernel help you to redirect the packet from the physical nick to the container.

29:34But in the virtual machine, virtual machine is a full isolated environment. They use a different kernel than the host one. So it's impossible for the host kernel to talk to the guest machine. So you need a different mechanism to redirect the packet to the guest machine. That's why we use virtual switch. So basically, these two technologies serve for a different purpose. To make sure that I understand it, the way that I'm seeing it right now, it's almost like there's layering. So if you have containers within a single machine or virtual machine, you can route between those containers purely with EVPF.