Cilium, eBPF, and Modern Kubernetes Networking with Bill Mulligan

26:52So if you're running multiple Kubernetes clusters, this is another thing a lot of people turn towards Cilium for because it simplifies that. Cilium allows you to look at network policy, not in just one Kubernetes clusters, but across multiple Kubernetes clusters. Kubernetes gives you basic network policy and Cilium allows you to do much more advanced use cases around network policy. Yeah, makes sense. So how about, is there anything more around network policies or do you want to go to service mesh? We can go to service mesh. So yeah, talk to us about service mesh. Again, I think what is a service mesh first and foremost and then how is Cilium helping to that end?

27:26Yeah, so anybody that knows me might know that I'm trying to kill the word service mesh and the category service mesh. If you go look at it, there's an article that I wrote that I think explains a lot of my opinion that it's called like the future of service mesh is networking. And so service mesh is a little bit newer than Kubernetes. And it's once again, like, okay, new cloud native world, like there's kind of a lot of things that we need to rethink. And a lot of this is like service routing. So service mesh is a term that tried to mean a lot of different things. So like if we have like microservices, we have a lot of new challenges of how do we do the networking between them how do we do the observability how do we do the security between all of them like microservices running all over the place and service mesh tried to solve this with a lot of like layer seven networking stuff so right it's like networking observability and security a lot around like layer seven stuff and this is why i kind of have a problem with like the category like service mesh is like if you look at all those like those are all like fundamentally like networking things.

28:31And if you're trying to do it at just one specific layer, you might be missing a lot of the context from all the other layers. So I think we had a big arc where service mesh was very hot and a lot of people were trying to implement it. But I think we're kind of getting into a phase where people are understanding that you can't separate out the different layers of the networking stack. You can't be like, oh, I'm just going to look at only layer three. You actually, if you want to have the full context for your application, you need to look at all the layers and you need to look them like holistically right because like an example that i've heard is people are running psyllium as a cni and then psyllium has a service mesh which i'll get to in a second but you can also run a different service mesh on top right so they're running psyllium and they're running a different service mesh and they're like okay well psyllium does a lot of smart things in ebpf it doesn't use ip tables it just reroutes traffic and do things like just route packets from socket to socket within the same linux host you do a lot of things that will make it much more efficient, scalable, performance, save you CPU cycles, but it doesn't mean all the other parts of the networking stack know what's happening.

29:37So an example would be like, Cilium can route things directly from one socket to the next, and it doesn't go through the whole Linux kernel networking stack. So the service mesh is looking at the end of the Linux kernel networking stack and it's like, okay, I'll do this like layer seven processing. Once it comes out of that, what like actually never goes through the networking stack. So you never see the packet. And so you're like, okay, like all this traffic is disappearing or we don't route it or we don't see it. And it's because it doesn't go through the traditional networking stack as the service mesh was expecting.

30:07And so service mesh, I think, can't be its own standing run-alone category. You need to think of it in the context of your whole networking stack. And so this is why Cilium came along. It was originally just a CNI doing a lot of this layer three, layer four, like routing. But then we're like, okay, well, networking is not just like a couple layer standalone category. It's actually like you need to have the context of the full stack. So what we did is we came up with Cilium service mesh. And this is where some of like the layer seven network policies started to come in and also doing other things like traffic routing in layer seven, some of the observability stuff, but integrated within the rest of the CNI and the rest of the networking story.

30:47And I think I also have a problem with the term service mesh because it's kind of nebulous. It's like, where does service mesh live? Where does networking start? Where do they overlap? Where they're all just overlapping concerns. What I think of today as Cilium service mesh is Cilium's gateway API implementation in Kubernetes. And some gamma. I think, sorry, I have a lot of problems with service mesh. Yeah, there's a lot of emotion comes up with service mesh. Yeah, it's good, it's good. But I think it's also really funny. So I look at the analytics for the website, and one of the top, I think, three pages is the Cilium service mesh page.

31:26So it's what people are interested in. But I'm like, so what are you actually interested in? Is it like the routing? Is it like the observability? Is it like the security? It's like people are trying to solve, like service mesh isn't a problem. What they're trying to solve is like, okay, we need to do like layer seven routing and like host tasks or something, or we need to do layer seven network security. Like that's the actual problem you're trying to solve. Like service meshes is like this kind of like nebulous term that somebody told me I needed it. Yeah. I mean, I think I can probably give then my perspective on that because yeah, working on a specific problem, you know, I'm not an engineer by day anymore, but working pretty sort of hand in hand with pretty advanced engineers.

32:06and when i was looking at what we're needing to achieve and did a sort of bit of running around sort of understanding okay what are the bits to the network stack that we need to think about here service mesh just kept popping up so i'm like okay guys do we have a service mesh uh it was kind of my first question just to kind of maybe get a sense of do we have a concept of this thing or not and then you know i got my answer and then we move on from there so i guess it is it's just sort of a catch-all term to help to that end and maybe that's why people are looking up so much on the website because it's sort of like yeah but that's just an anecdote i guess but yeah yeah i think you're exactly right i think it's kind of like as we saw this transition to the cloud native world there is kind of like a lot of like quote-unquote new problems that were like as we have a bunch of microservices like there's like new networking problems and new like network security problems and like service mesh was the label to solve a lot of those problems it's like okay, we want a service mesh to solve the specific problems that we have.

33:08But kind of how the Cilium service mesh came around is like, we also were like, people started asking us about service mesh. And we like looked into it and we're like, okay, what does the service mesh actually look like? And we're like, okay, with what we have so far, we actually have like 80 % of the service mesh because it's a lot of like networking, network security, network observability parts. The only part we're missing is like a bit of like the layer seven stuff. And so we're not actually building like a whole service mesh from scratch. We're actually just adding that last 20 % is what people are looking for.

33:37And so that's how we originally came out with what was the Cilium service mesh. Got it. So let's maybe move on from service mesh. That's been, I think, super helpful. And I'm sure there's a lot of people that will sort of look at the term service mesh differently now. Sorry if I'm destroying people's hopes and dreams. One thing to solve it all. Yeah, exactly. That's all we're looking for. We're always looking for a term that just solves all our problems. Yeah. psyllium also does observability to my understanding through i guess a sort of arm of the product called hubble could you maybe talk to us a bit about about hubble yeah definitely so this is like everything else in kind of the psyllium ecosystem is based on ebpf and so what hubble does is like okay since our ebpf programs are in the kernel routing all the traffic and we can see all of that what if we just like took that information and surfaced it to the user.

34:31And to be honest with you, I think Hubble is the favorite feature of basically every user I talk to. The quote from the ESNet Energy Science Network, which is all the national laboratories in the US, they're doing crazy stuff like IPv6 only Kubernetes cluster, and it's like, Hubble's a godsend. It lets me what used to take multiple days of engineering time, I can now solve it in 30 seconds. And so, going back, if we think about it, distributed computing packets are flying everywhere. We need to be able to understand it. It's not just like, okay, let me debug my one application. I can follow it through the whole program.

35:10It's like, okay, applications are making calls out to different programs. It's going over the network. We don't know where the packets are going. We don't know where the information is going, where things being dropped. This is where Hubble came along. It's like, okay, so if we're actually routing all the packets already with eBPF, why don't we actually observe them? So Hubble just piggybacks on top of the CIMM-CNI and basically takes the information, from the CNI and surfaces it to the user in a couple of different ways. So one is network flow logs. So it's basically saying, here are all the packets going through.

35:39This is where they're going. And then the other one is the Hubble UI. And this allows you to create a service map of everything that's going on in your cluster. And you can see where things are being routed, where different things are connecting. And also, I think more importantly, where traffic is being dropped. Because to be honest, when most people run into networking, Like everybody likes to not have to think about networking. The only time they do is when things are going wrong. And with Hubble, they're able to very easily visualize either through the UI or through the flow logs. They're able to see, OK, where is our traffic being dropped?

36:13Right. Because that's probably like most people are concerned. The security team is concerned about like, OK, where are things going that they shouldn't be? But like most developers on a day to day basis, they're saying, why isn't my traffic reaching the destination? And Hubble is a great way to understand that. So it can give you the reasons for like the policy drops. You can be like, okay, our security team wrote all these new network policies and deployed them into the cluster. And now all of my traffic's being blocked because they didn't want this type of traffic going in the cluster anymore.

36:42And then you can go have that conversation or deploy a new service. Why isn't any traffic reaching it? Well, okay, like it's in a new namespace and we don't have the network policy. We have a default deny all network policy in our cluster and we forgot to write one. Okay, we should allow traffic to this new namespace. So people love Hubble because it allows you to have insight into where all the network packets are going in your cluster in a very easy way. How would this be done, I guess, without Hubble? You just have to kind of roll your own? So Linux, right, like decades old technology, there's a lot of networking tools to be able to understand where things are going, like TCP dump.

37:20But that's right if you're using the networking stack. And the problem with BPF is like there's kind of like this sometimes people say EVPF magic that people sprinkle on. It's like this black magic happening in the kernel and the packet just disappears because it's routed from one socket to the other and it doesn't go through all the traditional tooling. Right. So as you're switching how you're doing things, you also need to come up with new tools. So, yeah, Hubble is one of them. And there's also another project under Cilium called Peru, like Packet, Where Are You? and that's another great debugging tool that allows you to essentially pull more information out of the kernel.

37:57One of the reasons that people love eBPF is because you can hook anywhere in the kernel. You can pull any system information that you want. You can modify any system information and you can essentially pull out this firehose of data. The limitations of some of the previous tooling is it's either one, it doesn't pull out the information you need or it's designed in a certain way. It's like, this is what it does. but eBPF allows you to look for anything that you want to. You want to pull out new type of information from the kernel, well, you can write a program to be able to do that. And so what Peru and Hubble allow you to do is to surface exactly the information that you want rather than relying on tooling that may not even work.

38:34Yeah, I think that's a helpful call out given, as you say, eBPF could be seen as a bit of magic. Then, unfortunately, with the magic often, yeah. I mean, my sort of long ago version of that was like when you start using Ruby on Rails, for example, and then it's like yeah but we need like tools to actually then understand what's going on because it's just a whole layer of magic passing data between the back end and the front end and i need to sort of understand what's going on there for example exactly so you can think about it new tooling for the new world so we've kind of looked at what psyllium does and why developers might want to look at it or why they're using it already let's maybe just spend a little time going a bit deeper on how psyllium actually works sort of under the hood so we've got the idea of like what is a psyllium data path for example and then we can maybe sort of jump into just the actual component architecture after that so i believe there's sort of a concept of bpf hooks maybe we could kind of start there and just sort of how does like psyllium actually work i guess so kind of like the basic architecture for psyllium is there's a psyllium operator running into your cluster and this runs kind of like the life cycle for all the psyllium agents and the way the psyllium agents work it's like daemon set so it's one psyllium agent running on each of the nodes in your kubernetes clusters and what the psyllium agent does is it installs all the ebpf programs onto that specific node.

40:02And so what the agent does is it gets information and basically writes all the BPF programs and installs them actually into the cluster. And the interesting thing about this architecture is it in some ways like simplifies a lot of the networking and upgrade lifecycle because the data plane, which is the BPF programs running in the kernel is actually separated from the actual lifecycle of like the control plane, which is the agent and the operator running in the cluster. And so one thing I didn't touch in before with eBPF programs is that you install them into the cluster and they start running and you can de-install them.

40:46And so what that allows you to do, and there doesn't have to be any communication between like kernel space and user space between like the control plane and data plane. So the agent is installed on the node. It installs all the BPF programs. the BPF programs start routing all the networking packets, or they're with Hubble, and they're observing all the networking packets. And with that, you're able to essentially, if the agent goes down, it doesn't matter because all the programs are pre-installed, and they're still going to be routing the packets. The only thing that won't happen is you're not able to update any of the data path because the agent's not there anymore.

41:19And so what you can do is you can update the agent, new agent is there, and it can then modify the BPF programs. It's a little bit different with Envoy. So that's with all the BPF stuff. Some of the Layer 7 stuff, which I guess I didn't touch on yet, is done with Envoy, which is a very popular service proxy. It's what a lot of the other service meshes are based on. And Envoy is also run as a daemon set. So one Envoy in every single node in the cluster. And that works with the Cilium agent and the BPF programs to do some of the layer seven based routing. And so the BPF programs and the Envoy together kind of create the whole networking story and they work in concert with each other.

42:03And then layer seven is a little bit different because this is something that they're working on right now. It's like some layer seven connections are like more long live. So if you restart the Envoy on the node, then it like resets some of the connections, but now they're working on like hot restart of Envoy. So that story's changing a little bit different, but I guess what's different about like the ceiling architecture is like The agent is separate from the BPF programs that are running in the kernel. So the data path can keep on running even as things are changing on the control plane too.

42:32So then we've got the Cilium agent. I guess there's also the Cilium operator as well? Yeah, the operator manages the lifecycle of all the agents in the cluster. Gotcha. And then the CNI plugin and then identity management, which we've obviously touched on a little bit as well. so in terms of getting up and running with psyllium maybe we could take sort of two brief examples one is like total greenfield which is obviously hopefully the easier one and then one is you've already got like a kubernetes cluster kind of running something medium advanced so maybe we take those kind of two cases like what are we talking in terms of getting up and running so you have a brand new kubernetes cluster there's a couple different ways to install psyllium so the first and the easiest one is you're on a cloud provider doing something like GKE or AKS and Cilium's already the default on that cluster.

43:24So the cloud provider sets up your cluster, you get Cilium off and running to the races, it's already in there. I think that's one of the cool thing is like Cilium's already the default CNI for a lot of managed Kubernetes clusters. The next one is you're setting up your own Kubernetes cluster. And this is like super common use case for like on-prem or other things. Cilium has like a couple of different tools. I think most often people install Cilium with the Helm chart. So install that with Helm into your cluster. You can also install it with the Cilium CLI, but this is, I guess, maybe not as recommended because then it's what things you pass into the CLI and trying to remember that versus like a Helm chart.

44:01So you're going to probably see most people install Cilium with the Cilium Helm chart. And then the migration story is something that we see commonly because when I look at the way like people users set up the kubernetes cluster it's like what i was saying before nobody wants to care about the network until they have to care about it and they don't usually turn to psyllium or they might not always use psyllium until they come across one of the problems that they're having with like that psyllium helps solves like i was saying before so something like performance and scalability the network policy or encryption the observability aspect or like multi-cluster networking because like if you're running on something like open shift open shift has their own cni that they install into their communities clusters or aws has the aws vpc cni so there's already one installed in there or maybe for instance a lot of tutorials start with like flannel and so you already have a kubernetes cluster running with a different networking cni in there and you run into one of these challenges that you're like okay well I need to have better performance and scalability in my cluster.

45:08And our security team says we need to have these layer seven network policies. And our application developers are having a hard time debugging the cluster because they don't have any insight into where the traffic's actually going. And we're thinking about spinning up some more Kubernetes clusters. And you're like, OK, well, Cilium solves a lot of these problems. So I think we should migrate to Cilium as our CNI. And the question becomes, OK, how do we do that? And so the migration path that we see most commonly that people do is the cool thing about Cilium is it's not like kind of like a big bang where you're like, okay, like we need to like switch this over.

45:46It actually allows a lot of incremental things. And so one common path is I see people doing is like, okay, we need better observability in our cluster. And so what you're able to do is to do CNI chaining. So you essentially have like your first CNI say, like, fine. And you're like, okay, I need better observability because it doesn't have any observability. And so you're able to essentially install Cilium on top of Flannel. Flannel still does all the network routing, but Cilium is able to see all that networking and being able to surface that information through Hubble. So you now have the network observability without having to change any of your data plane.

46:20And you also now have the added benefit of having Cilium as a CNI in there. Or people also install Cilium for network policy. And so we don't want to change our data plane. Actually, there's a lot of companies that actually wrote their own in-house data planes. So for example, Alibaba wrote their own CNI, but they wanted to add network policy. So they installed Cilium on top to do the network policy part. And so now you have two CNIs in there. One's doing the networking, one's doing the network observability or network policy. And what you're able to do is Cilium has this flag called Cilium node config.

46:55and you're able to specify on each node in your cluster which CNI you want to do the network routing. And so at the time you're able to essentially say, okay, I want all of my original CNI to do the network routing. But what you can then do is you're able to drain traffic off of a node and say like, okay, we want this new node to be, as you're adding new nodes to the cluster, we want this new node to be using Cilium as a CNI for traffic routing. And so you can basically roll over your whole cluster node by node. And each node that's coming online now is able to have Cilium as the CNI until eventually you're basically switching over all your traffic to the new CNI.

47:36Cilium's your CNI. And at that point, you can uninstall the old CNI. Wow. Yeah, that's really cool. Yeah, I like the CNI chaining. That's super cool in terms of... Yeah. Yeah, because as you say, migration is probably the more likely case for a lot of people maybe listening today that aren't using already. but unfortunately that's also often the reason it doesn't get adopted as quickly because it's often challenging yeah nobody likes to mess around with the network yes quite yeah if it's working just like leave it right but yeah if you start to run into some of these problems you're looking at a migration story and a lot of people want to do live migrations so if you go on the ceiling website there's actually quite a few stories like the most recent one was from db schenker which is like the German national rail like logistics thing and they needed to migrate over to Cilium and they did the Cilium node config and were able to do like a live migration of Cilium.

48:27Exactly that's infrastructure sort of at some of the most important levels so that's pretty cool. So kind of looking at as we touched on right at the beginning Cilium is a very open source project and again looking at the so just some of the stats from the report that you guys just put out there's a number yeah a hundred a thousand individual contributors you cross that line basically on the project in october 2025 so that's a huge number that's a huge number of contributors so like this this is surely one of like the i guess largest open source projects out there yeah i think this is like kind of interesting for me because it's also if you're looking at like other like vanity metrics like github stars and things like that it's like okay how do you measure the size, success of the project?

49:12And a lot of the people that I'm talking to, like Cilium is run by the platform team, and it's four engineers supporting 200 developers. And so if you look at a lot of other projects, they're going to be a lot bigger, but it's just because the number of people actually interacting with that is a lot bigger, like a front-end framework or something like that. It's going to have 200 developers for one SRE supporting those 200 people. And so it's kind of wild to me that Cilium is kind of growing into like such a large project, right? If you're looking at, there's a lot more developers that can write like HTML than can write BPF code, right?

49:47So like a thousand contributors is actually quite a lot in some ways. In terms of like stats, so depending on how you look at it, Cilium is in the top three projects in the Cloud Native Computing Foundation. So 200 some projects are the largest ones in the Cloud Native space. Kubernetes, obviously number one, because it's also the second largest open source project in the world. Number two and number three is Cilium. So yeah, one of the fastest moving projects in the whole CNCF ecosystem. Yeah, I think I said in the report, now the second largest. Cilium is now the second largest. That's pretty awesome.

50:23So yeah, I mean, I guess sort of on that note, sort of in terms of community and contributions, is it usually someone who's kind of, I guess, using already through their company, I guess, that sort of ends up then jumping in and making a pull request for something that they've seen? Or do you also just have kind of diehard Cilium fans that just work on this? Yeah, I would say the most common thing, it's kind of like what I was saying before, like the number of people that can write eBPF code in the world is not that large. The Cilium agent is written in Go, so it's a little bit different there.

50:54I think it's a bit more approachable. But yeah, Cilium is like a pretty deep networking technology or like technology as a whole. And so, yeah, the most common use case is we're running Cilium in our production cluster. We're running into this issue or bug. It's open source. So we're contributing this fix because nobody else in the community is working on it and we really need this. So that's actually how some of the earliest maintainers of the project came along. So some of the earliest ones were actually from Palantir and Datadog because they were running Cilium in production. They needed things solved.

51:28Easiest way to do that is to upstream the changes into the project. They got more and more involved and became maintainers of the project. And then in the exact same way, two of the other big companies, like I said, Google and Microsoft use Cilium as the data plane in their managed Kubernetes cluster. So they need to get involved to be able to upstream their changes. So it's people running Cilium in production that are trying to solve the issues that they have. And that's really how they get involved. Looking ahead, when it's sort of this open source, like roadmap, it can be a bit of an ephemeral term, but like, what are you, you know, 2025 looks like it was a pretty big year.

52:04Like, what do you think is kind of on the horizon through 2026? I think there's a couple of different things. So one, like in the Kubernetes world is really starting to see this transition to IPv6. I was just at Cilium Con in Atlanta, right next to KubeCon. And there is actually two talks from ESnet and also from TikTok talking about using IPv6 only Kubernetes clusters. I think maybe this is finally the year of IPv6, but I think we're really starting to get there. And a lot of the work done this year by the project was around how can we basically bring IPv6 feature parity up to IPv4. So I think there's a lot of work being done around there because we're actually starting to really see IPv6 only clusters going into production.

52:49And not only that, going into production at scale. If you look at the scale of the national labs infrastructure in the US, it's pretty big. Also, TikTok also has quite big, very different use cases. So that's one that's in Kubernetes itself. The next one that I didn't touch on that much is around, I guess, everybody's kind of aware of the whole VMware thing. And so people are looking to migrate off of that. I think that plays into Cilium a couple different ways. One is how can we bring VMs into Kubernetes? and another part is how can we connect Kubernetes or like the cloud native world with the rest of our IT estate, like sitting in virtual machines outside of that.

53:32If we're migrating and modernizing, like how do we still connect it to what we already have? So there's kind of like two pieces there. So one is how do we run like VMs in Kubernetes? And Qubect, I think a lot, is what a lot of people are using. And one thing that I'm really excited about that Cilium's coming up with is this thing called NetKit. and this was originally developed to solve the container networking overhead. So the problem with containers is it's a process running on the host in its own network namespace, and by traversing into the network namespace, it creates essentially some overhead.

54:07And what NetKit allows you to do is to basically take something off of the NIC and put it into the container with essentially no overhead, so eliminating the networking overhead of containers. right since kubbert is kind of like a vm running in the container in kubernetes what we're doing now is like can we use netkit to get the packet into the vm directly from the nick so eliminating the overhead of not just the container but of the vm running inside the container running inside of the host so it's once again how can we reprogram the networking stack to make it faster more efficient more scalable as we add more layers of abstraction how can we still kind of make those abstractions, things that we can still get the performance that we need to out of our clusters.

54:49So that's on running VMs inside of it. And then the second part is connecting to the outside world. So we're actually doing a lot of work at Isolvion, like getting Cilium to connect to like the outside world. So VMs running outside of your cluster or VMs that you're trying to migrate into your Kubernetes cluster in a seamless way. You know, it's once again, how do we make that transition that migration story as easy as possible and doing that in a very smart way with eBPF. Awesome. Very cool. So NetKit, is that that'll be available in 2026? No. So that's actually out now on a part of Cilium. So this is another thing.

55:29It was made by Daniel Borkman, who's also one of the co-creators of eBPF. That was kind of like the next project he's working on was NetKit. And that was a Linux kernel feature. So it's actually part of the Linux kernel, Cilium uses NetKit to be able to eliminate that container networking overhead. And then we're also implementing it to work with Kubevert too. So the whole part with four containers is already, it's in the kernel and it's in Cilium if you're running like the right versions. And then like the part with VMs is kind of for next year. Awesome. So just as we wrap up, where's best for a developer or just someone who's kind of interested in Cilium and maybe thinking about, I don't want to say throwing over the fence to the developers, but throwing it into the mix, like where's the best to go and just sort of get acquainted with Cilium?

56:17Yeah. So I'm extremely biased because I'm a maintainer of the website. So I would say go to Cilium.io first. And I think there's a lot of like helpful resources for the different things that I've talked about. So if you're interested in like, okay, like I'm interested in like this QProxy replacement or other things like that, we have pages for all the different features. So like, I want to learn more about Silliam as a CNI, about kubeproxy replacement, about BGP or cluster mesh or host firewall. I want to learn more about Hubble. We can do that. We also have like created different pages for different industries, kind of talking to the challenges that a cloud provider or consulting company or financial services company.

56:55So it's not just like, here are the features, it's here are these features and how do they apply to your actual industry. And then the last part that we have is like these outcome pages, because once again, it's like you have these features but like companies aren't buying a service mesh they're buying a specific outcome like how do we do layer seven routing or how do we do zero trust networking or how do we do network automation how do we consolidate our networking tools and so like actually looking at those outcomes and so it's kind of like moving up the stack from here's the future to here's like the business value that we're getting and how do we do it for the industries but if you actually want to get hands-on what i recommend is going to like the getting started and going to the labs.

57:35And there's a lot of hands-on labs around Cilium. And so what these will actually allow you to do is to, you don't even have to set up your own Kubernetes cluster. It's set up for you. Cilium is installed. And it'll walk you through the different features. So basically one is like, okay, how do we install Cilium? And it'll walk you through that. The next one is, okay, like I need to do network policy. How does Cilium network policy work? And there's a really famous Star Wars demo. It's like, how do we blow up the Death Star? How do we protect the Death Star? So that's like a fun lab. And it's different, like hands-on labs, you're actually in a Kubernetes cluster and walking you through these different features, how they work and how do you actually apply them to your cluster.

58:10Yeah, so those are all great sources. Obviously, there's always GitHub. We have a Slack channel if you want to jump in there too. But I would recommend if you want to get hands-on with Celium, go to the labs. I know I like to actually be in a cluster and be able to do things or just read through some of the stuff. And then there's also the documentation pages there too. Awesome. Sounds pretty fully featured on that front. So yeah, cool. Well, Bill, awesome to have you on today. Thanks a lot for coming on. As we were talking about before recording, you've been doing a lot of traveling, so you've managed to find a slot on that schedule for this.

58:44So that's really appreciated. And no doubt we'll be following along and maybe catch up again in a couple of years or something. Yeah, that'd be great. All right. Thanks a lot. Yeah, thanks for having me.

59:02Thank you.