In short
The Changelog Podcast: Episode Summary
Podcast Title
- The Changelog: Software Development, Open Source
- A weekly news brief, deep technical interviews, and talk show focused on software development and open source.
Episode Title
NATS and the CNCF kerfuffle (Interview)
Episode Description
Derek Collison, creator of NATS and Co-founder & CEO of Synadia, joins the show to discuss the origins, design, and evolution of NATS, a high-performance, open-source messaging system for modern cloud-native systems, and the tensions between Synadia and the CNCF regarding the future of the project.
---
Key Points from the Episode
Introduction to NATS
- NATS was developed as a solution for messaging in distributed systems, particularly at VMware's Cloud Foundry.
- Derek Collison shares how NATS was designed to effectively handle command and control telemetry, becoming a foundational element for Cloud Foundry.
Evolution of NATS
- Initially created in 2010, NATS aimed to provide a simple and reliable messaging system.
- Rewritten in Go for performance and deployment ease, it has become widely adopted in distributed systems.
Comparison with Other Systems
- NATS is positioned differently from traditional messaging systems like RabbitMQ, emphasizing simplicity and location independence.
- Derek compares NATS to Tailscale regarding its aim to simplify connectivity in distributed systems.
The CNCF Relationship and Challenges
- Derek discusses the founding of the CNCF and NATS' entry as an incubating project, noting the initial optimism about collaboration and growth.
- Tensions arose concerning the CNCF's graduation criteria focusing on diversity of contributions, which became a major obstacle for NATS.
License and Governance Discussions
- Synadia's consideration of a potential relicense of NATS to BSL (Business Source License) sparked significant controversy.
- Derek emphasizes the intent behind seeking a relicense: to ensure sustainability and commitment to open source while addressing vendor lock-in concerns.
Community Reactions
- The episode highlights emotional responses from the community regarding licensing changes and the perception of Synadia’s intentions.
- Derek acknowledges that while some reactions stem from legitimate concerns, they often conflate distinct issues, complicating the narrative.
Synadia's Future Plans
- Synadia plans to maintain a commitment to NATS as an open-source project while exploring commercial opportunities through additional services.
- Derek expresses a desire for a dialogue within the ecosystem about the sustainability of open-source projects, emphasizing that production-level software requires funding and investment.
Final Thoughts
- Derek reiterates the importance of understanding the nuances behind the decisions made regarding NATS and the CNCF.
- He is hopeful for constructive discussions that could lead to improved practices and relationships in the future.
---
Key Takeaways
- Nuanced Understanding: The conversation highlights the complex relationships between open-source projects, companies, and foundations like the CNCF.
- Community Engagement: There is a need for clearer communication and engagement with the community to foster understanding and support.
- Sustainability in Open Source: The episode raises questions about the sustainability of open source and the role of commercial entities in supporting essential projects.
- Desire for Dialogue: Derek emphasizes a commitment to ongoing discussions about the future of NATS and open-source governance.
---
Conclusion This episode offers a comprehensive narrative around NATS, its origins, evolution, and the recent tensions with the CNCF. It provides insight into the complexities of managing open-source projects while balancing community expectations and commercial viability.
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Transcript
Automatic transcript. May contain errors.0:28Well friends, this is your favorite podcast, The Changelog. to create Nats, where it came from, how they eventually decided to form a company around Nats called Cynadia to offer this to enterprises and more. Derek also shares the behind the scenes of the recent kerfuffle between Nats and CNCF. A massive thank you to our friends over at fly.io, our partners, our sponsors, our friends, and the home of changelog.com. You can launch your app in five minutes at fly.io okay let's nets
1:10well friends i'm here with terrence lee talking about what's coming for the next generation of heroku they're calling this next gen fur chance one of the biggest moves for fur in this next generation of heroku it's being built on open standards and cloud native what can you share about this journey. If you look at the last half a decade or so, like there's been a lot that's changed in the industry. A lot of the 12 factorisms that have been popularized and are well accepted, even outside the Ruby community, are things that are, I think, table stakes for building modern applications, right? And so being able to take all those things from kind of 10, 14 years ago, being able to revisit and be like, okay, we helped popularize a lot of these things.
1:51We now don't need to be our own island of this stuff. And it's just better to be part of the broader ecosystem. Like you said, since Heroku's existence, there's been people who've been trying to rebuild Heroku. I feel like there's a good Kelsey quote, where we can stop trying to rebuild Heroku. It's like people keep trying to build their own version of Heroku internally at their own company, let alone the public offerings out there. I feel like Heroku has been the gold standard. Yeah. I mean, I think it's the gold standard because there's a thing that Heroku's hit, this like piece of magic around developer experience, but giving you enough flexibility and power to do what you need to do.
2:26Okay, so part of FIR and this next generation for Roku is adding support for.NET. What can you share about that? Why.NET and why now? I think if you look at.NET over the last decade, it's changed a lot. .NET is known for being this Windows only platform. You have WinForms, use it to build Windows stuff, double IS, and it's moved well beyond that over the last decade. You can build.NET on Linux, on Mac. There's this whole cross-platform open source ecosystem, and it's become this juggernaut of an ecosystem around it. And we've gotten this ask to support.NET for a long time, and it isn't a new ask.
3:04And regardless of our support of it, people have been running.NET on Heroku in production today. There's been a mono build pack since the early days when you couldn't run.NET on Linux, and now with.NET Core, the fact that it's cross-platform, this.NET Core build pack that people are using to run their apps on Heroku. The kind of shift now is to take it from that to a first-class citizen. And so what that means for Heroku is we have this languages team. We're now staffing someone to basically live, breathe, and eat being a.NET person, right? Someone from the community that we plucked to be this person to provide that day zero support for the language and runtimes that you expect and like we have for all of our languages, right?
3:42To answer your support and deal with all those things when you open support tickets on Heroku and kind of all the documentation that you expect for having quality language support in the platform. In addition to that, one of the things that it means to be first class is that when we are building out new features and things, it is now one of the languages as part of this ecosystem that we're gonna test and make sure runs smoothly, right? So you can get this kind of end-to-end experience. You can go to Dev Center, there's a.NET icon to find all the.NET documentation, take your app, create a new Heroku app, run Git push Heroku main, and you're off to the races.
4:16So with the coming release of Fur and this next generation of Heroku, .NET is officially a first class language on the platform, dedicated support, dedicated documentation, all the things. If you haven't yet, go to heroku.com slash changelogpodcast and get excited about what's to come for Heroku. Once again, heroku.com slash changelogpodcast.
5:13Today we're joined by Derek Collison from Nats. and Cinedia. Derek, you were on one of my favorite episodes of GoTime years ago. Long time ago. Pandemic time. Challenges of distributed messaging systems. Yes, GoTime is no longer being produced, but the feed is still out there. The episodes are still out there. Please go listen to that. I learned so much from you about these things. So welcome to the changelog. I appreciate you guys having me on. Thank you. Literally years in the making. Jared, do you remember going to kubcon a while back i do circus that's right the circus and she was a chicago a couple years back we met the senedia folks and the nats folks i think it was not quite early days for senedia but like it was early-ish for announcements maybe even marketing derrick you can kind of speak to this but yeah you remember that circus and meeting those folks i do yeah i don't know if i was there what year was it because senedia was formed at the end of 2017 right on the on the heels of Apsera.
6:14Um, but I don't remember, I don't think I was at the one in Chicago. Many years after that to 2023. Okay. So not too far back yet. Yeah. Till into 2023. Yeah. You weren't there, but we had a few friends there that were listeners as well. And we became to know each other. So yeah. Been nice knowing you all. I've known about Nats for a very long time. When did that start? I mean, it's been around forever now, right? Yeah, Nets was kicked off as part of a system that I was building at the time at VMware called Cloud Foundry, which was my spin on Heroku for the enterprise back in 2010, I guess. And at the time, I had come off of time at Google, spent about eight years there.
7:01But prior to that, I was at a company called Tipco, which before that was called Technicron. And we had been designing lots of messaging systems that then were used to scale up distributed systems for customers and clients. So I just had a natural way of building systems like command and control, telemetry, eventing, using these types of constructs. But Tipco was closed source, right? And so we couldn't use that. So we reached out to Alexis Richardson, who was running a company that was kind of in charge of RabbitMQ, which at the time was kind of, you know, the major open source player from a messaging construct.
7:37And, you know, RabbitMQ is still part of, I think, VMware slash Broadcom now, part of Cloud Foundry or whatever that has turned into be. But at the time, it was more of an enterprise type of technology. What I mean by that is, is that if you asked it to do something, it would bend over backwards trying to do it for you, even at the detriment of everyone else. And, you know, we had some challenges there, which led to the creation of Nats, which was, you know, kind of like a dial tone and think of it like the electricity in your house or your condo or whatever. I can plug in a really bad blender and I might blow out something in my own place, but hopefully it doesn't take out the electric grid for the whole neighborhood.
8:20And so Nats was born on a very simple principle of just do a couple things very, very well. And yeah, it was the proverbial weekend project. So I left on a Friday and Monday, you know, we plugged it in. And believe it or not, applications that were written for that initial version, you know, the protocol that speaks between the clients and servers still run today. You know, so that you can just fire up something from the Cloud Foundry days and just run it against a modern Net server and it just works. So we've tried very, very hard and we're not perfect by any stretch, but we tried really, really hard to make sure we were always backward compatible with the protocol.
8:58And because we knew the protocol, even though it was designed very, very quickly in only a couple of days, that was kind of the really big thing from a technology standpoint there. I'm not a distributed systems person, but NATS feels a lot to me like Tailscale, but a different layer. Is that how you would describe it to somebody who doesn't build distributed systems or have sort of the depth and knowledge you might have? Yeah, I'm a huge fan of Tailscale and, you know, the folks that were either at Google or involved in the Go programming language that are over there. But it's an overlay on top of the constructs that we still know, which is IP-based addressing.
9:41Almost everything is point to point. Most, I don't know about Tailscale, but most cloud providers write trunk UDP broadcasts or multicasts. And so it's a point-to-point location-dependent type of a system that now has surfaced up over a very secure overlay with WireGuard. What NAS was trying to do was kind of change the notion of what we call intelligent connectivity. And specifically when folks that are listening say, well, what does that mean? It's really fairly simple from an abstract standpoint, which is everything is location-independent. Right. And so one might say, oh, well, things are kind of location independent today.
10:18But I would argue that there's a lot of unnatural apps going on below the covers to get that to appear that way. Load balancers, GSLB, anycast, DNS tricks. Right. So the biggest first one is location independence. And the second one is instead of one to one request reply is the dominant pattern. HDP, that's all you can kind of do. SANS, SSE and WebSockets. we are end to end and both push and pull. And the push becomes very, very interesting for certain use cases where instead of keep asking, hey, have you updated your temperature? Have you updated your temperature? Just tell me whenever you update your temperature, right?
10:53And for scalability and distributed systems, just those primitives from the connective layer become pretty powerful, right, at scale. So Nats really took off, right? I mean, huge adoption. Can you tell a story? Yeah. Yeah, so, I mean, Nats was built specifically to power those subsystems in Cloud Foundry, right? And I had no desire to do anything more than that. And so it was for at least probably the first 18 months of its lifetime specifically geared toward just supporting what we needed in Cloud Foundry from command and control, telemetry, eventing, things like that. But when we started the company, AppSera, which was trying to do a redo of Cloud Foundry on some of the things I felt I didn't spend enough time and energy on, security, governance, federation, all these different types of things, we rewrote NATs in Go.
11:45So that was originally written in Ruby. That's what CloudFactory was written in. Still love the language. Not great to deploy things into production that are Ruby-based, although I know DHH has done an amazing job of trying to simplify that. But back in the day, it was, you know, there was a lot of moving pieces to deploy this thing. And we were starting to hit some performance issues, right, with very, very large systems needing more performance there. So we had, at least at the time, we were looking at either Node.js, JavaScript, or Go, which was very brand new. I think when I started playing with it, it was 0.52.
12:24And I felt that the speed advantage with the concurrency, the I.O. mechanisms that they had, but mostly the fact that you could build pretty much a static binary, meaning shipping to production was just move one binary, was the direction that we took. And it was a great direction. I mean, I think Go has grown up and, you know, there's people that like it and people maybe that don't like it, which is totally fair. But we've gotten a tremendous amount of mileage out of it. And at the time, looking back, it was the absolute right decision. Now, going forward, when we're looking at massive, massive extreme scales that some of our customers are pushing us on, we're starting to bump into some things in Go that we need to address.
13:07And whenever I'm in a language and we start to do unnatural acts to get around certain things within the language, that's when we kind of say, all right, let's take a step back. And are we at another inflection point? And we're not there right now, but with the introduction of Jetstream, which is the data persistence mechanism that Nats uses that was added on, you know, about four, four and a half, five years ago, I guess, we're starting to see a lot of friction with the way Go does DiskIO and things like that. And so we're trying to figure out what makes the most sense as customers are saying, hey, remember when I said a million was good, then I said 10 million was good.
13:47now I need a billion, you know what I mean? And, you know, could you do that in the next, you know, two years type of stuff? And those are great problems to have because they're hard and they're fun to work on. But it challenges you to think outside the box sometimes. What are you looking at? You know, I don't do it anymore. I'm sorry to say that I've dropped the ball, but I used to learn a new language every year, you know, and I would just write a Nats client for it to learn it. And so I learned Rust and there's certain things in Rust that were really, really attractive. But to be frank, I just didn't enjoy programming in it.
14:24When you were doing a low level library, if you were consuming other things and building like an upper level app, it was kind of a composite of lots of different things together, especially the packages or crates were really, really well done. It felt really, really nice and clean. But I was at a low level library trying to do some things and we got it. and it's one of our most popular clients. I like, again, I'm older probably than the majority of the demographic that's listening to the podcast, but I like languages that I can hold most of it in my head. You know, I've never been a big IDE person.
14:58I still program in Emacs. I do use AI and I ask, you know, Claude or ChatTBT questions and things like that, but I don't have that co-pilot windsurfer code mentality yet, but I do like Zig quite a bit. Um, but Zig right now, you know, Andrew was trying to take on some really gnarly problems, which I have the utmost respect for, especially the colorless async. Um, and that kind of tied in a little bit to event loop structures and they kind of got pulled out. We need some sort of direction there before we finalize a Zig client. Uh, but I did write one. I have one that's kind of uses threads and does some interesting things.
15:41And then Joran from Tiger Vito, I've known him for a while. Full disclosure, I'm a minor investor in their company. Love distributed. All things distributed, I think, was their conference. And really been watching them and Bunn and, of course, Mitchell with Ghosty on what's possible. But each one of those event loops and IOU ring interactions is all kind of a little bit snowflake. Not exactly. but I think Andrew is now thinking through a way to uplift the abstraction of IO that could tie into an event loop as something similar to allocators in the language where you have to pass it into a package, right, or to a constructor, say, hey, if you need to use IO, please use this essentially interface, right?
16:25And so I'm really interested to see where that lands. But I like Zig. Believe it or not, I like Mojo, even though I'm not a Python person. I just am a huge fan of Chris Lattner's and what he's trying to do there. I think it is impressive. But Zigg's probably number one right now on my mind. But, again, we need a little bit more direction, not from the server perspective, but if we wanted to do a client on how do we interrupt between all the different event loops, you know, whether they reincarnate the one that they have in the language, but then Mitchell's with XEB, I think. And then, of course, Bund has one that's, I think, a pool from Tiger Beetles, and Tiger Beetle has one.
17:04So that'll be good when we get there. Well, completely different problem space, but you like Chris Lattner. The Ladybird folks are probably picking up Swift from C++ to Swift. Have you considered Swift? I haven't. We built a Swift client. I wasn't involved in it. Again, I drop the ball on my learning new language every year. I actually did program a little bit Swift. You know, I think the interesting part about where Zig is is that, you know, we can control direct IO, meaning we could go through IU Ring on Linux if it's, you know, there applicable or whatever. And it allows us also to control more precise layouts of the data structures and, of course, memory.
17:44Right. And so anyone who's been programming Go for a long time and has a tremendous amount of investment in a big code base. Right. You start to see things pop up like, oh, avoid JSON at all costs because the built in JSON marshallers, you know, user reflection can use lots of memory. The network IO and the disk IO subsystems in Go are very different. And the disk IO subsystems are pretty much a native thread that kicks back into your Go routine that says, OK, it completed, the right completed or whatever. And I think what we need is a lot more fine control over those things. So what I've been looking at is the adoption or the ability to introduce IO Uring in a meaningful state.
18:28I think Zig's done a pretty good job there, maybe not within the language per se, but with folks that use it, Tiger Beetle and Bun come to mind. And so that one's kind of a big one. The data structure layout, the garbage collection and taking control over disk IO, right? Which, you know, Jetstream, we've got folks that are coming in and says, oh, yeah, I need to do a million stored messages per second replicated three ways. And then an async mirror over in a different region and a different cloud provider type stuff. So we can abuse the disk, the storage subsystems pretty well, and we might be getting close to kind of everything we can accomplish with Go.
19:14I'm not saying that that's definitive, but it's starting to feel that way as, you know, we're starting to do not unnatural acts, but we're starting to do a lot of things inside of the server to get around how Go actually marries concurrency with Disco. So when you said some of your customers need like a billion, is that like message passes per second? Or like, what does that mean a billion? A billion what? Some need, you know, a lot of messages per second. Nats has always been pretty performant. Last time I checked, just raw speed, a single server could do about 20 million per second messages sustained.
19:51All of our clients can't do that, right? They have more of the logic, but a raw just shove it in there. But when we say billions of things, it's usually stored messages and then subject cardinality. So in event sourcing and event streaming, right, every single message has a unique subject. And what we found was is that people are starting to treat the stream of messages. And we have materialized views like key value and object store on top of them, but they're starting to treat them like a subject-based addressing lookup thing, meaning that they use subjects all the time to get messages out or filter messages or do some other operation.
20:27And so you could imagine trying to figure out an index that looks like a traditional data structure that anyone who's done a comp side 200, 300 level class would understand. But overlay subject semantics with wildcards and partial wildcards is a little bit tricky, if that makes sense. Yeah. Like a giant hash table in the sky that also has some very gnarly semantics that you can use. Yeah. And to be honest with you, it's called the S tree and it's an adaptive Radex tree with, you know, path compression and expansion. But then it has an overlay where we can say, hey, you just stored a billion subjects in here, hopefully in an efficient way.
21:10Pick out and call me back on everything that matches Foo.star. You know, in other words, or a wild card. And so we have that, but that lives in memory. And so if you have, you know, 10 million messages, but only a thousand subjects, you're fine. You'll never even see this. But we're starting to see people where they've got billions upon billions of messages and everyone has a unique subject. Next thing you know, your server is taking up 30 gig just for the indexing on the subject-based addressing layer. And so that's one of the big challenges that the team is looking into solving for 2.12, the next version of the server.
21:44Well, it's cool that you still have your head down in the technical trenches of this project. Cause it seems like from what I've seen from you all and your name signed on things recently, it's all been higher level business and maybe technical legalese, not, uh, not technical technicals. But it's cool that you're still, you know, close to the metal of this project and not just the business guy now. Yeah. I mean, I, I, I think I definitely, you know, wear multiple hats obviously. And as of late, I've been, you know, more concerned about upper level discussions around Synadia and NADS and the CNCF, obviously, which we can talk about, but also the business side of Synadia.
22:25But I identify with builders. You know, I feel that, you know, working this hard, you want to get to a place where you get to do the stuff you like to do, at least part of the time. Right. And I still love doing that. You know, am I as good as other folks in the company? Probably not anymore. but I still love to reason about things, think hard about things, you know, tackle really, really hard problems. That's kind of my relaxing motion in terms of work, if that makes sense. So if, uh, if Nats was roughly around 2010 ish, uh, era, you really didn't plan to do anything with it beyond its original case.
23:04At what point did you say, well, you know what, we should be a company. We should create Canadian. That was 2017 based on what you said before. How did you go from there in 2010 to there in 2017? Yeah, it's an interesting story. So one of the things that folks might not know is that when we were doing Cloud Foundry, VMware had never released anything that was open source. And my pitch to Paul Maritz at the time, who was the CEO who was running VMware, I said, I really think we'll have more bang for the buck here if this is released as open source. And so when I did Nats, I said, I'm not going to keep this closed.
23:42So I actually just opened it as MIT. Right. And then we did. I can't remember what license we put on Cloud Foundry, but it was open source, which was a really big deal. So when we went to AppSera, we did the rewrite and go. And we started to contemplate whether or not this could be something bigger than just, you know, an OSS project. I mean, it was powering all of the AppSera stuff, but similar to Cloud Foundry, right? Telemetry, command and control and eventing. And so as we looked at AppSera, and AppSera was doing a commercial version of Cloud Foundry++ to be very, very simplistic about it.
24:20But as soon as Kubernetes came out, you know, certain things Kubernetes did that people like, certain things that the Absera continuum platform did, they liked, you know. So it wasn't a technical battle per se. It was where is the consumer bias going? And Kubernetes, because no individual stakeholder needed to make direct money off of it, drove the consumer bias to zero. And so our ACV started declining. And so I called an emergency board meeting and said, hey, you know, I see the writing on the wall and all I can tell you for the 30 plus million dollars left in the bank over the next four years is that we can tread water, but I don't think it'll change the outcome.
24:59And that I think pulling it into Ericsson and doing something constructive with some of the internal projects that I know they were working on made better sense. At that point in time, there were certain people that were going to leave, certain people that were going to stay with Ericsson. And then there was Nats, the team that was working on Nats in the middle. And at the time, Ericsson said, you know, we don't necessarily have any need for Nats. It's open source. We can use it. But, you know, if you want to do something with it, we can. And a bunch of folks internal prior to that decision making process had already kind of approached me saying, hey, we might be able to build a business out of this.
25:38under the assumptions that distributed systems will continue to have more moving pieces, but more importantly, that these pieces are going to start to be stretched out. So stretched out across regions, across multiple cloud providers, but more importantly, out to this at the time, new thing called edge. And the bet that we made was that edge would dominate interaction models and would dwarf cloud within probably a decade. And that cloud's not going anywhere. Don't misread that. but that cloud would become the new mainframe, but all the interactions would be at the edge. And the rules are very, very different.
26:14So as many rules changed as we went from data centers into cloud and we had the Roku guys, Adam Wright, and the, was it the 12 things? 12 factor apps. 12 factor apps. And then of course, cloud native architecture, of course, then the CNC app, all of those things, that step function from data centers to that, I believe will be even bigger stepping from cloud into edge where it's not batteries included. You can't bring all the stuff that the cloud is offering and you have to kind of not think differently about the what, meaning you're still drawing the same triangles and squares on a whiteboard as you're architecting, but the how would be radically different.
26:50And so that's when we made the leap to create Synadium at the end of 2017, based around a foundation that modern distributed systems will look very different than they do today. And they will be pushed very hard in terms of dynamic agility. They can change all the time. You know, again, I probably am further along career-wise than some folks on the call. But it used to be that when a system was set up like this in the 90s, early 90s, no one touched it. I mean, I could walk into any financial institution on Wall Street and I could go into their server and point exactly where our stuff was running.
Read the full transcript
27:24And it never deviated from that. Right. And that's very different than what I think a lot of the audience probably thinks now, where it's like, I don't know, somewhere in AWS East, too. You know what I mean? But I have no clue what machine it's running on. And so our thought process and our priors were that that type of step change is going to accelerate with Edge and that the current tools of doing everything with like HTTP or HTTP derivatives just simply won't be able to keep up in our opinion. How has that bet played out? um very very well what i mean by that is is that in the 90s with tipco i was on the engineering side i wasn't in an executive role or had any type of position around pricing or anything like that but just from an engineering perspective right we cared deeply about interacting with these customers and and such like that when we watched what was happening with edge and how fast it was moving and how different the rules were.
28:24At Tipco, we had to educate the market. And I remember how painful that was. A lot of the sales cycles were 80 % education. We need to explain to you why we think you are going to have a problem in the near future that we think we might only be able to solve. In this world, which was good for Synadia was people were coming to us going, hey, I tried to pull Kubernetes and JVM-like stuff like Kafka and put it into an edge thing, and it just didn't work for me. And I need to think differently about it. Can you help us? And so when we started Cinedia, we wanted to start a company based on if you're running a production and it's critical to what you're doing, which luckily our stuff is critical to every customer that we have.
29:06I mean, there's no coffee breaks. If it goes down, they will light my phone up. That was the opportunity we were looking for in terms of starting a company. And we didn't have to educate the market. They were already in pain to start the company and that they would want a commercial agreement with us to wring our neck if something went wrong. And I think that's a really good way to start a company. It's not the best way to scale it. So you need a backup plan as you evolve. But to start it, I think that's a really good place for folks to start, especially if it's based around OSS.
29:51Well, friends, I'm here with David Hsu, CEO of Retool. David, I want to talk about awareness beyond Silicon Valley. Retool has a great presence and a great awareness inside Silicon Valley. But what about beyond? What's really cool is I think we've done a really good job of building awareness inside of Silicon Valley. And so when you look at customers that use Retool, pretty much every big company in Silicon Valley, about 1 ,000 people today, now uses Retool and builds internal apps via Retool. So that's really awesome, and I'm really proud of the progress we've made there. But I think the larger opportunity for us, actually, is outside of Silicon Valley.
30:26When you think about, for example, the Kroger's of the world, the Coca-Cola's of the world, many of them are customers already today. But I think we haven't done as good as a job building awareness, if you will, around the developers and all these companies. And that, to me, is where the opportunity lies, because so much of these companies run on software. Software is so important. If you think about Coca-Cola, for example, Coca-Cola has not really gotten any cheaper to manufacture in the last 10 or 20 years. Instead, the reason why Coca-Cola is doing well as a company is because they are getting more productive by better software.
30:57And so every company needs to become a software company, and Retool lets you do that. So Coke is a big company, but the principle rings true. become more efficient by using better software there you go well if you're beyond silicon valley raise your hand we want to hear from you i want to tell david we reach people beyond silicon valley that we're raising the awareness of what retool is and what retool does and some big announcements coming soon here on changelog which is cool well if you haven't yet go to retool.com get a demo try it out for free all that good stuff again retool.com
31:37so oss should we open the can of worms around oss most recently there's been a i don't know you call it a skirmish a disagreement kerfuffle a kerfuffle between cncf and yourself zanadia specifically around nats and its involvement in cncf and potential relicense of course we've been here many times in the world of open source and business and licensing. It's something that everyone's trying to figure out. And some people have or have not, and then other people think they have and then they have not. And then things change and people are offended. And so there's been a lot of, I don't know if maybe drama is just the best word around the business and the project.
32:21Maybe start with the CNCF and Nats' relationship to it prior to the most recent blog posts and press releases? Yeah. So what the audience might not know is that the CNCF was kicked off at a large meeting at the Switch Data Center in Las Vegas, of which I attended. And I was actually part of the founding governing board. So a lot of people might think, oh, Derek's role with the Nats project and Nats being a project in the CNCF, that's kind of the totality of it. But that's not the case, right? And even early on, we were looking at what value CNCF could bring to the ecosystem from a consumption perspective, the users to the people that are the companies that want to utilize that.
33:05And then, of course, the project themselves. And then for me, it was like the companies that are driving said projects. In the case of something like Kubernetes, where it's, you know, all of the big boys, so to speak, there's not a lot that the CNCF needs to do that, right? It's kind of taken on a life of its own, and it's marching down a path. What we started to realize is that, you know, as time goes by, because this was, I think, you know, over a decade ago, that both the CNCF and projects can evolve. And where we were at was looking at the question, are we the best fit for CNCF? Right. And no fault of CNCF per se.
33:49But it just felt pretty clear to us that certain projects were at this tier and other projects were way down here. And we kind of were in the latter category. In no way were we attempting to re-license the whole code base. I mean, one, you can't do it because once it's released, it's AP2, it's there. That will never change. But the CNCF and early discussions, because we said, hey, would you be open to letting the project leave if the landing spot made sense? Meaning we could do a joint statement where it's like, no, it's landing in this new foundation made up of lots of big users and customers or production uses of the Nats ecosystem, commitment to AP2, X by Z.
34:36And during those early conversations, which started in February, we were asked, hey, are you considering a license change? And we're very well aware that we can fork and do a Cineadio version enterprise server and XYZ. And so we said we're considering it just for the server, that we might do like a VSL. And at the time, the VSL, for us, it wasn't translated well out into the media. But the reason that we were considering it was we felt that it was the best show of commitment to both OSS and to our customers. And I'll explain what I mean by that. the BSL has usage clauses that you have to define, you know, when you release the software.
35:16And one of them is there's a period of time that after which it converts back to whatever language you pick, which we would pick AP2. And it doesn't matter what Synedia does, the Linux Foundation does, CNCF does. That is a legal contract that once we release, let's say, a server with, you know, the BSL license after two years, it's converted to AP2, even if we don't actually update the copyrights or update the license or whatever. And so we felt that from an OSS ecosystem. And again, we've gotten lots of additional information, lots of additional feedback. Most of it constructive, some of it not.
35:52But, you know, and we've been listening. But our initial thought process as we were going down this path was, hey, this signals really strongly that we're not going to hold anything back. It will always kind of revert back to generic drug prices, right? Some of the good parts of what the patent system originally was trying to do early on. I'm not claiming that the patent system is necessarily good these days. It's a necessary evil, right? In addition, if you would imagine a two-year BSL window, and let's say Jared wanted to be a customer of Synadia, right? And, you know, customers are worried about vendor lock-in, price gouging, all those normal things.
36:28You could say, I want to do a two-year contract, so I have predictable pricing. And at the end of that, that all stuff that I'm paying for converts to AP2, and I can now use it for free. Or I could say, oh, I like the next version, too. I want to use that and re-up, right, and do that. So I'm not saying that it transpired the way we wanted it to, but I think people's thought process that we're just evil and we're just trying to price gouge and be greedy and stuff like that, I don't feel that that necessarily is correct, at least as we were discussing internally. We care deeply about our commitment to open source.
37:03But we are starting to see a very disturbing trend where, you know, customers that, you know, everyone would recognize they're in the Fortune 50 that are using NATS to power production level services or functions or products, not only have never reached out for any type of commercial agreement with us, but actually have policies that say you're in the CNCF and you're incubating and graduating. We're not we're not paying for it, period. To me, I thought that that was at least worth a discussion, you know, and it might not be a discussion between myself or Cineadio or the CNCF, but the broader ecosystem, I think, needs to look at, do we want open source projects that drive value, that can be critical, that are driven by a small group of individuals or a company, right, that's trying to make, you know, business out of said value.
37:57And, you know, Synadia has lots of different revenue buckets, so I don't want listeners to think that this is our only drive. But I was at least concerned enough to say, are we signaling to the ecosystem, the broader ecosystem outside of us, that we only want projects that look like Kubernetes? We don't want any projects that are being driven by, you know, a single company. And I thought that was worth a discussion. For sure. How much time before the blog post that I think broke the news, so to speak, from the CNCF directly, how much behind the scenes conversation was going on to sort of illustrate your points and your concerns?
38:37Like, was there a meeting? Okay, so this was a surprise? Yeah. So we were not, we had gotten to a point where we were trying to assert the fact that, you know, we wanted them to consider letting us leave and they were stuck on some of the trademark stuff. Right. And so for the listeners, you know, who haven't been in this, when you're doing negotiations behind the scenes, you posture up and you do all kinds of stuff and then you figure out when you can't move the stakes anymore and then you make a decision. Right. And our decision was always going to be, hey, if they dig in, even if our legal team say that we could potentially win this, we have to consider the damage to the ecosystem and whether or not it's worth it.
39:22And we had already probably decided that they were just going to dig their heels in and say, nope, you absolutely can't do that. If you want to go down that, it'll have to be a legal recourse that we were going to say, OK, we'll figure out something else. But they I can't speak on their behalf and I'm not trying to. And again, I'm extremely encouraged by my talks with the Linux Foundation and some of the ongoing stuff with the CNCF and that mutual statement. Again, everybody's had a hand in that pie, so it kind of gets diluted down. But I can tell you from my individual discussions with folks, especially at the Linux Foundation, I'm very encouraged with them saying, okay, let's see if we can do better here, which is all I really kind of wanted at the time.
40:06But I did feel that, you know, we could be a good steward. And if we could say, hey, here's what we're committing to. And that is commensurate with kind of what the CNCF, you know, is doing around certain projects. I thought it was worth the discussion to have. Now, a lot of people don't agree with that and have very, you know, emotional responses to that. and although I don't necessarily agree with it I understand that people will fight very hard for consumption being zero but production is never zero someone has to pay for it right I mean you know unless it's just something whether it's time or money or both and so we were concerned enough to say hey let's have a discussion about that the main concern I think if I understood correctly around even making this a concern for you was the fact that Cinedia was having challenges securing certain contracts because of its incubation status of NAT, which Cinedia is built on top of as the commercialized version of it.
41:08Is that the summary of like your initial concern? No, I mean, we didn't graduate and that's a public record and it was mostly around diversity of the contributors to the server, right? So GitHub has all the stats so you can look at, you know, all the people that have contributed to it. And I didn't agree that that was super important for the success of a project long-term, right? And so I'm not saying the CNCF is wrong and I'm right. I'm just saying we had a differing opinion of what successful projects look like. And, you know, there's some that I've driven in my career that now have been around for 30 plus years.
41:47Two of I haven't been around for 20 plus years. So I kind of get what it is, but it doesn't have to only look like Kubernetes. Right. In my opinion. And so that was kind of the first one where I was like, hmm, you know, the CNCF is evolving where they bring in lots of different projects. And that's a daunting task of them having to do all that. And hats off to them on that. But as you watched how things moved through the channels and got towards graduation, there was some friction on does the project look like what the CNCF wants it to? And again, totally their right to do that. And then also I was noticing little perturbations in companies doing strange things to try to drive a business model where they were the main contributor, right?
42:34Their company was the main contributor driving said project outside of METS. And we were watching that pretty closely. We don't get too many people worrying about the CNCF or our graduation status. Every once in a while we do. It's like, hey, why haven't you graduated? And we just say, oh, because, you know, the CNCF's graduation criteria looked like this. This is how kind of the server looks like. And it usually is that quick of a conversation, like, you know, 15, 20 seconds max with customers every once in a while. I think I've gotten two of those in the last six months, you know, and it was just a, hey, why haven't you graduated?
43:07So it's not that per se. It was, as the CNCF evolves, are all projects good fits? And is the only answer, you have to fork it away and then we'll kill it. And I don't feel like that should be the only answer. How about that? Right. There's so much brand equity in the NATS name. Obviously, you founded it. I created the NATS name driving home from VMware on Friday, I remember. And everyone laughs and the marketing came up with a funny term. But I remember just shaking my head driving home going, geez, I don't want to have to create another NAT, a Tipco server. I was like, not another Tipco server.
43:46I just kept saying that to myself driving home, but I said, I feel like I have to. Well, at the end, not another Tipco server is exactly how it got named. Yeah. Yeah. So when you first joined or you first started incubating, is that the entry level term there? With NATS, were you aware of the CNCF's desire or were there requirements like this project will move towards a non-single corporate backer model or whatever language would fit? Like, did you know coming into it that they wanted you to kind of change the format or the org structure of NATS? I don't think that was discussed at that level early on.
44:30So, you know, we were at the switch kickoff meeting. We saw some of the early projects. They wanted us to be one of the earliest projects. I was like hesitant. So I said, I want to be part of this, but I want to wait and see. And Google was doing similar things early on. They're like, we just want to kind of see how this is going to kind of pan out. But I think it was more from a sustainability, how many people are using it, what type of impact does it have both within the CNCF but then externally, were more of what I remember very early on of how do we classify graduated project. Now, as the TOC, that's the Technical Oversight Committee, right, keeps changing and new people come into that, they started changing, you know, the criteria.
45:11And so instead of kind of a vote and a gut instinct, you know, on a vote type stuff, they started formalizing some of the things that they felt needed to gate between, you know, sandbox to incubating to graduate. There might be some more levels in there. I'm not sure. And so at the time, though, we weren't aware of that. It was only kind of really solidified when we actually went through the graduation process. And again, that's all online, the back and forth between us and the TOC on trying to graduate. So was your failure to graduate then a shock to you? Or did you see it coming? I did not see it coming.
45:53And it took us a while. And again, it's all public record. It took us a while to figure out what's going on here. And it was specifically about diversity of code contributions to core pieces of the project. Now, again, CNCF, you know, I think they've done an amazing job with all of the different pieces that are flying at them. But if you look back historically, you could see certain projects that were like, well, how did that one graduate? You know, I mean, that looks a lot more similar to these types of projects than others. And I think they've slowly cleaned that up and kind of solidified that process and those criteria.
46:30It's just I didn't agree with the criteria. And again, I'm not saying I'm right and wrong by any stretch, but I did fundamentally disagree that not all projects— You must think you're right then, right? Like if you disagree fundamentally, then you got to think you're right. Well, in certain projects, right? And so, you know, could Kubernetes have been driven only by Google and gotten to where it was? No, but it has a very different role of kind of greasing the skids between cloud provider mobility. That was kind of some of the original impetus around Diane and the folks that built it, at least from me talking with Diane back in the day of why they were doing it.
47:07But I don't think saying that that project fits means that a project like NAPS doesn't fit. Right, but they disagree with that. Correct. And they're the bosses of the CNCF. So they get final say on that. Now, when your conversations began back in February, this is when you realized you were not going to graduate or you'd already kind of known it for a while. Like you had decided, well, we're not going to be a good fit anymore. Right. Yeah, we that was the graduation. I think it was in 2020 as well as the trademark fight. So this was more of a, hey, we just think we're not a good fit anymore. And let's figure out if there's a way where both the CNCF and Cineadio can message to the ecosystem that this is okay.
47:48But as long as there's a landing spot that holds true to what the CNCF wants, commitment to open source, things like that, would that be possible? And obviously the answer is no. But I still think it was worth a conversation. And that's where we went. So in February, you wanted to kind of slip out the back, Jack. It wasn't going to work out. And then those conversations began. And then they asked you if you were thinking about a relicense. And you said, yes, we're thinking about BSL. And then there's probably some more conversations between there. But then they decided to publish the blog post.
48:22Or what am I missing in the interim between? So I think, you know, we were very clear our commitment to AP2 and OSS. and that it would be a fork that we put BSL work into that then would, again, in our opinion, would signal a commitment to OSS because no matter what we did or said after that, two years, which was the number we had picked internally as we were going through that process, it would convert over. And so I was saying the licensing stuff is kind of separate. You know, we're talking specifically about the open source nature of NATS and whether or not it's a good fit and if it can can land somewhere else with both the blessing from the CNCF and the commitment from Cinedia that allows them to feel comfortable doing that they then started getting to the point where we were starting to figure out when the stakes would move or not move again you know just that and it's totally fine you know negotiating types of stuff and they put a hard stake in the ground nothing will ever leave the cncf will kill it instead nats cannot leave the cncf nothing can leave ever nothing ever including nats correct and that was my question too was like has it ever happened before which right right right no i appreciate the details behind the scenes here okay so it's like hotel california is that the one you can check it right and again i think that you know that's totally fine that that's their position um i still felt it warranted a conversation of, because that doesn't feel right to me.
49:52So if you look at like Grafana, right, had an interesting thing with Cortex and they were going to start driving more stuff, but it wasn't necessarily compatible with the CNCF, but they were still committed to the open source nature, you know, of what they were trying to do, but it wasn't compatible. And so the CNCF said nothing ever leaves. So you can fork it and then abandon this one. And that's what they had to do. And they and again, I'm not saying they're right or wrong, but when they purport that's a success story, I talk to other people and it doesn't feel as much of a success story. Right.
50:27Mimir started and that's what almost everyone on the Grafana side uses. And I think Cortex is mostly supported by AWS so that they can run it, not have to pay anything to Grafana. Right. Again, the consumption of zero is is a sense of entitlement that I also think is kind of interesting. But production side is never never zero. Right. Someone's got to pay. Yeah, it's a it's a hard position to be in whenever it almost knowing this, I think maybe this even exposes a different scenario where maybe you were not fully aware. Like Jerry was asking, like, were you aware of X, Y and Z when you first joined the CNCF?
51:02you incubated that whole process. And I think over time you get, you know, more and more privy to certain details that are truths about what it truly means to join the CNCF. And I'm not saying anything negative or bad, but not being able to leave, like coming in the door, not being able to leave with the thing you think about over a weekend and you name is, is challenging. Cause if you don't pass the bar of incubation and graduate and you're not a, I don't know what it means to not graduate. Does that mean you're like not? What does it mean to graduate even? Like what does that mean? Yeah, I don't get it all of it.
51:37Like does that mean you're not valuable? Does that, I don't understand the graduation statuses either. And I'm sure there's a lot of nuance too. Yeah, there is. And even after our TOC process around graduation where we did not graduate, there was a slight change that said incubating or graduated projects are considered mature and production ready. But within the grand scheme of things, someone from the outside, a user, right, who's saying, I'm just looking for some software that might solve my problem, you know, they see graduated as a higher level than incubating. And so they're like, why haven't you graduated?
52:11Because you've been here for a while. You started at incubating, you know, back in 2018, I think is when we officially joined, if I recall correctly. I might be off on that one. And so, yeah, I wanted a conversation. We had spent, you know, And I'm not saying that CNCF has not also worked to promote NATs, maybe not as much as Kubernetes and things like that in KubeCon or whatever, but they did do a part there. But we lived and breathed this 7x24x365 for almost 15 years now. We've put everything we can, and not just myself, it's just all the folks that were at Absera and then, of course, now at Cineadia.
52:52And not saying that what we thought was necessarily correct per se, but I find it interesting when people go, oh, you really don't care about the project. You know what I mean? Or you're just trying to do a money grab and gouge people for money at this time. That's a little unfortunate. I get it. And again, I know this whole topic is very emotional and I can appreciate that. But that's about as far from the truth as you could possibly get, you know, from what we were really trying to do. And again, if you looked at some of these companies and, you know, we talked to other customers of ours, big, big customers and said, oh, you know, these people that are kind of on the same level of you and the Fortune 50, they refuse to pay.
53:39and when that kind of came up they actually kind of changed their tone and said oh so we could actually get access to stuff that we're helping fund by having a commercial relationship with for a period of time two years before you know one conversation they called them the freeloaders and he and he's like it's not bob and vinny's pizza shop out of venice it's you know people on the fortune 50 that have the means to pay and this isn't a little internal developer tool it's a production, you know, level critical service, they actually went, oh, yeah, we actually kind of like that idea. And so from a partner and ecosystem standpoint, those conversations, we had lots of them because we lost control of the narrative, right?
54:18The CNCF wanted to all of a sudden move things into the court of public opinion. That's totally their right. We didn't have a heads up, though. So my phone started blowing up, you know, when that happened. But we kept business as usual. We kept doing commits, PRs, and we actually did a release the very next day, a full blown server release, which is a really big deal. It takes a lot of effort for us to kind of do all of those. So we've been trying to signal to the right folks looking at the right pieces of data that, nope, there's a disagreement here. We're going to try to work it out. But, you know, we're still committed to what we've been doing, you know, all along.
54:58You mentioned some of the emotional responses online, and I've read a few of them. One of which, which I'll summarize because I use more flowery language than I'll allow myself. But maybe you have a chance to answer this particular outsider perspective, which if you're just watching from the sidelines, I can't remember if we hit record yet or not when we start talking about armchair quarterbacks, but you're just sitting in your armchair quarterbacking. And what you see is Cinedia brings NATS into the CNCF as an incubating project, gets all the benefits, even though it sounds like you're getting all the benefits that you maybe thought you would get because Kubernetes maybe sucks a lot of the air out of the room.
55:37but gets all the benefits, the conferences, whatever, the clout that you get of being a CNCF project and then hangs out there for a while, decides they want to relicense and take their ball and go home. And that makes people upset. What do you say to that sentiment? Why is that wrong if it is? The reaction is not wrong. What I have an issue with is the conflation of facts within the reporting. And so it's easy to draw that at conclusion. So I don't get upset at people being very upset at us and more so directed directly at me. The question is, is how much energy do we want to put into trying to make sure the facts are correct?
56:22Right. The whole licensing and leaving the CNCF were conflated when they never should have been because they were never conflated in my initial conversations with unrelated conversation. They just asked me and said, hey, because this is a problem. We're seeing this and we don't know what to do about it yet, but we wanted to know. And I'm like, yeah, because I said we're starting to see, you know, said three letter companies use our stuff and get a lot of value and contribute nothing back. And I'll be very transparent. Yes, contributions comes in a lot of ways. But for a company like Senadia, a commercial agreement and funds is the only one we really care about.
56:58Yes, we care about issues and PRs and all that other stuff. But just being very, very transparent, they don't keep the lights on. And again, Senadia has plenty of money and we have lots of revenue opportunities and we're driving lots of those. But at the time, I was like, this to me feels wrong. It just doesn't feel correct that the consumer bias is being driven to zero for companies that it makes sense for them, right? Because if you sell a piece of hardware or service, it's kind of nice to have a guarantee that all the software you're going to use to augment that will always be free and keeps your margins where they are.
57:37I just think as an ecosystem, we're evolving, things are changing, You know, the CNCF has changed a lot, you know, that we still need to kind of think through some of these and figure out what do we really want as a global ecosystem around this? And charity is not a business model, right? So the whole notion of, oh, you can do donations, that's not a business model. And so what this lesson has taught me is, is that the biggest challenge, obviously, is in a company that develops OSS software that wants to make a business off of that software, specifically as software, not necessarily as a service, right?
58:16Service changes the consumer bias, right? So if it's something physical or somebody's running something for you, your natural bias is, oh, I have to pay for that. But if I give you a piece of software that we spent over 30 some million dollars developing over the last decade or so, you start to see this notion of entitlement. Oh, I deserve this for free and I don't have to give anything back. And we even have some people that just like bark orders at us like you need to do this in the system. And we're like, OK, that's interesting. You're not a customer. We don't even know if you use it. And everyone has these problems, right?
58:50All the OSS projects have these problems. But for what we were looking at, I was like, that doesn't feel right to me. And I wanted to reach out and open a dialogue and have conversations that would hopefully spur some constructive dialogue. But just for the record, we never were abandoning open source. We were never going to blanket re-license everything. That's just simply conflation of facts in the media and then people reacting to them and then that being amplified quite a bit over the social media channels. So that's the – I have no problem with people being emotional about it. But the way it was presented and the fact that we not only lost control of the narrative, but then all of these articles never reached out to us or me at any point in time and said, hey, what's your side of the story?
59:42You know what I mean? And I was like, hmm, that's interesting. I even have one now where someone pointed me to a YouTube video where the guy just, you know, F-bombing me and trashing me, you know, left and right. Oh, my gosh. And I'm like, I don't know who this person is. I've never met them. They never talked to us, you know. And I asked the whole team. I said, has anyone interacted with this person? They're like, nope. You know, they're just, you know, a YouTuber that, you know. That's how you get the views, you know. Yeah. And I get that. But at the same time, you know, I kind of shake my head.
1:00:09I'm like, okay. That's helpful. No, it's not. And again, if as an ecosystem, we say to ourselves, we don't want open source projects that drive mission critical functionality that are driven by a single company. I don't agree with that. But if as a global ecosystem, we say that. But I want people to think really hard about what that really, really means.
1:00:38well friends building multi-agent software is hard agent to agent and agent to tool communication is still the wild wild west so how do you achieve accuracy and consistency in non-deterministic agentic applications. That's where the agency, A-G-N-T-C-Y, comes in. The agency is an open source collective building the internet of agents. And what is the internet of agents? It's a collaboration layer where AI agents can communicate, discover each other, and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for interagent communication, and modular components to compose and scale multi-agent workflows.
1:01:26You can now build with other engineers who care about high-quality multi-agent software. Visit agency.org and add your support. That's A-G-N-T-C-Y dot org.
1:01:43If you had your way, like if this went without a fight and you were able to do what you're trying to do, what were you really trying to do with the project? If things went your way, this fight didn't happen, and you could leave the hotel, so to speak, what would you have done? What was the goal? So earlier on, we were talking, I think, about, I don't know if we were being recorded or not, on programming languages, you know. And so what I was looking at originally was if we created a foundation that was, you know, had a Synadia representative, but also had representatives of all the major users of Nats and customers of Synadia in a 501.3C or something like that, that had a commitment to AP2, right, open source, you know, XYZ, that's probably what we would have done.
1:02:35But it was simply just for the betterment of Nats. And everyone that sat on the board, similar to kind of like the TOC, although the TOC is more of, hey, we'll get involved if you need to move from sandbox to incubating to graduation. or if something's off the rails. Otherwise, we let projects manage themselves. This would be more of bringing in, you know, a lot of our top customers, you know, the Walmarts of the world, you know, things like that to have a seat at this and be able to help direct where the project would go. And right now in the CNCF, that's just not a full-on reality. That's more limiting, more governed by other.
1:03:11Yeah, couldn't you do that inside the CNCF? We could, meaning we could, I mean, right now, Then you graduate, wouldn't you? I mean, you did that. Then they graduate you. No, the graduation criteria that we failed, and I don't know if it's still the same. I have to check with that, was the code contributions in the server were not diversable. Okay, so it's not like governance. It's actually code contributions. That was what tripped us up in terms of trying to graduate in 2020. That seems to be off. Like, that doesn't make sense. I mean, it's more measurable. But if you established even a TOC of your own, right, or TSC, like a technical steering committee comprised of Cinedia folk plus your largest customers, right?
1:03:54Yeah. And they didn't write the code. Like, who cares who writes the code? If they're helping vote on the direction of Nats, I would think that would be a diverse open source project. Yeah. You know, I would tend to agree, but again, I'm not trying to say, oh, the CNCF's wrong or try to speak on their behalf. I just – And they aren't here to represent themselves. Right, right. So I don't want to do any of that. We also have the inside of the story. We're not journalists. We're just podcasters. We're here to talk. Yeah. Right. Worth saying. They're not here to defend their side of any of this. Yeah, but what happened to us with us not controlling the narrative, I want to be very sensitive to the fact that they do have differing opinions.
1:04:33And again, I'm extremely encouraged by our talks and how they're like, OK, yeah, let's see if we can do better here from a, you know, graduation flow potentially. But also, I don't know if it's this notion of I don't know the answer and I don't think the CNCF knows the answer either. But like, for example, modular, I don't know if you saw they just did a license change as well. They're getting more and more permissive, right, because they want broader adoption. And it's AP2, which is great. Modular. They do Mojo. That's Chris Lapp. Okay. Modular is Mojo. Okay. Yeah. But what interesting thing was is that they say, if you use it in production, you just have to let us use your logo in the use case.
1:05:15And I was like, that would be good. I would like that. Right. Because we have so many companies. I mean, if you go to Nats.io and look at who uses us, those people opt in. They send us and say, hey, please put our logo on there. But there's so many more that, you know, use us and don't want that or actually say, you know, no, you can't use our logo whatsoever on that. So even something like that to kind of go, oh, we can see Acme Company is actually using that. And we know Acme Company. That's really cool. That would help, you know, a company like Cinedia, right? That's that trust factor. And so, you know, I shot that over to the folks at the Linux Foundation and just were like, hey, you know, would you guys ever consider something like this?
1:05:56I don't think they will, but I'm trying to do my best and operate in good faith and say, okay, if we want to make this better, I'm going to tell you things from my perspective that could make it better for a project that looks like Nats and companies like Stadia that are driving that. and we'll see how that goes, right? I don't expect anything material to change radically, but at least they're open to the conversations now and are starting to be receptive to, okay, we see that there's some challenges here that we might be able to improve, so. So modular setting that up, effectively Apache 2 with an additional constraint on it, isn't that then technically like a proprietary license now?
1:06:35Like they've customized a license and now it's not exactly Apache 2 anymore. It's like Apache 2 with this other thing, which is very similar to like what Meta did with Llama, which is like, it's pretty much open source. Except for this one, now their constraint was much more strict than, we want to put your logo on our website. It was like, if you have 10 billion users, you can't use it or something like that. But I think, I mean, you raise a good point. My, I'm not a lawyer and don't pretend to be, but I think in that notion, the license strictly governs what you can and can't do with the code, meaning the code itself.
1:07:15So, for example, even within the CNCF, right, the code is AP2, but the CNCF has very other regulations around how you use the trademark and how you can use the logo and how you can, you know, do derivative products and use the naming. So I think what Monterey is doing is fine. You know, it's outside of that pure, what can I do with the actual code? but what they're saying is that if you're using it in production then you're giving us the right to use your logo. I think those are separate but I understand it feels like it could be sticky because the difference between a trademark versus a copyright versus a code license are all nuanced in how they kind of interplay for sure.
1:07:53Which reminds me of something else that I read and this was just again armchair commentary so I'm not even sure if this is true or not but when you become a CNCF project you typically will actually turn over IP to the CNCF, right? Like the actual intellectual property becomes part of the trademark. Yeah. Trademark logos or whatever. But yeah. Yeah. And with NATS, did that happen? So what happened with the trademark was kind of interesting. And again, it wasn't necessarily stated. And I tried to correct some of the facts and I got pitchforked even worse. Okay. But essentially, when we did the original kickoff for the CNCF at the switch, this is my recollection, was that you only legally transferred those upon graduation, right?
1:08:47And I remember Google kind of being a little nervous that if Kubernetes started directly as a graduated project, that they would have to figure out all of that stuff with all of their lawyers and XYZ. And so that was in my head. Now, to the CNCFs, to benefit them, I think they quickly changed that once Kubernetes graduated. And they said, nope, as soon as you're in, you essentially have to transfer, or at least once you move past Sandbox. And we joined as an incubating project in 2018. So the rules had shifted from what my recollection from the original kickoff meeting was. But besides that, the thing that, to be honest with you, frustrated me, you know what I mean?
1:09:32Just being transparent. But again, I like the conversations that we're having with the Linux Foundation and CNTF. But as we were trying to graduate and we had just joined, all of a sudden Major Leagues Baseball came and said, hey, you know, we don't like your trademark because it conflicts with the Nationals baseball team out of Washington. And so I was like, well, that kind of is silly. And anyone who's done patents from your listeners know that there's a little bit of a game to that as well, the whole patent and trademark and all this other stuff. But on the trademark side, I was like, I can't imagine someone being able to argue correctly to confuse a baseball team with a piece of software.
1:10:11Yeah, different domains. I think that's spurious. Yeah. So I approached the CNCF and said, hey, we got this opposition. We want to fight. I asked our legal team. They think it's going to cost around this much. And the CNCF at the time was not willing to take that fight on and suggested that we just change the trademark and that if we do that and make it easier on them, that they would give us a keynote slot so that we could announce the name change. But I was wed to the name. I cared deeply about the name and the logo. For obvious reasons. Yeah, because remember, this had been gone. It was named and we kind of started to have a logo for almost eight years before we officially joined the CNCF.
1:10:49So it had a probably longer lifespan than people realized prior to the CNCF. And so we fought it. And on our own dime as a small startup that was only a year old had just gotten seed funding. And we successfully defended it. And so upon completion of the successful defense, the trademark was awarded to Cineadio with the deal. And so at the time, when I was talking to the CNCF, they were like, well, we own the trademark. And I go, well, technically you don't. Now, this will get listeners frustrated and upset. And I get that. And I'm totally empathetic to why you would get upset. And I'm not asking for empathy by any stretch.
1:11:33But I was very frustrated that something that they said they cared so deeply about, they threw away in one email and said, oh, no, you guys figure it out. and then to say, oh, but congratulations, you won. Now give it to us. That also rubbed me the wrong way. But technically what the CNCF was saying is, no, you know, the commitment was you were going to give it to us. And so I definitely understand their position as well. And we are committed to that, right? So in part of our agreement, we're going to transfer everything over. The lawyers are working out the legal docs and things like that. But that was kind of my frustration, you know, to be transparent.
1:12:14And then they never asked for it after we successfully won that. And so, again, the way things were conflated and released out was, I think, very targeted, you know, in terms of, you know, trying this in the court of public opinion. You got two questions, but one I'm going to pause on. When you, when anyone, a project, maybe this one in particular, but any project, decides to take that project, donate it to the CNCF and do all the things you did, incubate, etc. Is there ever a point where you sign a specific agreement that's clear? So I feel like, Jared, you've heard me say this before, clarity and expectation.
1:12:55Like if you have clarity on an agreement, it's the whole point of an agreement. It's less about binding you, but more like here's the line we both stand on. This is what we both agree on. Was there any clarity or is there a signator scenario where you're signing and it's clear and this will happen and there's a series of events? Was that, did you ever sign something? We signed something to get into KubeCon in 2018, which had a line that says, if you have a project in the CNCF, you promise, I'm making some up, but I think it's online. You promise to abide by all the rules for CNCF projects. But that link, you know, I still can't even get into that link today.
1:13:36And so I think there's an opportunity here to improve that process. And also, I would imagine the listeners are very well aware of those emails that say, our terms have changed or, you know, terms of service have changed. And you get all these emails and most of us don't even read those. I never saw any of that. I'm not saying they weren't sent, but I never saw any of that to say, oh, hey, what you think, Derek, from the kickoff meeting, we changed it. And now it's this. So I think there's an opportunity to make that a lot more clear. But most people who are coming from an opposition to me say that that's always been the case.
1:14:15That should have been crystal clear to you that you donate the trademarks and XYZ. And I understand that because they see the CNCF as it has evolved. They didn't see it when it was first getting kicked off. And so I think, you know, they're totally right in the assertion that I should have known better. But I still think there's an opportunity to make it a lot clearer. Like even now, I can't remember if it's as soon as you get part of the CNCF or once you move past Sandbox. I think if Sandbox, you can take it back. If you don't get past Sandbox, I think, but I could be wrong there. And so even with all of this going on, it's still not necessarily clear what all of the rules of engagement are.
1:14:55But I just see that as an opportunity to improve for everyone, if that makes sense. Yeah. Well, like in that case there, if the link is broken or there's some sort of like language that's compressed, like there's a whole other document that's compressed in like four words, essentially, based on what you just said. And so those are usually addendums to an agreement. And you know this because you're probably signed tons of agreements in your life. Those are usually addendums and that's usually compacted into a single product. This is the agreement I'm signing. And if there's terms beyond the agreement I'm signing, they're attached as an addendum to kind of to help everyone safeguard themselves from scenarios potentially like this.
1:15:35Okay. So my second question is this, is it sounds like y 'all have an agreement. The blog post is out there. It sounds like kumbaya to some degree. Maybe not. Is any of this conversation you're having with us right now going to get you in trouble more so? I don't know. Because you're being very clear. Yeah, I'm trying to be very clear. For good reasons, but you're also being very clear. And it could be very clearly wrong clear, not wrong in terms of truth, but more like, hey, come on now. Don't say that in public on a podcast that gets listened to by a lot of developers in our world. Yeah, I hope not because we're entering into this in good faith and I've tried to make sure to interject that I'm extremely encouraged by the discussions that we have committed to transfer the trademarks over.
1:16:26We actually did finalize early this morning, which I'm happy to talk about, you know, our actual direction in terms of whether we're going to do a fork or what commitment we're going to do to the server, X, Y, Z. and so we're operating in good faith and I would imagine they are as well and it's a trust but verify you know what I mean type situation but I'm very encouraged by them being open and receptive to some of the challenges that companies like Cineo or projects like Nats might have and hopefully there's some meeting of the minds around Pass Forward that might level the playing ground if that makes sense but I would imagine you know there's going to be some people that are going to, you know, pitchfork again, and that's totally their right.
1:17:11But that's, you know, that's how things went down, and that's kind of where we are right now. And I still feel deeply about the Nats project and where it's going. I mean, one of the other things that listeners might not understand is there's different, in my opinion, there's different personas to projects that are kind of on maintenance mode, meaning we'll fix bugs and if there's other stuff, but the features are pretty much complete. Nats and the server are definitely not in that camp, right? We've got over two years' worth of plans, and still the majority of our R &D budget goes into stuff that sits in the Nats.io ecosystem, right?
1:17:50Part of that open source project that sits inside the CNCF from all of the clients, obviously the server, a lot of tools, you know, Kubernetes, Helm charts, SystemD, all kinds of things like that. we're still probably spending, you know, 70 plus percent of our R &D budget just on that. And that hasn't changed. Right. And we don't plan on slowing that down. It was, hey, is there an opportunity for us to recoup some of those costs for people that care the most about some of those crazy enterprise features, massive scale security, whatever. And that's kind of how we started to go down the path of, hey, what if we did, you know, A commercial fork, but may get BSL, again, it didn't land well because of the narrative, but our thought internally was that to us signaled the biggest commitment to OSS and to our customers trying to avoid vendor lock-in or price scouting.
1:18:43But it was in a vacuum. And so once we got additional information, we realized that we probably would have to reevaluate that thought process. So how would the commercial fork with BSL be different than OpenCore? I'm not understanding that. We would be on, BSL also is source available. It's not an OSI compliant license. So NAS would be the entire NAS project? Just the server. We would put all of the effort into that fork, which meant that once it was released, then the two-year window started. And so then that would flow into the AP2 code base that's covered by the CNCF. That was the original thought process.
1:19:22Now, we're not going to do that. Yeah. What's wrong with OpenCore though? Like, it seems like you could just accomplish the same goals without relicensing. You just have, you know, you want your billion transaction messages per second, then we've got an add on for you or whatever it is. Yeah, you're absolutely right. And what we've heard from the ecosystem is we like that model because that's what's familiar to us. What I try to point out to people is that commercial forks, it's totally up to Sanadia. And maybe that's totally fine. And people are saying it's fine. But again, I don't think folks think all the way through some of these things some of the times.
1:19:57It's at our discretion what we put in there. And it's also our discretion that we don't have any commitment to put it back into the open source community whatsoever. whatsoever. So let's say there's something, Jared, that you really, really wanted and you're, you know, an avid user and stuff and Canadian sides, oh, we're only going to put it in the enterprise version. You're stuck. Now, of course you could try to do it, do your own version of it in the OSS side, but at the time, and again, this was in a vacuum and now we have additional information, but I was arguing hard that the VSL would signal a better commitment to saying everything will become AP2 after two years, no matter what we put in there, it will become that way.
1:20:34And so that was kind of our original thinking. We knew that was going to be a fork. The question on whether or not we could use the Nats name, who it was, you know, part of where we lived in the CNCF or if it came out and who technically owned that, that trademark. But we already knew, we knew that that was what we were, we could do. It's just, if it's commercial, we have no, you know, no guarantees to either our customers from vendor lock-in and price gouging or the OSS ecosystem. And so at the time I was like, ooh, you know, I feel this is a better signaling mechanism. But we needed to control the narrative, meaning once we actually did decide, and that's the other thing that folks might not know, we hadn't finalized our decision when the can of worms got opened up.
1:21:19And so everyone's like, you got to tell us what you're doing. I'm like, we don't even know what we're doing yet. We were just still having internal discussions on that because it was totally separate subject from the relationship with the CNCA. Well, do you know what you're going to do now? or are you still trying to figure it out? No, we've been meeting every day since the whole debacle started. And so we already had gone down a path of doing composition via Synadi IP. And so what I mean by that is, is that a lot of customers came to us and said, hey, we want to take control over AuthNNLC, right?
1:21:52We have a pretty rich AuthNNLC system, but they go, it's not enough, right? We need something else. And you could imagine a world where we said, oh, we're going to do a Synadia server and we'll put all that stuff in there somehow. Instead, and this is already done, this code was done probably a couple of years back. We did an off callout mechanism, meaning the AP2 server, you can say, hey, you're no longer in charge. Something over there is in charge. And then we, Synadia, could build lots of IP around for off end, XYZ, OPA, integrations, things like that. And so we already were going down that path.
1:22:29So our decision now is we will not do a BSL stuff. You know, we know we have the right or reserve the right to do a commercial fork. But I think what we're going to do is we're going to post and we can do this in the show notes of everything we've committed to go into the AP2 server without any delay. The BSL would have had a delay on there and that we're going to expand our commercial offerings through composition, Meaning you're running the server, but now all of a sudden it's got a security appliance, which we've been working on and some really interesting stuff. Think of it like a WAF for Nats plus a whole bunch of other stuff.
1:23:04And so we're going to go down that path. Well, I can tell you there's so much nuance in this. History, nuance, good intentions. Not a lot of bad intention necessarily, but maybe some confusion, some lack of clarity. OMG, please fix that stuff. When a subject matter is very emotional, right? It puts your brain into fight or flight, which means that you have to make decisions very quickly on imperfect information. That's how we all survive, right? That's why, you know, we're all here because we ran away from the lion instead of going, oh, look, a cat. Let's pit it, you know, type stuff. So I'm not being critical of folks doing that because I understand why they're doing that.
1:23:45but hopefully they'll understand that they definitely had imperfect information and not even close to a complete story and it's not it's not their job to understand all of that but at some point hopefully they go oh there's a lot more to this than than we might have realized and not saying we're right and or wrong and i'm not doing that at all i'm just saying it's worth a dialogue and there's a lot more pieces to the pie that folks might not know and to the credit i think of the potential pitchfork folks or the folks in the crowd not in the arena is that it seems to be almost a not literally a daily occurrence but pretty frequently these days something happening right out of license change or rug pull it's so much that it's become a thing we've said here rug pull not cool that is our phrase okay rug pull not cool is totally not cool and that's an additional one jerry we could put that one on a t-shirt you know I think they're used to, and we're used to hearing this and you think immediately negativity, you immediately think this is, you know, nefarious.
1:24:49There's, you know, something going on here. And so it, it's almost as if there's a level of PTSD in the community. And I think, you know, as purveyors of and participants of enthusiasts of open source software, they have a right to that in a way, but also there's so much nuance here that they have to let some of the details percolate and it's not always clear. So, and you're right. This is why we do this podcast. This is why we have you on the show. Like, like I said, this is yours. And I'm making is we're going to have you on the podcast anyways. We've been talking on LinkedIn about getting you on the pod.
1:25:25And then this happened. I'm like, well, Derek, can you talk? Are you able to pod? I mean, I don't know if you can even pod. And you're like, yeah, I'm cool. And so we had you on the pod. And that's why we do this is we give that clarity. Yeah, and I knew the rug pull stuff. I've been listening and I know like Joran was talking about it and others have been talking about it. And so I'm totally aware. And what my hope is, is that while we're focusing on license change and rug pulls that we up level to have a dialogue of why is this happening? What are we as an ecosystem failing at and what can we improve upon?
1:26:00What, you know, I said earlier on in the podcast, you know, charity is not a business model. Well, until you're a company that is self-sustainable, right, you're not just running on VC dollars, right? That's still kind of a charity model, right? So in other words, right now, our customers are funding a lot of the development, But the VCs who have funded Synadia are funding a lot of the development that everyone feels entitled to for free. Right. And so. My hope and, you know, my sincere hope is, is that we can up level and have a dialogue about why do we think this is happening? And are there things that we could potentially improve from an ecosystem perspective that could could do that?
1:26:45And again, I'm not a fan of this, but one might be, no, we don't want any projects that run critical infrastructure to be, you know, from just a bunch of folks or a single company. I don't think that's right, but that could be one answer. Or, hey, if you're a big, well-known Fortune 100 company and you're using something in critical production, give back. You know, it kind of like your rug pull thing. It's mine is, you know, no pay, not okay. I like that one too. You know, because again, production side is never zero. Someone has to put in time and money to produce this. But on the consumption side, right, we're getting to everyone expects it to be for free.
1:27:23And that eventually has to be resolved somehow. Now, for projects like Kubernetes, it's resolved because the Googles and Microsofts and Amazons of the world, right, are motivated to put stuff in. And they have other revenue streams that can offset the cost of participating in the software. That's totally fine. I'm not saying that that's wrong. I'm just wanting the dialogue of saying, is that the only model that we really think will survive? And I don't want a world that looks like that myself. Yeah. Well, good for you for sticking up, I suppose, for yourself or trying some attempt and having this clarity.
1:27:58I know the narrative hasn't always been your favorite, but I appreciate you coming on the show. Even if it gets you in some trouble, I sure hope it doesn't. I'm sure it will. Just come on here and being clear. And, you know, honestly, I think not that it's here at the end, but all along, anyone from the CNCF who represents, you know, sandboxing, incubation, graduation and the clarity there. And maybe some of the things we brought up here, which was the agreement you signed and the terms that you alluded to and the lack of clarity there. I'm happy to and we're happy to have anybody on this. I think it's good for everyone to have some of that clarity.
1:28:31So if you want to have something on the show and talk about the details of what it takes to join the CNCF, what those are, we would be more than happy to have that conversation in detail. Yeah, and I think that's great. I really do. And like I said, my hope for all of this is that we can have an open dialogue about are there things that we can prove upon, you know, outside of Sanedia, outside the CNCF, but just as a broader spectrum. And I think there's some interesting questions that should be discussed on, you know, how we want to go about that and what role the CNCF and the Linux Foundation plays, what role, you know, retainers play.
1:29:08Because one of the other things that we got a lot of pushback from is I feel deeply that the lifeblood of a project is in the maintainers. it's yes people want to concentrate on license what foundation are you part of all of the stuff and and i think they do play a role but at the end of the day it's the maintainers and if all sudden the maintainers withdraw their support from a project um i think you know a lot of projects that have a tremendous amount of investment um and are not just a simple weekend toy uh if that makes sense, you know, saying, oh, we'll just get some maintainers and they'll spend a weekend and then they'll know how to program, let's say, a NAT server, right?
1:29:52I think that's maybe a little short-sighted. You know, we have someone who's awesome internally that just got promoted and is now running the server team. You know, they graciously took that from me so that I can do other things. You know, that person came in and was under our tutelage and specifically under my to which, and it still took them about eight months or so. So we were paying them and we were tutoring them. And most of the server's infrastructure was built by myself and someone named Yvonne. And we were tutoring this person. And it still took about eight months, almost a year to become like, oh, I could go anywhere on the server code and be effective, right?
1:30:32Be productive. And so when you get statements that, oh, fork it, you guys go do your own thing and And then we'll, you know, archive it or try to find new maintainers. Again, I want folks to kind of really think through that and say, is that the best outcome for the ecosystem? I don't think it necessarily is, but someone from the CNCF might say, yes, that's absolutely the way it's supposed to work. But what people are looking at to the CNCF around stability for projects and longevity, you know, if the maintainers leave, right, there's going to be some level of destruction for sure. Depends on the project, but there's, you know, it's not going to be zero.
1:31:15And in some cases, you know, in our point of view, which people might disagree, Nats thrives when Cinedia thrives, right, because we're driving all of that stuff. At least that's my opinion. And again, not all projects look like that, and that's totally fine as well. But that's another one where I was like, that doesn't feel like it should be the only answer, right? There could be a dialogue of how do we better support these types of projects? What if they're not necessarily a good fit as a foundation evolves? You know, what options are there for doing that? And again, I think prior to the dust up, but also going forward, and again, we'll release a statement next week, probably about whenever the podcast goes out around our commitment.
1:32:08You know, most people might not understand, you know, of our total R &D budget, 70 % is going into open source. And it's funded by the VCs who fund Sanadia and it's funded by our customers. and the ones that can pay that aren't paying that are like give me give me give me you know i'm entitled to this you know hey get off your ass and change this thing for me which we see every once in a while like github issues um that's kind of what i'm like maybe we need to figure out a way to have them change their perspectives a little bit i'm not saying you know i i don't want hobbyists to be able to use our stuff for free or embed it all over the place right my my original tagline for NETs was connect everything, meaning it had to run everywhere and was openly accessible.
1:32:50But again, it's that production problem, which is non-zero. It's like, how do we solve that problem for different levels of projects and open source? For sure. And I don't have all the answers, obviously, as you can tell, but I do believe... I mean, I think the important thing is you have a lot of thoughts, you have a lot of care. And I think that's the thing you have to really come back to is you got a lot of care for what you're building, what you've built, you know, the community, the spirit of open source, your customers, your employees, your investors. I mean, there's so much things I'm sure that you're holding that we're not privy to that we can definitely see that you care.
1:33:25But it's cool having you on this show talking about this stuff and even more cool that you're a listener and you know, road pools are not cool. I know. And, you know, no pay, not okay. That's right. No pay, not okay. And then that on the back. That's right. represent both sides. That's right. Okay, get it together. Derek, thank you so much. Thanks, Derek. Nice to meet you. I appreciate it. Thank you, guys. And, you know, congrats on all the success. Really, really cool to see. Oh, man. Thanks, Derek.
1:33:56So my takeaway from this conversation, specifically around the kerfuffle with the CNCF and how that all played out, is that there's a lot of nuance in these things. There's a lot of detail hidden behind the scenes, a lot of time spent, emotion, investment, time, energy, and all the things to really complicate the relationships in and around open source and ultimately any commercial open source companies that spin up around open source. It's not just black and white. There's a lot of details and this conversation really helped me see Derek's side of things. Now, with that being said, of course the conversation is open we would happily have anyone from the cncf discuss with us the incubation process what it takes to graduate the clarity they offer to anyone donating or assigning ip or copyright or a project to the cncf because there's a lot to discover there there's a lot of detail in there and again it comes back to that nuance so if you listen to this from the cncf reach out to us editors at changelog.com so a side note i suppose for next week for our loyal listeners and those who listen to the end of the show that's you listen right now yeah that's you we're out next week we're in seattle at microsoft build i'm not sure what the publishing schedule will be we're heading into summer it's the last week of school and i feel like everything needs to be done right now with that being said if you're going to be at microsoft build make sure say hi hop in zulip go to changelog.com slash community sign up it is free to join at mention us or whatever on the socials dm us email us all the things whatever just say hello of course a big thank you to our friends at heroku for sponsoring this episode the new generation of heroku is so exciting heroku.com slash changelog podcast our friends over retool retool.com slash changelog and of course the beat freak in residence break master cylinder okay friends that's it this show's done we'll see ya
From the publisher
Derek Collison — creator of NATS and Co-founder & CEO of Synadia — joins the show to dive into the origins, design, and evolution of NATS, a high-performance, open-source messaging system built for modern cloud-native systems and part of the CNCF. Derek shares the story behind NATS, what makes it unique, and unpacks the recent tensions between Synadia and the CNCF over the future of the project.
