Elasticsearch is open source, again (Interview)

26:34Yeah. It's important to know that, you know, because a lot of companies struggle with this same struggle you have there. And we see rug pull at rug pull, whether it's because of an IPO company that wants to protect its moat or some other route that over years things change. There's always a reason to do a relicense. And regardless, the community feels the pull of the rug. I think it's fair that there's more companies changing the license today because they're really worried about cloud vendors. And, you know, it's a bit, it's funny to me that there's not more of, hey, this is real pain that you're feeling.

27:11Startups, you know, I developed Elasticsearch in my free time in my living room. You know what I mean? It's like, and somehow, I don't know, maybe it's human nature. I find it funny that, like, people take the side of Amazon in this case. Like, I found it hilarious. But I think there's real fear. And we need to acknowledge the fear. And if the outcome that we want is more open source in the world, why do these companies change it? It's not because they IPO-ed. I'm talking to tons of startups that go and say, I'll never open source my code that would have open sourced the code because they say, we don't want the cloud provider to come in and take all the stuff that we built and pull something around it.

27:55I think there's a few companies that says we need to bring trademarks back and enforcement and things on those lines. All of these are good discussions. Those are the discussions that we need to have. What I can say, at least with our change, and go back to what we spoke about in the beginning, I think that enough time has passed from the hectic days of early cloud that AGPL is potentially good enough protection. AGPL is the license that we chose to be open source. And it's potentially a good enough protection. So when I talk to companies today, I say, go open source, build things in the open, choose AGPL, and it's probably good enough protection because we're choosing AGPL.

28:37You know what I mean? Because we're showing that that's the case. So hopefully that changes the trajectory of choosing an open source license and which one ends up being chosen. Yeah, I thought that was interesting that you did not go back to the previous license. you went to AGPL because it's going to provide you more protection than was it Apache 2? Is that what you said it was previously? Yes. Yes. Yes. Yes. And of course, AGPL, cool with OSI or OSI cool with AGPL. So it is officially open source. What about slightly modifying the definition of what open source means in order to account for the change in the world that we've seen?

29:19because while I believe that open source definition needs to exist and there needs to be people that protect it and all that stuff, I'm not hardline on exactly that definition being written in stone tablets. Like you could slightly modify it in order to broaden the tent slightly. Is this something that you've approached OSI with? Like, hey, here's a license. Like why isn't SSPL good enough? And can we change the definition slightly because the world has changed? Yeah, I think it's a good question. MongoDB that created the SSPL license. So MongoDB changed their license from AGPL to SSPL, being concerned that cloud providers will take AGPL software, will test AGPL, basically, right?

30:03So they changed it to SSPL. So even from then, I think enough time has passed that I think when you put AGPL out there, it's a good sign for cloud providers that this is the contract that you expect to have with them. I think SSPL tries to solve it. MongoDB went through a whole process and you can read the mailing list discussions and things like that around trying to get SSPL approved by OSI. It didn't happen. I'm not pointing fingers at one side or another. There's a lot of emotions and the devil is in the details and what have you. I think there's a good question. I think there's a balance that OSI tries to put around what is open source and how do you define open source, which is very important to maintain.

30:45That social contract is important. There's a lot riding on the term of open source that we should cherish it and treat it very dreadfully, if that makes sense. And that's like rushing to adding like five licenses to it and it's fine. And then it becomes, you know, if we do it, it becomes meaningless. Right. So we tried to figure out if there's an option with OSI to maybe revisit the SSPL discussion. They have a new leader, and I think they're open for it. And they're trying to figure out what is the open source, how does open source live also in a world not only of cloud providers, but in a world of models and open weights versus open models and open data and things like that.

31:27And how does open source play in it? They have a lot on their hands. I think they want and they really want to try to figure this out. But for the same reasons why we felt like we had to change Elastix because we don't have time, I actually think that we're in a position where I'm not worried that someone takes Elasticsearch and provide it as a service. So I can conceptually take the very, very minuscule risk of just going with AGPL. And potentially with that, we're actually creating a fact. And it's like, that's it, AGPL is fine. Like that's how it works. And you don't have to go through the process of approving a different license.

32:00Well, we did have Stefano Maffoli, is how you say it, Maffoli on the show. When was that? This year. And definitely their hands are full right now with trying to define open source AI, as you said. It's tough. It's all hands on deck to figuring that out. So probably not much bandwidth for reconsidering the current open source definition. Yeah, we even posed the question, and I'm curious if you agree with this, Shay, is that we question, Stefano, if OSI cannot properly land a plane on the open source AI definition, if that might, how do I I can't recall how you word it essentially if they can't properly and in a well way define the open source AI license if that might change the perception of OSI's hold on or arrangement of the OSI approved licensing ability like if that changes the trust in their ability to land the plan on open source at large I suppose it wasn't worded that way but it was a version of that it's in the transcript you'll find it I'm doing a terrible job about it here in this moment?

33:07So I think that sometimes we make the mistake of judging open source based on, you know, it's like, we always liked the songs when we were teenagers and they're stuck with us and it's like, it will never be better. So it's like, some of us remember open source licenses and usage like it was in 2004 and five or three and seven. And it's like, how many hours did I spend around the differences between GPL and Apache? That was within open source, by the way. It's like, oh, you can't choose one versus another. I think developers today, they pick an open source license and they're fine. They're fine.

33:48You know what I mean? It's like they care much less about the fidelity of difference between Apache, for example, and GPL. I actually also think that big companies care much less about, like enterprises. We used to think like, we used to say Apache is much easier for enterprises to adopt, but look at MongoDB. It's all over the place in large enterprises and it's AGPL, you know? So it's like, obviously that's not hurting adoption or was AGPL. So it's not hurting adoption. And I think the similar thing might happen in AI and that might be the missed opportunity. We see open models, we see open this, we were saying open source AI, open source models when they're not strictly matched the definition of open source.

34:33Right. I don't know about, it doesn't matter about OSI or not. I just don't want the term open source to get diluted. And I think that's the responsibility of OSI and us, by the way, to help, to try to figure out, because we obviously play a big role in Gen AI as well and building models and things like that, to figure out how do we build a system that allows for it because you just don't want it to be diluted to become a term that means everything. Right. I actually think that what Meta is doing with Llama and its license, which is incredibly permissive, borderline open source, but not because of that one clause in there that if you are operating at however many million monthly active users, then it's not for you like that one little thing which makes it not open source according to any open source definition it's similar to saying you just can't re-host it as a service right it's like similar to that kind of a clause and compete with us but they're calling it open source and because it's so sneaky and awesome like what they're putting out is hugely valuable i mean just the raw cash value they put into training that thing over and over again and it's great i use it every day and regular people now, non-open source nerds like us, getting into this stuff, and they're just, Mark Zuckerberg calls it open source.

36:00It's open source. It's pretty much open source. And so I think the OSI, maybe they're already missing the opportunity to define that sucker because I think Mark Zuckerberg might be defining it for the next era. Yeah, I almost wonder if we should have a, it's pretty much open source.com. because pretty much open source that's what people say right it's like a princess bride it's only slightly dead yeah exactly yeah good one and that was kind of my sentiment when i asked that question was it was exactly this year like if if the osi is not the governing body that gets to define open source ai does that change things in terms of not so much open source but it kind of puts it almost in the hands back of the community.

36:49What do we feel is it's almost, what was the .com? It's almost truly open source. What was it? I don't know. It was 30 seconds ago. We already forgot it. Gosh. Open-ish is what I've been calling it. It's pretty much open source. Pretty much, yeah. It's pretty much open source.com. Which I think is actually pretty cool, honestly. I mean, that's my fear for the OSI is that if they can't act swiftly enough and it gets, the conversation gets lost in the minutiae enough, which they've welcomed, they've done the process the right way, but are they the value creator like Meta has been with, you know, with this, you know, to be able to essentially own what defines it because of the premise, the prowess of it.

37:31I think there's a lot of urgency in being able to properly define it. So we own the term, like we as a society own the terms open source and it's like it defines what we do. I will say this is not like an us versus OSI. That's not at least how I think about it. I think OSI is doing a lot of amazing work, and it's our responsibility to help OSI to figure this out, whether it's monetarily where we can contribute to OSI so they have more people and more money to help figure this out, or by working with them to try to figure this out together. I do think in such a new space that moves so fast, like Gen AI, like models, open source has been hijacked.

38:15The term from OSI. I think so. And I think it's important to try to figure out how do we consolidate that fact that is happening as we speak. Yeah, I mean, it's getting harder and harder. And perhaps at this point, it's impossible because there's no actual definition to hop in and say, well, actually, you know, like to actually point out this license is against the spirit of open source because they're doing arbitrary limitations on use, for instance. And that's going to be an uphill battle, and it might already be a battle lost because of the pervasiveness and the value put forth by those who are calling it open source and just don't care what OSI says, i.e.

38:55meta. You know, it might have such a microphone, or a megaphone, I should say. I'm sure they have microphones too over there. That, you know, OSI and anybody who cares about open source definitions are going to be such a fringe group that we're going to be able to call anything open source. Then you have to go read the license and realize it has all these arbitrary restrictions on it. And now what are you going to do? I think it's more than the megaphone that Meta has, right? Because Meta, for example, released quite a few open source projects over the years. For sure. And if they would have called one of them open source where it wasn't OSI approved, I'm pretty sure that we could have resisted it.

39:34I think the other factor that is at play that's a model that they're putting under a very permissive license that costs hundreds of millions of dollars to build. That's like pretty crazy. You know, it's like in a very short amount of time, I don't know, probably, I don't know, I didn't do the math of how many like money did we spend Elastic as a company and personally on just building Elasticsearch, but in a very short amount of time, a lot of money was spent to build that product. And this is pretty remarkable to be able to put it under such a permissive license. So that's another reason why people go and say, okay, we'll want to use it.

40:12Yeah, 100%. I mean, I think that it's so compelling of a piece of software slash data slash whatever it is that the value it brings is immense and almost incomprehensible to everybody except for those like seven companies that happened to hit that one clause. and it's like we don't care about those companies necessarily it's like okay they're met as competitors it's like a handful of orgs everybody else have at it it's so close to open source and so valuable that I think it can actually completely hijack the term and it won't mean what it used to mean and that might just be something that we have to accept at some point.

40:52Well they were not they were very forthcoming with the usage of the word open source they used it in the hyphened version of it on llama.com It says, in quotes, the open hyphen source AI model you can fine-tune. So it's used there. And then in the announcement, which they link up, they say, see how Llama is the leading open source. And in this case, they don't use the hyphen version. They use just open space source model family. And it links off to a blog post that is titled, I won't title it. It's a long one. I'm not reading that. I'm not reading that. But then all throughout this linked up post, which is how Llama is used in this calendar year 2024, is just open source all through it.

41:36Open source promotes. This is the leading open source. It's again and again the usage of the word open source. So it's pervasive. Pervasive usage of the word open source hyphened and non-hyphened. And so it's not like there's any dispute. They're trying to say it's open source. That's the point. Yeah. And I think the interesting part is that, for example, they made the license more permissive just recently, right? If I remember correctly. So it's a - And those versions is even less restrictive than the last one. And I think this is the important work that we need to do because open source within models might have a lot of fidelity.

42:08There's the weights that might be open or open source. There's the retraining opportunity that you might be allowed to do or not. There's the data sources that you use to train the model. They don't have to be open source, but are you opening which data sources, like just listing them so you can go and certify them or not. So there's a lot of areas in these places that if we have a standard way to think about them like we do with other open source licenses, I think it will be good for the world. I think there's a meta question around if we're going to have really like, it's a question also of how many models are we going to have?

42:49Especially like LLMs, like large language models that are really expensive to build. If we're going to have five, then maybe it doesn't matter. and people go and say, oh, we know the Lama one, we know the challenges that we have. Yeah, they call it open source, but I know, you know, we know the restrictions. If there's 20 or 50 or more specialized models and things like that, that's when I think we really urgently need a common way to understand what is our freedoms and what do we get when we use a specific model. It's really important to understand it in a standardized way, if that makes sense.

43:24Yeah, good point, good point. And who wants to go to war with Zuck? Who wants to go on X or Twitter? We don't need to go to war with Zuck. We need to just bring Zuck in. Exactly. Listen, Mark, if you're out there, just take that restriction off. It'll be fine. You'll still be rich. Meta will still succeed. Just take that one little restriction off. It can do what we do. It can just AGPL it. And then it's like, you know, if you end up using this LLM, then you need to open source the rest of your infrastructure as a source. Yeah, exactly. Reverse rug pull, Zuck. Do it. Just pick an open source license and let it ride.

44:00Well, maybe his retort would be, well, there is no open source AI definition. How can I pick one?

44:14What's up, friends? I'm here with a new friend of ours over at Assembly AI, founder and CEO Dylan Fox. Dylan, tell me about Universal One. This is the newest, most powerful speech AI model to date. You released this recently. Tell me more. So Universal One is our flagship industry leading model for speech to text and various other speech understanding tasks. So it's about a year long effort that really is the culmination of like the years that we've spent building infrastructure and tooling at Assembly to even train large scale speech AI models. It was trained on about 12 and a half million hours of voice data, multilingual, super wide range of domains and sources of audio data.

44:57So it's a super robust model. We're seeing developers use it for extremely high accuracy, low cost, super fast speech to text and speech understanding tasks within their products, within automations, within workflows that they're building at their companies or within their products. Very cool. So Dylan, one thing I love is this playground you have. You can go there, assemblyai.com slash playground, and you can just play around with all the things that is assembly. Is this the recommended path? Is this the try before you buy experience? What can people do? Yeah, so our playground is a GUI experience over the API that's free.

45:35You can just go to it on our website, assemblyai.com slash playground. You drop in an audio file. You can talk to the playground. And it's a way to, in a no-code environment, interact with our models, interact with our API. to see what our models and what our API can do without having to write any code. Then once you see what the models can do and you're ready to start building with the API, you can quickly transition to the API docs, start writing code, start integrating our SDKs into your code to start leveraging our models and all our tech via our SDKs instead. Okay, constantly updated speech AI models at your fingertips, well, at your API fingertips, that is.

46:12A good next step is to go to their playground. You can test out their models for free right there in the browser, or you can get started with a$50 credit at assemblyai.com slash practical AI. Again, that's assemblyai.com slash practical AI. And also by our friends over at Wix, I've got just 30 seconds to tell you about Wix Studio, the web platform for freelancers, agencies, and enterprises. So here are a few things you can do in 30 seconds or less on Studio. Number one, integrate, extend, and write custom scripts in a VS Code-based IDE. Two, leverage zero setup dev, test, and production environments.

46:55Three, ship faster with an AI code assistant. And four, work with Wix headless APIs on any tech stack. Wix Studio is for devs who build websites, sell apps, go headless, or manage clients. Well, my time is up, but the list keeps going on. Step into Wix Studio and see for yourself. go to Wix.com slash studio. Once again, Wix.com slash studio.

47:36Well, Shay, how did this news land? Like the four years later, you know, you're back, you're fully open source. You have an AGPL license now. You feel great. Was the community welcome Elasticsearch AGPL with open arms? Like what was the response been? Obviously, any change that you do is scary. And it's like the easiest thing to do is not to fight inertia. Right. And, you know, I won't lie, like internally, we people were saying, why do we need to change it? People got used to it. Like, why do we need to go and change to open source? So like these decisions are not straightforward, but we really felt like we are an open source company.

48:16We walk and behave like open source and this is who we are. And this is like, it goes all the way back to our, you know, value system as we were founded. And yeah, we announced it and it's been, it was an amazing, I was scared. I was obviously happy about being able to announce it. I was scared that people would twist it because you know how the whole social discussions might happen. Oh, Elastic is in bad shapes or they're choosing open source or something on those lines. And sometimes people really like to twist it. But it was just pure joy. It's like the vast majority was just like, this is amazing.

48:58Thank you for doing it. Welcome back to open source. prominent open source people were engaging and appreciative of doing it like Adam and others. So it's like, it was great. Obviously, there are still people that try to find the bug in something good, but I think there always are. I try to focus on the fact that this is just a net positive progress for Elastic in bringing back open source. And it's hopefully an even bigger net positive for open source world because hopefully it will convince more companies to do open source versus less. Why? That's the question I have. Why open source again? I get it, but truly why?

49:41I think there's a few reasons. The first one is we never stopped behaving like open source, right? All our code is still on GitHub. All of our issues are on GitHub issues, pull requests, reviews, people send us pull requests. You know, it's like never stopped. And if you go back to ask yourself why you changed the license, we changed the license because of the trademark, but the trademark is no longer an issue because Amazon changed the license to OpenSearch. So really, the only question is why not in that case, because if we're not afraid of Amazon calling it back, Elasticsearch, which we are not, because we work well together now and they went with OpenSearch and just totally fine, then I think the question is sometimes like, why not?

50:25Sometimes the easy one is why not? I also totally believe in the value of open source. As I mentioned, there's so much encapsulated in these two words, and so much of it encapsulates who Elastic is as a company with our contribution to open source, our many open source projects from Logstash and Kibana. Kibana got forked into Grafana and created a whole other company. There's so much that roots itself back to Elasticsearch. we invest so much in Apache Lucene, which is an amazing story in the open source world and an incredible one. There was always a hole left in our hearts when we took away open source and we just felt that we wanted to fill it, if that makes sense.

51:13I don't know. Well, it's cool it all worked out for you in the end. The fact that you were able to, I mean, the maneuver that you made, regardless how controversial, difficult, perhaps damaging to a small part of your community, all the things, it seemed like the series of events that came after it, Amazon deciding to fork, right? OpenSearch becoming a thing, that being a clear delineation from Elasticsearch, and then the changes that followed in the world that you're saying were like AGPL now is probably good enough. And I think if Elasticsearch can use it and can maintain it without problem with an AGPL from an Amazon or.

51:59Or others. Yeah. Yeah, who else? Microsoft Azure or something. Then that leads the way for other people to do it. Like, it seems like it all worked out for the good in the end. I think so. Or is the story not over yet? I still would like to take back the years that we were not open source. Yeah. It worked out, but not in the way that I would have loved it to work. You know what I mean? It's like it's still... Right. You would have loved it just not to do it in the first place. Yes. Yeah, because all of this at the end of the day, when you're just trying to run a business, an open source project, this is all side stuff, right?

52:31Like this is all headache. Yeah, I mean, we had a great story around how to get people on our cloud. We had a great story around like commercial features on-prem, like the whole business model that around open source that we, I think we were a big part in building as a company and as a community, you know, with other open source projects. Like this worked, like you don't want that headache. To rewind back a little bit, the challenge was the trademark issue, right? calling it Amazon Elasticsearch, which they kept doing. Did you really have a problem with the freedom of being open source that they would re-host or provide it as a service?

53:10Was your only true concern or true issue that it was a trademark issue and not a, while it was free to do, and they were free to do what they did with your source code and to re-host or re-service or provide a service, was that at all an issue? Because that's more like the freedoms of open source provide that. Under Apache 2, yeah. Under Apache 2, they're allowed to do it. Right. And obviously, as long as we have Apache 2 license, I'm not going to go to Amazon and say it's illegal. You know, so yes. Right, because it wasn't. It wasn't cool to do that, but they were free to do it. Yes. So some people say it's great, but it doesn't align with the norms of open source or something along those lines, which is fine.

53:52It's still legal to do it. Our case specifically was just around the trademark. The trademark. Okay. Yeah. In that case, I mean, it was a massive blip then, really. I mean, because we haven't really called out the blog post yet, Jared, but I think this is a fantastically written blog post, and it was written by Shea himself. I'm sure you probably had some feedback from other team members that said yes or no, or I don't know if you wrote this in isolation, but I love the way you wrote it. You wrote DNA, love, not like some, you know, phrasing some of the things you sort of earmarked in here. And the big part of this, the way you opened it was the DNA.

54:28The DNA for Elastic was and has been and is to be open source and what a pure joy it was. Like as I read that, and I'm sure, Jerry, we've had conversations as you read it as well. You can hear the excitement proverbially in your written words. We can hear. Also, can't you see, is this Kendrick Lamar? You can see some other passion here. Yes. Yes. My son is a huge Kendrick Lamar fan. And as a result of it, I became a Kendrick Lamar fan. I mean, I introduced him to him, but I didn't know him as well as I do now. Gotcha. Right. So that was like running through my head when I wrote it. Yes. So these little prefixes to the paragraphs, that's what Adam was reading.

55:07Love, DNA, not like us. These are all Kendrick Lamar references. Yeah. That I could just tell. We know that you wrote this yourself, Shay, and not like an LLM wouldn't come up with that. No, definitely. I sat down and wrote that blog post I care about. Maybe some people made fun on Twitter on this, like, why is he doing these references or whatever? But one of the things that I love about open source is that we're quirky and we're adding some fun to the stuff that we're doing and it gets interesting. And it's like, that's the thing that I love doing in IRC or Slack or GitHub or doesn't matter.

55:46Source, SourceForge, you know, it's like all of these things. We had fun building software and, you know, it's like, that's, that's the thing that I cherish a lot. And this is part of it. At least that's how I, I really. No, I saw, I thought the same thing. I saw people making fun of you, I think on Reddit for those references. And I thought, you know what, this is how I know that he wrote it. and it wasn't some like press release from some like suit, you know, like otherwise, especially with a company as large and successful as Elastic, which you all are a large company now, that seemed like something that Shea Bannon did and not somebody else.

56:22And so whether you like Kendrick Lamar or not, or you think it's lame to put a bunch of references in a blog post, too bad, it's not up to you. It was Shea's call, you know? Yeah, yeah. Thank you. so Adam did you get the Kendrick Lamar references or is that news coming in I really wish I was more of a hip-hop fan I don't even know what is what is Kendrick Lamar in terms of music yeah he's a rapper I mean is it called rap is it called hip-hop like what's the both okay I mean I listen to music but I wasn't they'll say hip-hop is a culture you know rap's an art style I don't know okay we'll have different ways of thinking about these things there's there's a hip-hop definition that's maintained by the OSI no just kidding yeah it was the true definition of hip-hop you know it's almost harder than open source yeah it it went over my head unfortunately because i'm just not that steeped yeah it's okay yeah it's all right now you're on the inside yeah you're with us i didn't get it at first either i am a fan of hip-hop but i'm not a huge kenrick lamar fan don't dislike him just don't know his work very well yeah and i wouldn't have picked up on it unless i saw people making fun of it i think on reddit i'm like chill out guys of course that's what people do on reddit they're the entire point of reddit is to make fun of everybody else totally that's already interesting and that's fun that's why we're having like fun discussions that's what we should be having not like heavy depressing right accusational discussions you know what i mean it's like that's the fun that we need to get back to having yeah i'm excited to uh to see elasticsearch be open source again i'm i'm excited for you to it's kind of it's kind of a strange thing because you've kind of had to be something you weren't for a little bit you know yeah if not you know not as a product as a team as a company but as a as little as a license change it is like it's such a core as you've said a dna component to who you are it's the beginning of what you've created and built to have to to change that for this protection which we have investigated disseminated discussed etc to its degree more than I think we probably should ever again.

58:27But it's good to know. It's good to know. I think the people of open source and this is, you know, this would be transcribed at some point. So this would become the part of the zeitgeist of what we consider as open source or not. So when all the LLMs eventually forge together and mine us for information, they'll have this conversation to look back on and say, this is the reason why they made the change. And this is the reason why they came back. I'm just happy you did. Yeah, I think details matter, as you said. and for people that are interested in it, it matters. I think the end result of having more open source in the world, it's much easier to just all agree on.

59:03And I think if we're all like the arc of it is where we're heading, I think it's goodness, right? And to be honest, I actually think that if you build enterprise software, that's my recommendation. You can pick AGPL if you want to go open source and not worry about the cloud stuff. As we discussed, I'm more worried about the large language models and AI area, which we need to figure out. And that's an area that we also need to invest in. And hopefully this podcast also helps push it in the right direction, if that makes sense. Right. Can we close with a technical topic? Talking about Elasticsearch?

59:41Sure. So I've been thinking about search products a lot because it seems like all search is kind of like up for grabs once again, isn't it? because now all of a sudden there's this brand new vector, I guess pun not intended, but there's a different way of talking to a thing about finding stuff versus crafting a query. Now you're crafting a prompt, but it's more conversational. And probably I think what would happen from that is elastic searches, the type senses, the name your open source or non-search product, the algolias of the world. Of course, you're probably integrating some of the stuff into your product, but aren't people probably going to start questioning their search functionality across their applications more than they ever have before?

1:00:26I think that people are going to see a significant value in their search applications by embracing semantic search and vector type use case. That's the opportunity. I always like to think positively around the opportunity. And I think the opportunity is huge. If I loved search as a way to liberate your data before, this is like 100x better. And when we, obviously, we at Elastic and we huge contributed to Apache Lucene, which we use, we added HSNW and vector capabilities to these systems years ago, right? Before this whole Gen AI happened. But once it happened, obviously, the weight of being a great vector databases has increased significantly because the improvements that you have in search capabilities is significant.

1:01:17And we're saying internally at Elastic that we want to make Apache Lucene and Elasticsearch the best vector database in the world. And then on top of it, the best hybrid search, because it's like it's not only vector search, right? It's like there's a lot of layers on top of it to make a great search system that we've been building for years. And yeah, I mean, like any new type of capability, I think you end up having like new competition or other new vendors or products, open source or not, as you say. I think Elasticsearch is a great database today. In Apache, we contributed so much work into Apache Lucene.

1:01:53We have some of our best developers working just on Apache Lucene on making it better and then integrating it into Elasticsearch. By the way, by making Apache Lucene better, we're making a lot of other systems better. So it's like our focus is to put the right technology at the right place and then win based on value. And yeah, so far, by the way, that's the fastest growing market in the world, in our business. So that's definitely the fastest part for us in terms of usage and business. And people are seeing the value. I do think one of the things that I'm trying, working really hard to figure out is one of the things I think Elasticsearch did really well is that it made search approachable to users.

1:02:36You could just dump a JSON document into it and start to search it, and you didn't have to understand vector space and ranking and BM25 and inverted indices and FSTs and all of these things. And then we progressively disclosed complexity if you wanted to or you wanted to tweak things, but it just worked. and we've invested over the last year a lot of effort, not only in making us a great vector database and great in hybrid search and things like that, but also making it as easy as it was to index. The first, we literally took the curl request that I had in the first blog post when I released Elasticsearch around indexing tweets and we're literally trying to make it this simple to do semantic text.

1:03:22And I think if we go back to hundreds of thousands, if not millions, of developers that know Elasticsearch and know Elasticsearch API, that's a huge opportunity. Because they don't have to become experts in vector databases and product quantization and HSNW and dimensions and embeddings, models and things like that. We can really simplify that layer to them and they can use Elasticsearch the way that they love. and to me like that's a huge opportunity because if we make every single elastic search developer out there like someone that uses elastic search just use semantic search even without almost without noticing because it's so simple like that would be amazing for them because it's significantly better and for us as a as a as a product in elastic search and as a company and that future is here or that future is coming i think it's here uh so we i don't know i'll get geeky for a saying, but we have like a, when you, when you define something in elastic search, there's mappings for it, which is basically the schema.

1:04:21And then there's like text mapping, which basically does keyword base, like analyzer BM25 type matching. If you do semantic underscore text and don't change anything else, it just becomes semantic. And we pick the built-in embedding models for you and we run it and we do everything out of the box, but you can, Every single layer is open. So you can change the embeddings model that you use. You can change the way that the query is being generated. You can go all the way down to just providing the vectors yourself if you want to. But the simplicity at the surface area is there. And we worked really hard for the last year.

1:05:01That required significant investment for us. And by the way, all of this is free. So all of this is in the open source now. I can say open source and just go and start to use it. And if you know Elasticsearch and you know how to index data, let me tell you a secret. You know how to do semantic search now. So that's exciting. That is very cool. Well, Shay, I'll just say welcome back, I guess, to the official open source community. And thanks so much for sharing your journey with us. Yeah, happy to. Thanks for hosting me. Well, that's a first. a reverse rug pull, a rug push. And I'm excited about that.

1:05:41I'm excited that Elasticsearch is back, back to being open source, not just open, but literally open source. And that's a very good thing. New frontiers ahead for search, all new interest, all new opportunity. And right now is the best time to go back to being open source for Elasticsearch. Good for you, Shay. I'm excited for you. If you haven't been in a while, Elastic.co. Check it out. That is the home of Elasticsearch. Okay, we have a bonus for you. This bonus was going to be for our plus plus only subscribers, but there was such goodness in there. We thought, you know what? Let's give a treat to everyone.

1:06:19So the bonus is for all today. Stay tuned. In just a minute, we go for another session with Shay. And if you can't wait, there's a chapter. So do that. By the way, if you want to become a subscriber, a supporter, a ChangeLaw++ plus member you can do so at changelog.com slash plus plus it's better you know what it's better because it's better that's all i'll say 10 bucks a month 100 bucks a year you get to drop the ads you get a little closer to that cool changelog medal you get bonus content and you even get a free sticker pack mailed to your actual address if you'd like again changelog.com slash plus plus Okay, a big thank you to our friends and sponsors for today's episode.

1:07:04Fly.io, Speakeasy.com, AssemblyAI.com, and our friends over at Wix and Wix Studio. Check it out, Wix.com slash studio. And those beats, those banging beats from Break Mess and Zylinder, we love them. Thank you, BMC. You're the best. And one more thing, we're going to be at All Things Open next week. So if you're there, say hello. We'll be at Booth 66 near the ballrooms. Come say hi. And if you're getting a last-minute ticket to the conference, use our code MEDIACHANGELOG20 and get 20 % off registration. That's it. This show's done. We'll see you next week.

1:07:51I'm not sure what to call this. I wanted to call it. I almost brought it in the show. I was like you know what it's a deviation from the main task and I was like I don't want to call it a rug theft maybe it's a rug misappropriation but then I think about this conversation we had with JJ Jared you remember this and sure you don't because you probably weren't there you may have not listened to the show but I asked JJ what is his full name? Joseph Jax isn't it? Joseph Jax thank you Jared do you know him of OSS Capital? Oh, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah. He's pro-open source as well, but he's also pro-commercial open source.

1:08:29Open core. Open core. And we asked him, or at least I asked him on that podcast, I said, are you okay with open source companies literally cloning proprietary sourced companies? Right. Essentially, you know, take the easiest version would be to take a Calendly and to turn it into Cal.com. And maybe it's not literally a one-to-one in terms of technology because Cal.com seems to have gone a different route in terms of its product. But this idea that there is a closed source proprietary software company that does not have a open source counterpart, are you okay with an open source company cloning the essence of that company, making the software creation that comes from it fully open source and doing that?

1:09:21And so I kind of feel like that's a, I'm not sure how to draw the line, but it kind of seems similar in nature to what Amazon did to you and Elastic, which was to use the trademark of Elasticsearch. And here's where I kind of like bring that back is that you have a super base out there that has said for a while, the open source alternative to Firebase. And so here they're leveraging the trademarked name, Firebase, in their value proposition to the world, which was by and large their claim to fame, which set Hacker News on fire. And so I just kind of wanted to conversate around with y 'all. I don't have an opinion necessarily.

1:09:59I'm just like, I see this happening, like a appropriation or a usage of a trademark term like Firebase and saying, hey, Superbase is the open source Firebase alternative. So I'll lay that out there for y 'all. What do you think about that? I mean, I can say, sadly, I know more about trademarks than I want to. So that's totally a legal usage of trademarks. That's totally legal. If you say you're allowed to say I'm a shop that service Toyota cars, you're not allowed to say I'm a Toyota service shop. Like that's the difference, right? It's like, and as long as the name is Superbase, they're totally fine to say it's an open source version of Firebase.

1:10:46It's like, that's total, that doesn't create confusion in the market between Superbase and Firebase. And that's totally allowed. It's fine. I actually encourage that type of marketing for any open source projects that there is. Like, that's, go for it. That's cool. I'm cool with it too. Yeah. It's the leveraging part of it. It's the trademark side of it that I'm cool with. But if there's a lot of folks who would say, hey, Amazon, please don't, while you can and you have the freedom to, please don't, because it's not cool to re-host Elastic as a service. That's the cool part, I guess, or that's the okay part, but it doesn't like socially or societally, like that's not kind of cool.

1:11:29Just don't name it Amazon Elasticsearch. You call it Amazon OpenSearch, which they've done. I just saw some corollaries and I wasn't really I couldn't articulate in the show so it didn't make sense but I wanted to bring it up here in this right in this zone where it's a little bit more free to like examine the thought process. Well I guess let me hop in real quick Shay and say that I think I fundamentally misunderstood your beef because I thought it was hey it's not cool to take Apache licensed elastic search and offer it as a service you're going to take all of our business. But you're saying that that's cool.

1:12:06That's fine. I mean, you said it's legal. So it doesn't mean you think it's cool. But it's not something that you're going to stop, try to stop them from doing. It was for you is all about trademark. That's basically what Matt Mullenweg is saying right now as well with WP Engine is that it's about the trademark. Now, he's not taking the route you guys took. He's taking a different route. Scores the earth. Nobody can understand at this point the route that he's taking. But, and even when we talked with Adam Jacob about this, like he was trying to explain why it wasn't wise for Elastic to do that because of Amazon's, you know, bringing more customers into the funnel and all this kind of stuff.

1:12:40But that was more about them rehosting. But for you, it's not about the rehosting. It's like about the trademark. It was all about the trademark. It was about the market confusion. Yes. I didn't understand that before today. I would say I agree with that, Jared. I didn't understand that either. I thought it was about the re-hosting and competing dispute, not the trademark only dispute. And maybe that was our failure to understand the circumstance four years ago, or maybe that was, it got lost in the minutia and the arguing. Because a lot of the arguing is about, is it okay for Amazon to re-host other people's open source projects?

1:13:14You know, or is it cool or not cool? When you get caught in that, that's hard. And sadly, you know, it's easy to bucket things. I don't blame, you know, I tried to explain it in our nato. okay blog post, you know, and, and be very blunt and explicit about it. And we actually made a mistake of like our first blog post around the license change was bad because we were like, we just don't want to make a lot of noise about it or something like that. And we made a mistake of like not being explicit around why we're changing it. And a day afterward I wrote, you know, this not okay blog post. I think the social streams just like simple, simple messages and bucketing and then it kind of took a life of its own and it's very very hard to change what it once it's happened i don't blame any individual you know what i mean it's like i i get caught by it every once in a while sure but that but that's okay i think that's you know that's that's life and you have to figure out how to work through it if that makes sense right yeah well in light of all this i wish there was a faster way to mitigate or to adjudicate trademark disputes like it It shouldn't take three or four years of them to continue to muddy your trademark to get the trademark dispute adjudicated, right?

1:14:30I mean, obviously that's like now I'm going to utopia and stuff. Like that's not the real world. But wouldn't that have fixed everything if you could have just said stop using our trademark, rename it, and then they say no. And then you sue them and it takes three months and it's adjudicated. Wouldn't that have fixed it? You would have never had to relicense? I think it would have helped. Yeah. But I also think that getting to a point where you sue someone is never helped. You know what I mean? And I think truly, I think Amazon, they landed in a gold mine called Cloud. And they were running very, very fast for the first few years.

1:15:09And you just took Elasticsearch, provided a service and called it Amazon Elasticsearch. And then I think it was very hard for them to decide to change it. And they wanted to get, I don't know, it's hard for me to explain why they refused to do it. But you can see that they learned and they evolved and they didn't do that level of naming afterwards for other services that they took and provided as a service. And to be honest, like now we work really well with them. It's funny. It's like, it was very hard for us to work with Amazon on the marketplace, for example. Because when we would go to the marketplace and we would say like, please promote, let's work together to promote our Elasticsearch hostages.

1:15:47And they'll say, but there is already one. It's called Amazon Elasticsearch. So it's like even trying to work with them was hard because of the confusion, because the confusion was internal because Amazon is so big. So once the name thing got cleared up, it just unleashed also the opportunity to even start to work together properly as two companies. So do you think if they would have come out with a hosted version of Elasticsearch, just like they did, and they had called it Amazon OpenSearch and then the subtitles like Elasticsearch and AWS or something like that, so they could at least, then none of this would have happened?

1:16:25I think that would have been fine. I don't know. Maybe it's kind of like hard to know. Maybe we would have ended up doing something similar to what Grafana did and changed the license from Apache 2 to AGPL. So still remain within open source, but try to be more restrictive around whether Amazon really wants to take the software and provide it as a service. I'll get back to, I do think that there is an imbalance in building open source and hosting open source between the large cloud vendors and open source companies. And if we want more open source in the world, I think we need to figure out how to fix that imbalance.

1:17:03and that's a valid concern that companies have. And sometimes the way that I think people talk about it is very, it creates tensions like, oh, all of these like big IPO companies, they IPO, they were successful and now they're like changing their license because they have shareholders or something along those lines. But to be honest, I was never worried about this. I was worried about the five startups with two people that I'm talking every week that don't choose open source. you know or just like people that live there and go like well we'll never open source our software we don't want to happen to us what happened to elastic you know so it's like but you think agpl would will help strike that balance i hope so because i like being very empathetic to osi they have to deal with the ai stuff and that's tough uh and you know we donated money to osi and we're trying to help, but that stuff, and I think it's more urgent to figure out than approving SSPL or something along those lines.

1:18:06So I do see a path where AGPL is good enough as a solution, if that makes sense. Have you looked at the stuff that came after SSPL, like Sentry's new fair source licensing? Are these interesting to you? I think they're very interesting. They're very similar to Elastic License v2, and we're talking to all of them. It's still the, OSI will never approve these licenses, right? Because of the restrictions. Yeah, exactly. I think what MongoDB tried to do with SSPL is to actually create a very valid license that had a chance to get approved by OSI. And I think that's the path that I would want to take.

1:18:45Not like expand open source to allow for like 50 things get into it, but more around let's figure out how do we still keep the high value open source definition, term definition, and apply it to AI, apply it to enterprise software. Cool. Adam, any other thoughts? No, I was just drawing that thought. I was cool with it too. I was cool with the open source alternative to X, whatever that is. Well, I think anytime you call yourself an alternative, then it's pretty clearly no market confusion that you are that thing. Yes. You're an alternative to that thing. The part I was drawing the line to was less about the usage of the trademark, but more so this not coolness, I suppose, of, hey, Amazon don't re-host our thing, which you and I both had a misunderstanding about.

1:19:34But I think there's been other examples where XYZ open source says, hey, don't re-host my thing and compete with me. And that's almost the same where you just simply clone a proprietary closed source company and say, you know, to quote JJ, totally cool with it, 100 % cool with it. you know well cloning the product meaning like you look at how it works and you make something that works that way like i could clone riverside right now which is proprietary yes open source it would take a lot of effort for me to do that like they still had to put in all the work yes to build the cal to make the calendly clone you know and you gotta reverse engineer the whole thing basically i get it it's not a direct corollary but i was it was a fun experiment yeah so you take the end product and you build it versus just rehosting what was already out there like that's the easy button right just taking elastic search and offering it hosted requires very little effort but cloning something at least for now still requires a big lift yeah yeah i mean cloning close source and trying to replicate and re-implement well at least we go until we can go and chat gpt prompt something that says create me an open source version of x yeah exactly but yeah i think that that's true by the way the same is true you know it's like um When people say rug pull, and by the way, I totally understand where it's coming from because this is how it feels.

1:20:55And you never argue with what someone feels. You know what I mean? It's not fair because that's the first impact reaction and that's how they feel. And it's your responsibility to figure this out. But I think one of the beautiful things about open source is the ability to fork a project and the ability to change it and the ability for it to grow and become something else. And when you change their license, then someone can go and fork the project. And suddenly like a whole other project comes up. And you remember, I don't know what was like the best example from my old days was Jenkins. You remember there was like Hudson and then Jenkins came along, got forked and Jenkins became the thing and it worked.

1:21:42And we're seeing some examples of it today. And I think one of the, I think, points of reference on why when we changed the license, we still behaved as open source is that no fork got successful, including open search, because we still behaved like open source. We didn't suddenly put everything behind closed source. We didn't suddenly made everything commercial. And I think most users were fine with it the same way that most users are fine with Llama as an open source model. You know what I mean? And so it's like, and I think that's why it never happened to us. But yeah, maybe for other companies it might happen and it will be amazing if we can solve it.

1:22:23Well, we saw a lot of support behind Open Tofu in light of the HashiCorp realicense of Terraform. I'm not sure if that support has continued. You know, we don't really swim in that pool very often. And so is Open Tofu actually getting a foothold where it's going to become the new Terraform or is it not? We'll probably find out at this year's KubeCon or something. But that's one where there was a community fork that came out immediately, right? Versus what happened with Elasticsearch, which was Amazon created the fork, right? OpenSearch. Yeah. And I think Valky for Redis. Yeah, Valky. That's another good example.

1:23:01But this is where I think most users saw how we behaved in the next few days and they were like, nothing changed. and they were less worried. Obviously, a lot of the people that are in the details and go back and have opinions about open source were hurt justifyingly. I was hurt. But most users were like, it's fine. It's Elasticsearch. We know the team. We're talking to all the developers all the time. It's like nothing really changed. Gotcha. Cool. All right. Well, thanks. Thanks for sticking around. Plus, plus people. Thanks for supporting our work. Yes. And thanks for sticking around, Shay.

1:23:41Yeah, yeah, yeah. Happy to.