Databricks Founder Ion Stoica: Turning Academic Open Source into Startup Success

0:00In general, this was our approach. We are going to be aggressive also about partnerships, even though the partners could compete and overlap. Because you have to trust yourself that at least when it comes to Spark, you can build the best products. You're saying internally, well, you know, if someone else is building a better product for Spark, then we deserve to lose, right? So that's kind of always was the confidence that we can build the best products for Spark and eventually, if Spark we are going to win.

0:51Hi everyone, welcome to Training Data. Today we're excited to welcome Jan Stoika, Professor of Computer Science at UC Berkeley. and co -founder of both Databricks and AnyScale. He is a uniquely exceptional career as a leading professor and founder of companies at truly legendary scale. Today, we dig into questions like Databricks positioning in AI, how research projects like Spark and Ray have led to the founding of Databricks and AnyScale, how he ties his research projects closely to industry from day one, new projects out of his lab like VLLM, MGMGBT, LMSIS and Vekunah and what research fields he's thinking about next.

1:31John, thank you so much for joining us today. We're really excited to have you on the pod. To kick things off, we'd love to hear a little bit about where DataBricks aspires to fit into the overall ecosystem, especially with some of the recent launches. What are you personally most excited about? First, thanks for having me here. So, I think with DataBricks, alloys we wanted to provide the platform, an intent platform, which helps our customers get most of the value out of their data. And one of the best ways to get value out of the data is using this kind of new developments in AI, including Larry Language Model and everything else.

2:22And the one thing is that to note that this was all always our vision from day one. Actually Spark was created one of the main reasons to create it, but when we build it, it was to solve, to speed up classical machine learning algorithms, right? and to scale them up. Yeah. And so in some sense, for us, it's full circle. We started with AI with machine learning. It's a classic machine learning. And right now, we are back and doing more and more AI because to take advantage and to create value out of the data. So you mentioned that you were founded around enabling classical machine learning. How do you see the current AI moment as same and how do you see it as different?

3:15And I'm curious, what are the specific things you're doing for this moment in time, like the Mosaic acquisition, things like that? Yeah, so certainly the momentum around AI, it's on a different level today. Right? You just look at the investments out there, right? I should tell the big part of the story. So I think that obviously what the way we are looking at is that taking advantage and being successful is AI is not easy. AI ecosystem is growing in complexity. It's not just a simple call with a model. Right? You have so many techniques now like rag and raft and right. to improve the accuracy of your application using AI.

4:12Obviously, everyone is excited about AI because it solves so many problems, and there are so many, you know, generally so many headlines. But still, it's not what you want. You want a product of AI. A lot of things still we are seeing today are fantastic demos. Right? And demos are inspirational. And when people see demos, whatever charge -up it is, all this kind of problems, all impacts math problems, it's very easy to think that, wow, it's doing that, it's going to do everything. So, but going from demo to production, it's like I said, it's a big step. A demo means that you need to find an instance, at least an instance, to be really impressive.

5:00In production, is there exists such instance? But when you go from the demo to the product, when you have a product, it has to work for all cases. So that's kind of the big gap. So that's why the effort is to improve the accuracy, improve reliability, of course, eliminate hallucinations as much as you can. And really also to find where it provides the best value, you know, because you know, you can apply AI to 1 ,000 use cases, but which other use cases which are going to provide you the biggest value. I think that's what we are also trying to help our customers with. So to navigate that, how to successfully apply AI to their product, to their, to improve their services, to their business.

5:49Yes. In that vein, one of the most interesting things that I thought came with the Databricks AI launch was the new Open General Purpose LLAM created by Databricks. What was the reason behind training your own models, open sourcing them, and what do you think are some of the best use cases for that model? Yeah. So I think the main, if you look at like our main market, its enterprise and enterprise customers. They do have a lot of concerns about data privacy confidentiality and obviously the kind of one control is not only that, but our disability. Right? You want to be able to audit what data you use, what the results have been used and what the decision have been made, which is what data, the decision have been made.

6:37And so that's why in general the enterprises, everything being equal, migrate towards the open source models, we can host on their machines or in their VPC and so forth. So it's one of the reasons for releasing the outbreaks is to help our customers. And then the customers say, you know, in many of these cases, they start from this model and then find you on with their own data and To optimize for their particular use case, right? So and it's again, so our enterprise customers, they want, you know, open source model, they want kind of control as much visibility as they can. They want privacy and confidentiality, especially in the recent light of the breaches, which are widely published.

7:38The other thing is that the DNA of Databricks has been open source. It's not only Spark, but Delta and MFLO and many others. The other thing I thought was fascinating about deep -brex is its excellent programming abilities. What do you think has made it such a capable code model, especially even compared to something like CodeLama 70B? Look, I think it's about, obviously it's about data and how you train it. And it's one thing, I think, you know, one of the major Databricks advantage is Mozayk, right? We have also the entire infrastructure for training. or so we have not only about the data, but the entire infrastructure of training and fine tuning.

8:27And that is making it much easier and most cost effective to optimize models for different use cases. And obviously the co -pilot programmability, it's one of the very important use cases in this enterprises because software engineers are still very expensive. Yes, that's interesting. So making the people you have, and it's not only about that, it's like hiring top software engineering is very difficult. If you are a large company, like maybe, I don't know, 4G or a thing like that. And so making those people productive, it's extremely important and critical for their business. You mentioned Mosaic as a key part of the strategy.

9:16I guess what do you think are your most important chess pieces in this kind of AI battleground? I imagine Mosaic is one of them. And our most of your enterprise customers looking to train their own models. And how does Mosaic and your other oppositions fit in with your customer's needs? Yeah. So I think there are a few enterprises who would like, so it's a kind of a pre -training and then it's fine tuning and using the model on your own hardware, on your own VPC, on your own machines, you know, you rent it, you own it. So, and you may expect there are a few which are going for training, but there are still a few who are doing going to pre -training ones that data have enough data.

10:09A lot of them, they want to do fine tuning, right? Because it's a look, if you are an enterprise, right, and a company, right? And you want to improve your business, right? What do you have? What do you have and which other don't? The, the, the, we think you have, which other don't, It's the data, the data about your business, about your users. Right? Right? So, therefore, because this is something you have and other do not, you want to take advantage of that. Right? So, how do you take advantage of that? Again, there are many ways and you try different ways to do it. Right? One is about fine tuning, you use that data and you have an open source model, you find you know that.

10:52Right? the other one to use RAC, right, and seeing like that. And so you are going to do any of those. But then you want to do, like I said, in your own BPC, to preserve security, to have, you know, 10 -4 security perimeter, perimeter, you want to do it in order to protect confidentiality of your users. And obviously right now, GDPR, California, consumer privacy act and so forth, is there also a regulation? And the number of regulation is going to increase. So the fact that you can have having an open source models, you can find you and you can use your swag in your own VPC. It's very compelling value proposition.

11:42And are you finding that most of your customers want to go that approach versus, you know, OpenAI and the very powerful close source models in the market? I think enterprises and still we are still early on. And there are many enterprises obviously using OpenAI for different use cases, different applications. and I'm sure OpenAI and Microsoft Azure will come with new products to provide better confidentiality, better security. But at the end of the day, what I was saying is that everything being equal, right? As an enterprise, I will prefer more control. Security in strategic, right? It's like it's less kind of locking or thing like that, right?

12:39So I think that's what we are saying. So if as open source models are going to catch up with the proprietary models, in particular in the use cases which matter for the enterprises. It doesn't need to be perfect for all the particular use cases. It just need to be very competitive or it matters. Then enterprises will prefer these solutions on which they have more control and more secure. And how far away do you think we are from that moment? like do you think we're there today where all else is equal or when do you think we cross that that? You tell us of the open source, that's the proprietor.

13:24Yeah, open source being on par, all else being equal for the core use cases. So you have a lot of use cases, you know, it's again is the open source plus data. And right now the application are more complex, it's not just a call to a language model. You have this kind of, you know, it's very cold, you know, you compose, you have application which build from many components and it's not called compound AI. So actually it turns out that if you can build an applications for doing you know, recommendation or single agvato or, you know, for programming for a particular copilot for a particular tool, you can actually do better than even something like OpenAI on the latest chat GPT because you have more data.

14:32And the other things about, you know, data waste, which I think they're also announcement about that is that is a kind of unity catalog and and see like that you have access and you know also about the structure of the data which helps you tremendously to improve the accuracy of your applications right so is not only the model right it's everything else you have around it And the quality of the data you feed it in. So I think that's, it's a data stupid like you'd say, right? Like at the end of the day. It sounds like control and security are two primary areas of things that enterprises really care about that you've noticed.

15:23And Databricks obviously has a tremendous advantage with having access to data as well to help these companies use models. Control and security for the customers. Exactly. What about some other factors have you noticed that they also care about is how much does cost matter, how much does diversity of other models matter. I think obviously cost is important. And it's like this, right? First of all, you know, you want to, you know, it's initially the important thing is about the value. Right? Can you provide the value? Right? It is like, so that's the first thing. And at that stage, cost is not as important.

16:03Right? And in here actually, in some of these early stages where people also try to use the most powerful model like OpenAI and so on and so on. But once you cross that and you have a use case which you conclude that is good at value to your business. Now you want to scale it up, right? And now you are talking about having more control and more security and all of these things, protect confidentiality of the data, privacy of your users, and so forth. And now basically people consider how they are going to deploy it. And this is where having multiple choices and having, like you said, more control, security, it matters.

16:49And where open source models and platforms like beta bricks are very valuable. And again, it's also all the other components we share together in the beta brick, like I mentioned, you need to catalog and every single to increase the value of your applications. Yeah, super interesting. I'd love to talk about compound AI systems. I think you guys probably coined or popularized the term. And it seems like that's a lot of what the industry is latching on to now. Maybe for audience, can you explain what is a Compound AI system and what our enterprise is thinking about when they're building these? Yeah.

17:28So a Compound AI system is basically consists of multiple components, multiple calls to language models or agents. And it's very much you can think about when you write a program, you have multiple components. You have different function procedures to do different things. And then you then put them together to create the program. The same is very similar here. You can use maybe one model to parse the data to extract the data. You can use, and then you can use, for instance, depending of the prompt, you may use a model. You know, if the prompt is about mass problems, right, it's like you can use one model.

18:19If the prompt is about, maybe about programming, you may use a different model, right? And then you may use, for instance, about formatting the result, right? That's another one. You may use that, for instance, now more and more, you are talking about agents and with agents. you have to, you know, you call external services or functions like search, or you can use a calculator and things like that. So now you have different models, you can do a better job, and there are small models actually to take the prompt and convert it to a function call. So that's kind of what it is. But the way to think about it is like the conceptually, it's like your writer program has different components and make it easier to develop and deploy and manage the same thing you want to apply to applications.

19:19So a collection of smaller components that work together where some of the parts is greater than the one monolith that you're placing. I'd love to dive a little bit into the Databricks story quickly, which I think is an incredible legendary journey over the last decade. But with a lot of nuances that I think maybe many folks don't yet understand, from today it looks like you were building the right company, the right at the right place, at the right time. But the actual nuances that Databricks was originally started originally for data scientists. It happened to cater well to machine learning workloads because of all the work that you were doing in data.

20:00And over time, you made the right strategic decisions to actually really grow with the AI market. Can you share a little bit of some of the learnings and journey that got you to where Databricks was today? Yeah, I do think that very happy to. I do think that a lot of it's also about being the right time that I place and things like that. It's being lucky, or I think that's all of this is true. And there are many things, needs to go right to be successful. Some of the things you control, some of the things you do not control. When we started, indeed, we are focusing to build a cloud product, hostage product for data scientists.

20:46Like we have this kind of notebook, book and we have provided hosties Spark and we targeted data scientists. And we targeted data scientists. It was one of the reasons we targeted data scientists because again, we all have, like I mentioned, Spark early on, also targeting machine learning applications, machine learning workloads. And at that time there are not many data scientists. It was like 2013. However, when you look around, most of the universities already have data science programs, like degrees, you're starting to offer data science degrees and you're saying, okay, we are, it seems a good, you know, pass forward.

21:37A good market. And I remember you're looking at LinkedIn to see how many data centers there because it's not our users. Not initially, they're not as many. Thousands. Especially when you convert the data with database analysis, analysis, and so forth. Who started the build? And I think it was a reasonable good product. And then we started to grow quickly in terms of the number of customers. Initially we had small customers.

22:14But then, and still, the interactive analysis, the data science has been for a long time. It's actually one of the biggest workloads, in particular in terms of revenue. because the interactive workloads are price higher than batch workloads. But I remember that I sold to these companies to use data bricks and they are aspirationally buying it to the data science AI. And after a few months we look up to them because we didn't go much earlier, because we saw that they were doing well, they're usually just growing and everything seems good. So no reason to worry. So we went back to them and said, you know, to see what they were doing, maybe we can write a blog post or whatever they were doing, right, it's like marketing and everything.

23:15The surprises are very few are doing actually machine learning at that time. And we asked them what happens, right, or still. Well, it turns out that obviously, that in order to the machine learning, you need data, like we discussed early on. And they realize that for the particular applications they want is they don't have the data they need. So they need to now put maybe to add new logs, you know, to call like new logs in their products and things like that. And also they need to clean up the data, you know, curate the data and so forth. So now we are lucky because Spark was very good also for data engineering, right?

23:57For data processing, right? It's a data processing tool at the end of the day. And so they are using Spark for data engineering. And then obviously we start focusing also on serving data engineers much more than before. and just how we started. And then obviously when later now, you know, God's, you know, data engineering, right? And still, you know, have all these data scientists exploring and starting to build models. And then it is a very natural extension to start to add more products for our user, for our customers, like I mentioned, to get more value out of their data and that meant building machine learning models and using the open source models.

24:57Very interesting. I want to talk about the Databricks Microsoft partnership. I think that was the stuff of Legends. I think still probably one of the only case studies of a successful, like truly transformative partnership. Maybe walk us through What that partnership was? Was it a bet the company moment at the time? Do you think Databricks would have become what it became? If you hadn't struck that partnership, maybe just talk about that? Look, obviously, the partnership with Microsoft was a great partnership for us. The one thing I want to say is that, and this is most visible, we are very, from the day one, we are actually very focused on the partnerships.

25:42Our idea was always, you know, like we now we make Spark successful, you know, hopefully the fact of standard for data processing and then we make Databricks the best place to run Spark. So early on actually, you know, we started, you know, a few months in the life of the company we had this kind of partnership with Cloud Data. And then we had these Hortonworks. And these partnerships was mainly because you are, you know, to advance Spark, right? Because Spark, it was created in Hadoop ecosystem, right? And these are the Hadoop companies, right? And so we had this partnership, or data stacks and so forth.

26:31So this was like in the first year or two. So we had all this partnership. Despite the fact that some of these companies, we knew that they could become our competitors, right? Because, you know, just making them, is making and helping them to deploy, to manage and to sell Spark based services, right? It's like, so in some sense, you know, the Microsoft is like, it's, it's, it fits our, kind of, approach of trying to, you know, strike as much, you know, the partnerships with other, being aggressive, doing partnership with other, you know, organizations in the ecosystem. Even though, again, in some cases, it was not a clear cut whether we are, you know, they are going to compete with us or not.

27:32Right, so you are very just growing the ecosystem and growing spark that was the priority. We even had a point in the partnership with Snowflake. So it requires a lot of heavy lifting. So we always look for partnership which are meaningful. And I think great negotiation, Ali and so forth. Did a fantastic job there. But at the end of the day, we also need to commit to it. It's like it took to build your data bricks. So we are an IWS before that. It took tens of engineers one year, and you have a small company at that point. So it was a huge commitment and a huge bet also from our perspective to do that.

28:28And yeah, and we, you know, I think engineering and everyone executed very well and it was a successful product. Right. And, you know, Microsoft was great partners, you know. And yeah, this is what happens. We are obviously a bit lucky, but in general, this was our approach. we are going to be aggressive also about partnerships even though the partners could compete and overlap because you have to trust yourself that at least when it comes to Spark you can build the best products. You're kind of saying internally, well you know if someone else is building a better product for Spark then we deserve to lose right.

29:18So that's kind of always was the confidence that we can build the best products for Spark, and eventually if Spark wins, we are going to win. Do you think that Databricks will have become the company that is today without that Microsoft partnership? I think so. It maybe could have taken a little bit longer. Yeah. But yeah, I think so. Right, we would have still have a good, very good offering on Azure. Like we have with GCP, you'd have taken a bit longer. But I don't see any fundamental change in dynamics, right? Because one of the advantages of Databricks of course is one spark won, and we could provide the best product for spark.

30:04We are in a very strong position. And compare with other clouds, remember that one of our advantage, like every one else's advantage, like confluent and so forth, versus that, you can provide a service on multiple clouds, right? And multiple clouds is, you know, multi -cloud has been more and more very strategic for, especially for large enterprises. We do not want necessary to be locked in or to one, they want a choice. So yeah. I love the confidence and conviction that you have in Spark and your own execution ability is internally, But that married with the practicality and aggressiveness of winning as a business and pursuing the right partnerships and doing whatever it takes to win.

30:53Yeah, I mean, you try to simplify things. That's what I was saying. You know, you like, initially when you said, like, look, you know, like, we have to make Spark Queen. You know, there are many, I remember we look at all these combinations. It's like spark queens, the product fails, right? Or spark loses, but we have a product which is successful or boss fails. No, that's not very interesting or boss are successful, right? And we convince ourselves that, you know, we need to bet on spark to win because that's the most likely way also for the product. Again, for better or worse, right? But sometimes, there could be many ways to success, right?

31:39And in retrospect, you cannot go back and try other alternatives. Maybe there are better alternatives at that point. But it's important to commit to un -sync, which hopefully it's a reasonably good solution, a good password, right? Okay, and there are many passes to the peak, right? The most important is to go meet one which list is a peak, right? It's like not the shortest one or the easiest one may not be, but it has to be one together. And to the highest point. And that's how you said, okay, Spark has to, has to end, right? And we have to build, you know, the best, need to be the best place for Spark.

32:15And then we are saying, you know, to be the best place for data and AI. We need to eventually we knew and we assume that if you are to be hugely successful, you are going to go beyond Spark. Right? It's like that's why the name of the company, the Databricks is not Spark Labs or something like that. Yes. So that's kind of you try to simplify. So you do that and you start to execute. OK, so you want to make very successful as an open source. So you want everyone to use it. So that's how you do Cloud era and try Horton also and so forth to do this Spark. Because there's a time there are other solutions.