Naveen Rao: Inside the Legendary Databricks Acquisition of MosaicML

0:04Hey everyone and welcome to Generative Now. I am Michael Mignano. I'm a partner at Lightspeed And this week, I'm talking to Naveen Rao, co-founder of Mosaic ML, and now the VP of Generative AI at Databricks, where Generative AI is used to lower cost, assure production quality, and give complete ownership to their clients' needs. I had a great conversation with Naveen, talking about the acquisition of Mosaic by Databricks and what the integration has been like since. So take a listen to this conversation I had with Naveen Rao. Hey, Naveen. Hey there. Hey, Mike. Good to see you. You too. Thanks for doing this.

0:43Absolutely. Glad to be here. Yeah, this is one I've been looking forward to. So, you know, really appreciate you agreeing to do this. The story of Mosaic ML and Databricks, you know, definitely becoming something of a legendary story in the very short but exciting past couple of two years or two years or so of generative AI. So I've been really, really looking forward to digging into it. Maybe just to start, because you have this incredible background that I feel like maybe some people don't know about because they've been so focused on Mosaic and what's happened with Mosaic over the past couple of years.

1:20But tell us a little bit about what came before that. I mean, you had an incredible career at Intel, and you're an academic as well. I believe I also heard, are you like a race car driver or something too? I am, yeah. Yeah, this year, actually, I did the 24 hours a month. Yeah, I think, you know, from the AI perspective, this is something that's been a long-term interest. This is not something I just got into recently, even in the mid-90s when I was an undergrad dating myself there. But, you know, I was doing work on neuromorphic computing and, you know, really like the intersection of what animals and human brains do and what we can do in the sort of synthetic computing world.

2:08And really, it was always fascinating to me that we are so far away. And we're still very far away, by the way. I guess I have a lot of context on this. I've been doing it for a long time. And when I hear people say, oh, you know, LLMs, we've solved intelligence. I'm like, guys, no, we're not even close. Like there's still a long gulf between what animal brains do and what like an LLM does. And then not to mention the efficiency part of it. I think people don't really remember the fact that our brains run on 20 watts of energy. Everything we are, everything we've learned, everything we will ever be all runs on 20 watts.

2:45That's a far cry from what we see now. Elon's talking about 100 ,000 H100s. That's hundreds of megawatts. This is 150 megawatts, I think. So we're still pretty far away. I don't think it's going to be a simple paradigm of scaling one dimension and we achieve intelligence. Now, because of that interest, I worked in tech through the dot-com bubble, kind of went away from the AI stuff back then and works in whatever application stacks were being built back then. It was like, this is before Wi-Fi and wireless was coming on board, video compression, like lots of different areas. I actually started my initial career at Sun Microsystems doing like CPU design.

3:32Oh, wow. So after like 10 years in industry, I'd built a few computers. I knew how to build chips, you know, application-specific chips. I said it was kind of time to go back and think about that interest I had, the long-term interest I had. And so I actually quit my job and went back to get a PhD in computational neuroscience. ends. And, uh, you know, it was a little bit of a nuts move at that point. I mean, I had a family, I had a one and a half kids and, you know, all that stuff. Uh, so it was like something I did for myself. It was interest. And, uh, you know, maybe it'd be interesting to the world.

4:06Who knows, right? I, I sort of just did it because I wanted to, and I was like, I wanted to have no regrets. You know, I kind of, I actually said this, my friends are like, look, I want to look back when I'm 80 or 90 years old. And so like, I did something that, that I thought was interesting and maybe mattered. So yeah, I took a chance when did it. I loved it. Computational neuroscience, quick definition of that for the listeners. Yeah. I mean, it actually neuroscience is a hard field to define because it spans everything from genetics, proteomics, all the way through applied math. I'm much more on the side of applied math.

4:38It's actually machine learning and computational neuroscience actually have a lot of similar underpinnings. Neural networks that we use today got their inspiration from neuronal networks. And so really it's, it's how brains compute the mathematical underpinnings of how brains compute. So it was really the main question, you know, how, how do brains compute? And yeah, I, after doing a PhD and some research in this and training a bunch of monkeys and all that stuff, I mean, I don't know that we have a definitive answer, but we definitely know that things are quite a bit different from what, uh, what we do in the synthetic computing world.

5:15So I took some inspiration from that. I actually went and got a job as a researcher at Qualcomm doing neuromorphic architectures. And that's right around when deep learning really started to work. And I kind of recognized early on that it was much more than just like, you know, some sort of intellectual interest. It's actually a new way to think about computing. And that's when I left and started Nirvana, which was actually the first AI chip company in this modern wave. And we really thought about the architecture from a tensor standpoint, a matrix multiplication standpoint. We were acquired by Intel actually just two and a half years after starting the company.

5:57I then started a new division inside of Intel, grew the group into this large division, had multiple product lines. It was interesting to go through that. I learned a ton. There are lots of hardships along the way, we'll call it that. Like the integration hardships? or? Yeah. I mean, I learned a lot about integration, honestly. I would say what I learned more than anything else, I mean, definitely some stuff about technology and business, but much more about people. How to get an org, a big org, not like a startup where it's like 30 or 40 people, but like thousands of people. My group with direct and indirect reports is something 2 ,500, 3 ,000 people.

6:35So getting that alignment to happen in such a big company and Intel itself was 110 ,000 people. Wow. It was hard. So we did turn the Titanic to some degree. I don't know if we turned it enough. I mean, that's the other problem is, without getting into too many details, as you know, if you've been through acquisitions, like in big companies, there are a lot of politics that go on. And there's a lot of things that aren't really aligned with the goal of delivering products. And I think as a startup founder, what do you care about? You care about delivering products. Like, I actually, I couldn't give a shit about my title.

7:07I don't care about any of those things. I want to deliver my products. And I think that's a fundamentally different motivation than a lot of people who've been in large companies. I'm not saying it's a bad thing or not. It's just different. We're all wired a little bit differently. And I think that creates some frictions, which you have to manage. If you're more experienced, you'll understand that's going to come and manage it. And I think that was some of the things I could have done better in that role. But, you know, we did actually move the needle, shipped a bunch of products, changed the emphasis of the company, which I think was a big success.

7:41The company could have moved faster, obviously. And NVIDIA is now like clearly the clear winner. But we really kicked NVIDIA into high gear in terms of their execution. They are a formidable competitor. And, you know, once the acquisition happened, they shifted to this new architecture of Tensor cores really, really fast. I mean, it was impressive, honestly, to see. So, yeah, there's a reason why they are where they are right now. Yeah, indirectly, maybe you and your team. Yeah, I mean, I think it was part of it. It was a trend and they sensed, you know, I think as a founder, like you're sensitive to competitors coming after your holy grail, right?

8:21And Jensen is no exception to that. You know, fiercely competitive, probably smelled that that was going to happen and just said, okay, let's go and really just moved everything in one direction and made it happen. So it's impressive for sure. Sure. So, okay. So you're at Intel. You said the acquisition happened only two and a half years, which is, I didn't realize it was that quickly. We're going to talk about another quick, quick acquisition, but how long were you at Intel and how do you go from Intel to eventually Mosaic? Yeah, I was at Intel about three and a half years, I think. Yeah. Okay.

8:57I quit in early 2020, right before the pandemic started, actually had nothing to do with the pandemic, just happened to be when I was ready to move on. And I started thinking about what's next. I took a little bit of time off, honestly, not a huge amount because I stopped Intel in March of 2020 and started Mosaic basically January of 21. But there was work leading up to starting it. So yeah, I took a few months off, was really thinking about where we can have a big impact. And hardware at that time was, I saw it as like, okay, well, NVIDIA is executing really well. What can we do that will actually have a bigger impact than hardware right now?

9:36And it came down to, can we be much smarter about how we use hardware, how we use every computing flop and decrease the cost and accessibility of AI? You know, we came to the conclusion after I reached out to Jonathan Frankel, who was an academic. He was actually a PhD student and his advisor, Mike Carbon. They published a paper called the lottery ticket hypothesis, which kind of laid out the framework for some of these inefficiencies in training neural networks. And so we just got to talking over the summer. Everyone was at home. We were just on Zoom calls. And yeah, we came to the conclusion we could actually find 510x.

10:11and a few years on, when we look back, we actually show that kind of learning per dollar is increasing on an order of 4X per year. Internally, we call this mosaics law where basically an LLM or a neural network of a certain capability today will cost one fourth in a year to train. And that's due to algorithmic innovation, software innovations, and hardware. It's like the whole stack. So it's actually quite incredible. It's no surprise to me that generative AI has moved very fast because things are based on this exponential of 4x per year. It's not Moore's Law, which was like 40 % per year, right?

10:50Moore's Law is 2x every two years. It's about 40 % per year. What was the lottery ticket hypothesis? Yeah, so basically this showed that there are representations formed within a neural network while it's training, while it's learning from a data set. And his research is all about how those representations form, how stable they are, how modifiable they are through time of training. But the lottery ticket basically says you may start at one initialization point and you may have a lottery ticket, which leads to a good outcome. Initialization matters, you know. And so or that there's a essentially a small neural network inside of a big one that is doing all the work.

11:32That's sort of the lottery tickets. if you have to get to that point. But it's actually like there's a lot of over-parametrization in a neural network. You may have 100 billion parameters, but really a billion or so are doing most of the work. And so it showed that there's an inherent inefficiency. It's somewhat required. I mean, you can't learn some of these representations unless you have some of those over-representation of parameters. But are there ways we can exploit that to use less compute overall? And with Mosaic's law, how consistent has that been? Like that's basically been since 2020 or has that been happening well before?

12:14And how far do you see that going? Yeah, great question. For the first part, yeah, I think, you know, definitely since the LLM era, which I would call a transformer architecture, which is 2017, 2018. We've seen something around that number about 4X. How long will it go? I mean, I think we have line of sight for another, call it three years. I wouldn't be surprised if it ends in six, seven years, but there's got to be a paradigm shift. I don't think that the way we train neural networks today is necessarily absolutely correct. There's a lot of points of evidence why that's true. But, you know, we'll get on to something else and it may even be faster.

12:58Who knows? Maybe a faster law. Yeah, I was going to say, like, there are probably things that could accelerate that, right? Like a new chip architecture or, you know, something beyond transformers, right? A new type of model. That's right. Or it could slow down, right? I mean, like data, I don't know, could like, you know, we run out of data or something and now it slows down. Is that possible? Well, this run out of data thing is interesting because humans learn on much less data than these LLMs do, three orders of magnitude or four orders of magnitude less. So I think we need to be smarter about how we use data.

13:36It's not about quantity, it's about quality. And yeah, I think you're right. There could be some breakthroughs in hardware, breakthroughs in architecture, breakthroughs in learning paradigms. So, yeah, I think there's a lot coming still. My view is that it's going to be an acceleration to that 4X. And the evidence for that, I would say, is if you look through evolution, we have seen these kind of point accelerations. There's sort of this unsolved mystery, if you will, in anthropology about why humans kind of took off. I mean, humans haven't really been around as a species very long, like 200 ,000 years.

14:13And, you know, we've had profound changes in intelligence and capabilities. You know, if you look at dinosaurs, it was 65 million years ago they died. And humans have been around for 200 ,000 years. So there was this huge step function of capability. Why did that happen? I mean, you can make a lot of different arguments, but I mean, we were able to communicate. So once we were able to talk, make lots of different sounds and have rich representations in our speech, we can now communicate. We can use multiple brains together. that was an unlock for acceleration. So, you know, maybe there are other capabilities inside of our brain that were these kinds of unlock.

14:45So I think we see a parallel evolution happening in, in synthetic compensation, which is actually, you'll see the step function of acceleration. And I think we have a number of accelerations left. Yeah. The, the analogy that comes to mind for me, when you talk about talking and speaking, being, being the unlock for humans is maybe this notion of synthetic data as a source of learning for models? I mean, I actually think there's been a couple recent breakthroughs there. Like what's your whole take on this synthetic data hypothesis? Yeah, I think some of it is like everything in Gen AI, it's overblown, but there is something real about it.

15:23Yeah, of course, of course. I'll put it that way. I think synthetic data allows you to essentially explore a space of different distributions layered together. So you can create more mashups and essentially allow your LLM to be exposed to more variation. And that generally aids in learning. What it doesn't do is provide net new information about the world. I think observation, interacting with the world, can't really be replaced with synthetic data, unless your synthetic data is somehow simulating the real world. That's a different kind of synthetic data. I think there are also multiple versions of synthetic data, like the kind where I run a neural network alongside a simulator of the world.

16:06Actually, I think that sort of synthetic data can lead to net new representation. But the kind where I basically train at LLM, a big one, and then produce lots of outputs from that based on training data set, I'm not sure that's creating net new representations of the world. Right. It sort of sounds like we're just like recycling the same thing over and over and over. A copy of a copy of a copy of a copy. Yeah. Yeah. There was a paper recently, I actually shared it on my Twitter about this. And actually it shows this idea of taking some neural network training on data and then distilling that by producing a bunch of data from it and then training another neural network is actually destructive in terms of information.

16:47Interesting. So it's sort of the copy of a copy of a copy idea, right? You kind of lose fidelity to the original distribution. you know right it's fascinating so so maybe back to mosaic so you know we went off some tangents there you're kind of explaining the genesis of the company and what you guys were focused on it was 2020 and we started in earnest like january of 21 and uh we're sort of exploring this idea that if i can lead to higher efficiency and i can make things easier to use i can have more people building and so we're naturally very inclined toward the open source movements democratization of these capabilities.

17:21And the reason I believe that actually, it's funny because now it's become front and center, the whole safety argument, very much believe that if I have many people building, I create stability and safety. Basically, if I can build neural networks that have biases that I either consciously or unconsciously build in and other people do the same thing, we actually end up in a place where we have some adversarial kind of interactions and actually things are kind of stable. Things become unstable when we have monocultures. And that's actually why I was kind of against this idea of those capabilities.

17:55I'm not against it from a market standpoint. I think any company should be able to do what they want to do. But from a safety standpoint, I actually think open source AI creates safer models on the net. Anyway, so we want to bring these capabilities to many people and do it through a set of tools that are very developer friendly. And this is what we built at Mosaic was a platform where companies could build from scratch, could fine tune, could serve models and do it at sort of the Pareto frontier of efficiency. And that actually started to take off very quickly in 2022, 2023. We had a pretty fast rise in revenue.

18:39At the beginning of 2023, we're somewhere around a million ARR. And I think by the middle, right before the acquisition, we were close to 20 million. So in six months, things just blew up. It was really like myself and one other sales guy. We hired a couple other people right along that point in time. So it was busy. When Allie came and we started talking, it was clear to me that their customers were similar to our customers. And we need to build a better channel into those customers. I mean, you're a founder yourself and you start to realize that product matters. You want to ship products. It has to be a great product.

19:16But if you don't have the channel, all of that is for naught. You have to be able to sell it. So 20 million ARR is great. We're very happy about this. But at the same time, how are you going to get to 100? How are you going to get to 300? How are you going to get to a billion? Channels matter a lot. And so we're kind of planning ahead. How do we do that? Maybe before we get to that, because I definitely want to dive into that story. I think you just said something interesting a few minutes ago about how open source potentially being the key to safety, right? If you really democratize this thing, you make it something that everyone can do.

19:45You sort of level the playing field. Whereas if you just have a couple of centralized players, you know, you end up in a much more risky position. Interestingly enough, that whole premise that sounds like, you know, was one of the hypotheses for the company. Zuckerberg just came out a couple of days ago with the llama release. I think he wrote like an essay and this was one of the core sort of tenets of the essay, which is, hey, open source is actually the key to safety. China probably already has all of the closed source or will have like access to all of our closed source models. So let's level the playing field.

20:20So that, you know, that hypothesis is now turned out to be something I think that is very well believed and becoming more and more of the norm. How have you seen that sort of play out since since sort of putting the stake in the ground with Mosaic? I mean, I think it's still playing out. It's maybe too hard to say one way or another. But yeah, I think there's, I don't know, it's weird because in this field, running an LLM, yeah, you need GPUs, but like, it's mostly about knowledge and knowledge distribution happens very fast. So, you know, building semiconductors is hard. Like you need a ton of equipment.

20:55There's a lot of like tribal knowledge in people's heads on how to do it well and make that equipment work. You're working in the physical world. With LLMs, you're not really working in the physical world. It's in software and algorithms. And I don't know, I thought there was a hubris from the West a little bit there. It's like, oh, we're so far ahead. It's like, guys, what the hell are you talking about? Like China has 1.4 billion people. India has 1.4 billion people. Do you really think they're not going to be able to figure this out from reading a few papers? I mean, it was kind of silly. So yeah, I think everyone's kind of a parody.

21:26Like even two years ago, I don't think anyone was more than maybe a year and a half ahead. OpenAI is probably the only one that was maybe a year and a half ahead. Everyone else was within six months. So we're not talking leads of 10 years here. We're talking small leads. And really that is collapse in my mind. I mean, partially because of Zuck and a lot of models. Now we're at a place where I actually don't... I think there's a number of places that you can train a state-of-the-art LLM. I mean, it's a matter of whether you want to do it for your application. Are the costs justified but i think a lot of people can do it now so that capability is out of the bag yeah i wonder if like the real differentiator is you know maybe for some of these bigger bigger players is in the vertical integration right i know you know as an example elon's kind of like doing the whole thing right he's got the data center what they just launched with xai obviously got their own models that they're training up they've got the distribution with twitter like yeah it's it's all a question which way do you want to go do you want to go vertical or do you want to go really hard is off.

22:32I mean, this is not atypical for new technology transitions. I mean, people sort of like, they grow up around the dark for a while, right? Before you figure out exactly where the traction is and how people use it. So I think companies like OpenAI and other, call it infra providers like ourselves. I mean, we're all still figuring it out. How do people use this stuff? And enterprise AI is, yeah, if you're going to go horizontal, you need to basically hit the developers and the people who are building the stuff within companies. And if you're going to go vertical, you got to specialize and be really, really good at it.

23:07So you start talking to Ali at Databricks and you realize that there's an opportunity for the channels, like you said, and distribution. I heard a story. I don't know if it's true. Maybe you can either verify or debunk it, that this all went down at the Cerebral Valley conference. Is that what happened? Tell us the story about how the acquisition came together. Yeah, actually, it is true. I met him there. I think it was April of 23. That's the first time I'd met Ali. I never actually... We talked to Databricks, like the CorpDev and partnership teams. I mean, frankly, what we wanted was, hey, can we get access to Databricks' customers?

23:46That's what I cared about. As a founder, I was like, all right, can I get access to them? Can I form some sort of partnership? And we were also talking to Snowflake. We were talking to everyone at that time. I mean, that's what we should have been doing. And I talked to Allie, yeah, at this quote unquote VIP dinner afterward. And Clem was there from Hugging Face as well, whom I've known for a little while. And it was funny because I talked to Allie right away and I was like, I'm pretty sure he's thinking about buying us. I actually said this to Clem at the dinner. And I was not quite there yet at that point.

24:21But I got Allie's info and we We just started chatting about stuff. I mean, he was giving me some advice about going after government contracts, that kind of a thing. Is that a good idea or not? We just kept in touch. And yeah, I think it was May, like early May where we started actually. That's when he kind of quote unquote popped the question about it. And I kind of knew that was coming. So we formulated a response already. Like, yeah, that could work, but we need to figure out the economics of it all. The only reason I even entertained it was because I did see a pretty strong cultural alignment between Databricks.

25:00In fact, in our pitch, when I talked to investors, I was like, look, we're going to be kind of Databricks of AI here because we're open source. We're a bunch of academic minded people. We're going to sell an enterprise. It's a similar motion. We even kind of copied the idea of running within your VPC, running software within the VPC, which subsequently we're moving away from now as Databricks. But we modeled a little bit after Databricks. So I was like, all right, there's definitely a lot of cultural alignment here. So maybe we can make this work. And from my prior experience, I found that cultural alignment is actually extremely important.