The Future is Small Models, with Matei Zaharia, CTO of Databricks

0:06So we really wanted to see whether it's possible to democratize this and to let people build their own models with their own data without sending it to some centralized. provider that's trying to sort of learn from everyone's data and, you know, kind of control their destiny in this space.

0:26This is the No Priors podcast. I'm Sarah Goa. I'm Alad Gail. We invest in, advise, and help start technology companies. In this podcast, we're talking with the leading founders and researchers in AI about the biggest questions.

0:45If you have$30, a few hours, and one server, then you're ready to create a chat GPT-like model that can do what's known as instruction following. The latest launch, DALI from Databricks, which is available in open source, foreshadows a potential move in the industry towards smaller and more accessible, but extremely capable AIs. Matej Zaharia, co-founder and chief technologist at Databricks, is here to tell us all about DALI. We'll talk about how big data sets actually need to be, why manual annotations becoming less necessary to train some models, and how you went from a Berkeley PhD student with a little project you may have heard of called Spark to the founder of a company that's now critical data infrastructure that's increasingly moving into AI.

1:20Welcome to the podcast, Matei. Thanks a lot. Excited to be here. Can you start by telling us a little bit about the origins of Databricks and how it led you to where you are today? Sure. Yeah. So Databricks started from a group of seven researchers at UC Berkeley back in 2013. and we were really excited about democratizing basically the use of large data sets and of machine learning. So we had seen, you know, the web companies at the time were very successful with these things, but most other companies, you know, most other organizations, things like scientific labs and so on weren't. And we were really excited to look at making it easier to do computation on large amounts of data and also to do machine learning at scale with the latest algorithms.

2:02So we had started, you know, during our research, we worked with some of the web companies. We also started open source projects like most notably Apache Spark, which, you know, was essentially, you know, the first version of it was my PhD thesis. And we had seen a lot of interest in these. And we thought, you know, it would be great to start a company to really reach enterprises and make this type of thing much better and, you know, actually allow other companies to use this stuff. Can you just give us a sense of what Databricks looks like today from a scale and product suite perspective? Sure, yeah.

2:33So Databricks offers a pretty comprehensive data and ML platform in the cloud. It runs on top of the three major cloud providers, Amazon, Microsoft, and Google. And it includes support for data engineering, data warehousing, machine learning. And most interestingly, all this is integrated into one product. So for example, you can have one definition of your business metric that you use in your BI dashboards and the same exact definition is used as a feature in machine learning. And you don't have this drift or copying data, and you can just kind of go back and forth between these worlds. The company has about 6 ,000 employees now.

3:11And last year, we said that we cost a billion dollars in ARR, and we're continuing to go. It's a consumption-based cloud model where customers that are successful can go over time and bring in new use cases and so on. Did you think the opportunity was as big as it has been when you started the company? Yeah, well, we definitely didn't anticipate necessarily to go to this size, right? A lot of things can go wrong, but we were excited about the confluence of a few trends. So first of all, it's so easy to collect large amounts of data and people are doing it automatically in many industries. And second, cloud computing makes it possible to scale up very quickly, do experiments, scale down and so on, which enables more companies to work with this kind of thing.

3:57And then the third one was machine learning. So we thought, you know, these are powerful trends. And the exciting thing for, you know, us as a company is we didn't invent cloud computing. We didn't necessarily invent big data or anything, but we were able to start at a point in time when many companies were thinking to move into this space and just provide a great platform for that. And there's this migration already happening. And, you know, if you provide the best platform as people are migrating to the cloud, they'll consider it. You still keep roots in research. You have a research group at Stanford.

4:28Can you talk about that? Yeah. So I'm a computer science professor there. So I split my time between that and Databricks. And we work on a bunch of things. We usually like looking farther ahead into the future. And we've worked a lot on scalable systems for machine learning, how to do efficient training on lots of GPUs and stuff like that, or how to do efficient serving. And then another thing I'm really excited about that we started about three years ago is looking at knowledge intensive applications where you combine a language model with something like a search engine or an API you call or something like that.

5:05And you try to produce a correct result, maybe for a complicated task, like do a literature survey and then like tell me, you know, what you found about this thing with a bunch of references or counter arguments or whatever. And I have a great group of PhD students that are working on that and, you know, exploring different ways to do it. How did Databricks decide to start working on Dolly? What sparked that? And how did you first get going on that? Yeah, so we'd had customers working with large language models of various forms, even before ChatGPD came out, but they were doing the more standard things like translation or sentiment analysis or things like that.

5:41A lot of them were tuning models for their specific domains. I think we had like almost a thousand customers that were using these in some form. But then when ChatGPD came out in November, it got people interested in using these for a lot more than just analyzing a bit of data and instead creating entire new interfaces or new types of computer applications, new experiences in them. And so there was intense interest in this, even at a time when the industry in general is being conscious about spending and which things are really required and so on. This was an exciting one. And the really exciting thing about ChatGPD, as you both know, is the instruction following or basically the ability of it to kind of carry on a conversation and listen to the things you're telling it to do and do those, as opposed to just completing text or just telling you a small amount of information like this is a positive or negative sentiment.

6:34So we really wanted to see whether it's possible to democratize this and to let people build their own models with their own data without sending it to some centralized provider that's trying to sort of learn from everyone's data and kind of control their destiny in this space. We were exploring different ways of doing it. And in particular, like Dolly is partly based on this great result from some other faculty members at Stanford called Alpaca, where they tested a way to, you know, basically they use the model to generate a bunch of realistic conversations. And then they use this to train another model that can now kind of carry on conversation on its own.

7:12And so we tried essentially cloning that approach, but starting with an open source model and it actually worked pretty well. And so that's how it became Dolly. But yeah, we've been looking at this space for a while and seen incredible demand for these kinds of applications. Yeah, I think the industry has really been very focused on scaling data, parameter size, and flops. And I think you all really have showcased the power of instruction following, even at something that's relatively smaller scale. Could you explain that and how that all works? It's very interesting. And I think there's actually a lot of research still to be done here because these models have been mostly locked up in these very large companies for a while, and everyone thought it's too hard to reproduce them.

7:53So the interesting thing is, you know, language models had existed for a while, you basically trained them to complete words, you know, here's a missing word in the text, can you fill it in? And then at the beginning, when people tried to apply them to real applications, not just, you know, I erased a word on my homework, like fill it back in, but like actual applications, They had always done various ways of, you know, training something else on top of, you know, say the feature representation in these. And so there was a lot of domain-specific work, but you could build like a sentiment classifier or stuff like that.

8:25Is it positive or negative? Probably like three years ago now. OpenAI published a GPD-3 paper, which is called Language Models Are Few-Shot Learners. And they said, number one, like we trained a language model to 175 billion parameters. And we trained it on, I think it's like 45 terabytes of text. So lots of data, lots of parameters. And it's like pretty good at language modeling. And then number two, they said, you can actually kind of prompt this with a few examples of a task and it picks up on the task and does it. So lots of people were working on that. You know, how do you prompt it? What's the best example to show?

9:02But everyone assumed that for that capability, you need a giant model to begin with. So even the researchers in academia were calling into GPD-3 and trying to build stuff based on it and study this phenomenon. And then last year, 2022, OpenAI published this other paper, which was sort of instruction tuning these models, where they said, hey, we used some human feedback and then some reinforcement learning, and we got this GPD-3 model to actually just listen to one instruction. It doesn't need a complicated prompt with lots of examples, and it kind of works. And then they released a version of this at ChatGPD.

9:39So I think in a lot of people's minds, the scientific view of it was, first, you need a giant model, and then you need this reinforcement learning thing. And only then do you get this conversational capability and broad world knowledge. So it's actually very surprising. In Alpaca, we just had a larger data set of human-like conversations. And we had this very kind of modest-sized open source model that's only 6 billion parameters. only trained on less than one terabyte of text. So like 50 times less data than GPD-3. And it still has this behavior. I think it's been pretty surprising to a lot of researchers, the size of model that still gets you this kind of instruction following ability.

10:22So I think this kind of an open research problem, like what exactly about these data sets is it that makes them good at this? What are the limitations? Are there tasks that these are clearly worse at or better had. It's actually kind of hard to evaluate with long answers because it's hard to like automatically score them and say, you know, like this is a good Seinfeld skit that you generated and this is like a bad, you know, Barack Obama speech. But I think we'll figure this out. Yeah. Were there anything that emerged from the model that you also found surprising? Like you mentioned one aspect of it, just in terms of the approach you took and, you know, with dramatically more limited data and approach, you ended up with really performant behavior.

11:01Were there other things that were unexpected properties of what you did with Dolly? Yeah, I think to me, the most interesting thing is it's surprisingly good at just freeform, like kind of fluent text generation. So you can tell it to like create a story or create a tweet or create a scientific paper abstract. And it does a pretty good job at that. And before that, whenever I talked to my, you know, NLP like researcher friends, they thought that that creativity was the thing that required a lot of parameters from something like GPD3. They actually told me, oh, the knowledge-intensive stuff, like remembering facts, tell me the capital of France and whatever.

11:39That's not surprising that a small model with a few parameters can do it. But the creativity, that's really hard. So this one is actually pretty good at the creativity and generation. It's less good at remembering lots of facts, which kind of makes sense given the parameters. So if you ask it about common topics, it'll be good. If you ask it like the author of a book, it might give the wrong one. I think we had an example because we've actually been building a slightly bigger version of this, too. And we had this question with, like, who is the author of Snow Crash, which is Neil Stevenson. And the initial Dolly model said Neil Gaiman.

12:13So, you know, it's still a Neil. It's still an author, but it's still not Neil. A sci-fi writer. Yeah. So it's less good at remembering facts, but pretty good at coherent sort of generation. Yeah. The name Dolly basically references the first cloned mammal, Dolly the sheep. Can you explain the reference within the AI space? Yeah, so it's based on cloning this other model from Stanford called Alpaca, but doing it with an open data set. And that itself was based on something that Meta released, I think maybe three weeks ago or less, called Llama, which is they took a modest size model, 7 billion parameters, and they trained it on a ton of data.

12:50I think they said 1.4 trillion tokens or something like that, which is, I don't know how many bytes of data it was, but it was multiple terabytes of data, basically. And they said, hey, by just training this for longer, we got a small model that's actually producing pretty high quality content for its size. So there were all these kind of, you know, woolly sort of animals out there. And we thought it's just too perfect to like clone it. And there are all these other things like, you know, it's like the Dalai Lama. I don't know. There are all these like things. Yeah. So yeah, that was a great name.

13:21That's a good name. Yeah. Are there other things that you can share that you all have coming in the background at Databricks or at your Stanford lab in terms of this more general area of language models? Yeah, I mean, at Databricks, definitely, you know, we're using everything we learned from Dolly and we're learning from our customers to, you know, to just offer a great suite of tools for training and operating LLM applications. We already have a popular MLOps platform, and we also have this open source project called MLflow that integrates with a lot of tools out there that our offering is built around.

13:53So you can expect some nice integrations into that. You know, separately, we're also working on Databricks product features that use language models internally, and learning a lot from developing those and, you know, feeding that into our product. So I think in the next few months, you can expect it. And we also have this big user conference Data AI Summit coming up in June that will probably have, you know, a lot of stuff about this. And I would say, you know, as a researcher, and also kind of with my Databricks hat on, the thing I'm most excited about is really connecting these models with reliable data sources and making them really produce reliable results.

14:29Because if you use ChatGPD or GPD4, the two big problems with it are, number one, the knowledge is not up to date. It only knows stuff it was trained on. And number two, a lot of the things it says are inaccurate and it's confident, but like wrong in various ways. And I think you can tackle both of these by combining some kind of language model with a system that pulls out vetted data, either from documents like a search engine or from APIs and tables and stuff like that inside your company. For example, when I talk to the chatbot in my bank, it should know my latest bank account balance and transactions and stuff.

15:11If I'm like, can you cancel the payment I made because I unsubscribed, you should just know what that means. So cracking how exactly to do that isn't easy. It may actually be easier with small models than with big ones to reduce hallucination from them. But I think it's still an open question. But I think if we can figure this out, then these become a much more reliable component in an application. Maybe we'll go from there to just like projecting a little bit about like architecture and research. You know, so much of the industry is focused on model scaling, right? Improving reasoning that way.

15:43How much do you think that matters in terms of, I guess, real-world usage in production with your customers in the near term? Yeah, great question. So to me, at least the relationship between scale of the model versus quality of the data and supervision you put in versus design of an application around it and those things and overall quality, I think the relationship is not 100 % clear yet. To get a really reliable model that, say, I don't know, can make a pharmacy prescription or something like that, maybe you need a trillion parameters. Maybe you actually need a really carefully designed data set and supervision process, which is kind of traditional sort of ML engineering type work.

16:29Or maybe you actually need a clever application where you're chaining together a couple of models and things and you're saying, well, does this make sense? Can I find a reference? Can I show this example to a human if it's really hard? So I think it's a little bit open. The thing I can say for sure, especially and Dolly and like other, you know, results like this really highlighted is it does seem that the core tech is getting commoditized very quickly. So if you just want to run, you know, something like today's chat GPD, it will be a lot cheaper because all these hardware manufacturers are building devices that are specialized and much cheaper.

17:05And another thing that's making it less expensive is we're figuring out ways to get a smaller model with less data, fewer parameters and stuff to get similar performance. So that, I think, is happening faster than at least I would have thought a few months ago. So at least to get something with today's capabilities, I think it will be very affordable and you might just be able to run it locally on your phone or something. The question of how large can, you know, if you make a much larger model, is it going to be a lot smarter? I think it's still a bit unknown. I mean, there are people who argue it's going to be very good at reasoning, but at the same time, this kind of token by token generation we're doing now is not an amazing format for reasoning because you have to like linearly like say one thing at a time.

17:50So it's not really good for like making plans or comparing versions. I think to get a really smart application, you'll need to combine today's language modeling with some other sort of framework around it that uses it multiple times or explores a planned space or whatever. And then you might get something good. And it's also possible that the very largest models are simply memorizing more stuff. So they're impressive in terms of trivia. I can ask it about some random topic and it will know, but they're not really smarter at solving even a basic word problem. So yeah, I'm not sure. Unfortunately, especially with training from the web, it's often very hard to tell apart like reasoning from memorization, essentially.

18:30They didn't see that thing before. So I think actually being able to do experiments where you train these on carefully selected data will lead to better understanding of what they can do. Yeah, yeah, that makes sense. Maybe if we think a little bit, just because you have great visibility from your role at Databricks, like what other tooling do companies need, like your enterprise customers or just generally enterprises need to make use of these models? Because you said, you know, we believe the core technology, the models themselves are getting commoditized. Yeah. So definitely the first piece is you need a data platform that could actually build, you know, reliable data, right?

19:08So we think that's like the bread and potatoes of like getting anything you need, you know, a basis to like sort of build on. So we think that will become really important. And maybe data platforms will have to evolve a little bit to be better at supporting unstructured data like text and images and so on, and to do quality assessment and stuff like that for it. That's one piece. I think another piece you need is you need the MLOps piece of being able to experiment with things, deploy them, A-B test them, and so on, and see what does better and improve it incrementally. And I also think these models will need a good connection to operational systems inside the company to do really powerful things with the latest data.

19:50So you saw probably the support for tools in ChatGPD. Before that, there were lots of groups working on at least models integrated into search engines, sometimes into calling other tools as well, like calculators. I think it's still a little bit open-ended. There's one extreme where people say the model will figure out what tools to use on its own. I think for enterprise use cases, that's a little bit more than you really need. You can kind of give it some tools and feed it stuff, and it doesn't have to discover and read the manual to figure out which one to use. But yeah, I think that's another piece you'll need for really powerful applications.

20:25And then I do think infrastructure, just basic training and serving infrastructure is important too when you start to care about performance, like about latency and speed. And you can see some of the new search engines using these models are not that fast, right? like a little bit slow, you know, it would be nice to have it faster. And for automated analytics, it's even more important that it's efficient. So there could be, I think there'll be a lot of activity there. Yeah. Where do you see enterprises getting the most value from investing in, I guess, more traditional ML and then like some of the language model stuff to date?

21:01Yeah, great question. So traditional ML, we're seeing actually virtually all major enterprises, you know, in all industries are using it. It's changed a lot in the past decade, actually. So So it's very good for forecasting things in general and for automating certain types of decisions. So basically, for example, optimizing your supply chain. You don't have time to look at exactly everything that's going on and think about it and have a meeting. But if you do order the right amount of parts to meet your demand this week, or if you minimize the amount of time an agricultural product sits in a warehouse and degrades in quality or stuff like that, it matters a lot.

21:40And it can have a huge impact on the profitability of a company. So we're seeing a lot of that, people applying it to automate supply chain and to automate basically their operations in various ways. And then there are more classic use cases like FART detection and stuff like that. Well, it's always an arms race and you're trying to do the best you can because every percent of accuracy you do better in can translate into a huge impact. With language models specifically, and especially with kind of conversational ones, I think the really exciting thing is interfaces to people. And I think customer support is a very obvious one.

22:18Maybe things like recommendations or asking questions on a product page in retail. Things like search augmented with stuff is one. And we've also found that just internal apps in a company that have a lot of internal data can benefit from this kind of thing. So one of the things we've built, for example, is inside Databricks, we have all these resources for engineers to understand how different parts of the product work, how to operate it, all the APIs. And people used to just ask each other questions in these Slack channels for each team. And we could use that data, like the questions and answers, plus the data in the actual documentation to essentially automatically answer many such questions and just save people a lot of time.

23:02So I do think that any app that has kind of business data or like stuff written by humans in it, like your issue tracker for your software development or like your Salesforce or something like that could benefit from these kind of interfaces. Yeah. Yeah, it seems like any type of forum or anything else instantly becomes like data that you can use to fine tune or train a model that's specific to your customer support use case. Or you could use an embedding or something to do interesting things with it. So it seems like there's some really cool stuff to do. Are there any specific areas that Databricks is not focused on that you think would be especially interesting for somebody to build from a tooling perspective for enterprises trying to use some of these technologies?

23:41Yeah, I think there are a lot of these. I think it's very early on. Probably one of the most obvious ones is just the domain or vertical specific models and tools. I actually think even a lot of the enterprises that have a lot of the data in various domains might turn more into data or model vendors of some form in the future as they use this to build something that no one else can. So I wouldn't be surprised at all if you see the next wave of companies for, say, security analytics or biotech or analyzing financial data or stuff like that really build around LLM technology in there. And I also think in general, in the app development space, how do you develop apps that incorporate these tools?

24:25It's very open. It's not clear what the best way to do it is. And you might end up with really good programming tools that focus on this problem. I would say, you know, for people thinking about startups and so on, like you want your startup to have, you know, a long term defensible mode, ideally something that goes over time also. So anything around a unique data set, for example, or unique like feedback interaction you have is always good. Right. Like honestly, even something like adding ML features in your product that just kind of learn from your users and, you know, do better recommendation and so on could eventually become a mode where like, you know, others just can't easily catch up.