Safety in Numbers: Keeping AI Open

0:00I think the battle is for the neutrality of the technology. This is the story of humanity making knowledge access more fluid. Basically in 2021 every paper made this mistake. It means that we're only trusting the team of large companies to figure out ways of addressing this problem. All of the people that joined us as well, deeply regretted because we think that we are definitely not at the end of the story. As it turns out, if you look at the history of software, the only way we did software collaboratively is through open source, the why change the recipe. Scaling loss. These underpin the success of large language models today.

0:37But the relationship between datasets, compute, and the number of parameters was not always clear. But in 2022, a pivotal paper came out, often referred to as Genjila, that changed the way that many people in the research community thought about that very calculus. demonstrating that data sets were actually more important than just the sheer size of the model. One of the key authors behind that paper was Arthur Mech, who was working at DeepMind at the time. Now earlier this year, Arthur banded together with Guillaume Lampel and Timothy Lacroix, two researchers at Metta who worked on the release of Lama, and together they founded a new company, Mistral.

1:16Together, the team has been hard of work releasing Mistral 7B in September, a state -of -the -art open -source model that very quickly became the go -to for developers. And they just released, as in in the last few days, a new mixture of experts model that they're calling PIXTROL. So today you'll get to hear directly from Arthur as he sits down on A16Z General Partner, Anjane Mita, as the battleground for large language models heats up to say the least. Together they discuss the many misconceptions around open -source and the war being waged on the industry. Plus, they'll discuss the current performance reality of open and closed models, and whether that gap will close with time.

1:55Plus, the kind of compute, data, and algorithmic innovations required to keep scaling LLMs efficiently. Now, it's really rare to have someone at the frontier of this kind of research be so candid about what they're building and why. So, I hope you come out of this episode as excited about the future of open source as I did. Enjoy! As a reminder, the content here is for informational purposes only. Should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16z fund. Please note that A16z and its affiliates may also maintain investments in the companies discussed in this podcast.

2:36For more details including a link to our investments, please see A16z .com slash Discoachers.

2:46All right, why don't we start with the founding team story. We flash back to a few years ago, labs are building foundation models and the consensus across the research community was that the size of these models was what mattered most. How many million or a billion parameters went into the model seemed to be the primary debate that people were having. But you had a hunch that the role of data mattered more. Could you just give us the backstory on the Chinchilla paper you choreo? what were the key takeaways on the paper and how is it received? Yeah, so I guess the backstory is that in 2019, 2020, people were relying a lot on the paperical scaling laws for large -language models.

3:26That was advocating for basically scaling infinitely, the size of models and keeping a number of data points rather fixed. So just saying that if you had four times the amount of compute, you should be mostly multiplying by 3 .5, your model side and then maybe by 1 .2, your data size. And so a lot of work was actually done on top of that. So in particular, DeepMine, when I joined Project Gofer, and there was misconception there, there was also misconception on GPT -3, and basically in 2021, every paper made this mistake. And at the end of 2021, we started to realize there was some issues when scaling up.

4:04And as it turns out, we turned back to the mathematical paper that was actually talking about scaling others, and it was a bit hard to understand. And we figured out that actually if you thought about it a bit more in a theoretical perspective, and if you looked at like empirical evidence we had, it didn't really make sense to actually grow the model size faster than the data size. And we did some measurements. And as it turned out, what was actually true was actually what we expect, which is in commonwealths, if you multiply by four your compute capacity, usually multiply by two, the model size and by two the data size.

4:40That's approximately what you should be doing, which is good because if you move everything to infinity everything remains consistent So you don't have a model which is infinity big or a model which is infinity small with infinite compression or like close to zero compression So it really makes sense and as it sounds like it's really what you observe and so that's how we train chinchilla and and that's how we wrote the chinchilla paper at the time you were a deep mind and Your co -founders were at meta What's the backstory around how you three end up coming together to form Mistrol after the compute optimal skating was work that you just described?

5:13So we've known each other for a while because Gium and I were in school together and Timote and I were in master together in Paris. Basically we had like the reparlene later years. Timote and I actually worked together as well again when I was doing postdoc in mathematics and then I joined DeepMine as Gium and Timote when to become from permanent researchers at Meta. And so we continue doing this. I was doing large -neggorge models in between 2020 and 2023. Guillaume and Timothy, we're working on solving mathematical problems with large -neggorge models. And they realized they had to have stronger models.

5:46And on my side, I was mostly working on a small team, deep minds, so we did a very interesting work on Red Troll, which is a paper doing retrieval for a large -neggorge models. We did Chinchilla, then I was in the team doing Famingo, which is actually one of the good way of doing a model that can see things. And I guess when Chagy B2 went out, we knew from before that the technology was very much game -changing, but it was a signal that there was a stronger opportunity for building a small team, focusing on a different way of distributing the technology. So redoing things in a more open source manner, which was not the direction that Google had this was taking.

6:20And so we had this opportunity, then we left the company at the beginning of last year, and created the team that that started to welcome the 5th of June, and on recreating the entire stack and training of first models. And if I recall correctly, right before they left, Tim and Guillaume had started to work on Lama over at Meta, could you describe that project and how it was related to the Chinchilla scaling was work you've done? So Lama was a small team production of Chinchilla, at least in its approach of parametrization and all of these things. It was one of the first papers that established that you could go beyond the Chinshira scaling loads.

6:57So Chinshira scaling loads tell you what you should be training if you want to have an optimal model for a certain computer cost at training time. But if you take into account the fact that your model should also be efficient at inference time, you probably want to go for beyond the Chinshira scaling loads. So it means you want to over train the model. So train on more tokens than would be optimal for performance. But the reason why you do that is that you actually compressed models more. And then when you do inference, you end up having a model which is much more efficient for a certain performance.

7:29So by spending more time during training, you spend less time during inference, and so you save cost. That was something we observed at Google also, but the Lama paper was the first to establish it in the open, and it opened a lot of opportunities. Yeah, I remember both the impact of the Chinchilla scaling was work on multiple labs, realizing just how unoptimal the training setups were. and then the subsequent impact of Lama being dramatic on the industry and realizing how to be much more efficient about inference. So let's fast forward to today, as December 2023, we'll get to the role of open source in a bit.

8:05But let's just level set on what you've built so far. A couple months ago, you released Mr. Ols 7B, which was the best in class dense model. And this week, you're releasing a new mixture of experts model. So just does a little bit more about mixture and how it compares to other models. Yeah, so mixed studies are a new model that wasn't released in open source before. It's a technology called Spars mixture of experts, which is quite simple. You take all of the dense layers of your transformer and you duplicate them, you call these layers expert layers. And then what you do is that for each token that you have in your sequence, you have a router mechanism, so just a very simple network that decides which experts should be looking at which token.

8:47And so you send all of the tokens to their experts. And then you apply the experts and you get back the output and you combine them and then you go forward in the network. And you have eight experts per layer and you execute only two of them. So what it means at the end of the day is that you have a lot of parameters on your model. You have 46 billion parameters. But the thing is that the number of parameters that you execute is much lower than that because you only execute two branches out of eight. And so at the end of the day, you only execute 12 billion parameters per token. And this is what counts for latency and throughput and for performance.

9:23So you have a model which has the performance of a 12 billion parameter network that have much higher than what you could get, even by compressing that a lot on a 12 billion dense transformer. Sparsomix2offexperts allows to be much more efficient at inference time and also much more efficient at training time. So that's the reason why we just to develop it very quickly. just for folks who are listening who might not be familiar with sort of state of the art architecture and language models, could you just describe the difference between dance models which have been your primary architecture to date today and mixture of experts, intuitively what are the biggest differences?

10:00So they are very similar except on what we call the dance networks. In the dance transformer you alternate between the attention layer and the dance layer, generally that's the idea. A sparse mixture of experts you take the dance layer and you duplicate So that's where you actually increase the number of parameters. So you increase the capacity of the model without increasing the cost. So that's where you have decoupling what you can remember, the capacity of the network to its cost at inference time. If you had to describe the biggest benefits for developers as a result of that inference efficiency, what are those?

10:31It's costs and latency. Usually that's what you look at when you're a developer. You want something with your cheap and you want something which is fast. But generally speaking, just the trade -off is strictly favorable in using mixed -trial compare. to using a 12 billion dance model. And the other way to think about it is that if you want to use a model which is as good as LAMAD to 70B, you should be using Mixtral because Mixtral is actually on par with LAMAD to 70B while being approximately six times cheaper or six times faster for the same price. Could you talk just a little bit about why it's been so challenging for research labs and research teams to really get the mixture of expert architecture, right?

11:09For a while now folks have known that the dense model architectures can be slow, they're expensive, and they're difficult to scale. And so for a while people have been looking for an alternative architecture that could be like you were saying cheaper, could be faster, could be more efficient. But it's taken a while for folks to figure this out. What were the some of the biggest challenges you have to figure out to get the M .A. model right? Well there's basically two challenges. the first one is you need to figure out how to train it correctly from a mathematical perspective. The other challenges is to train efficiently, so how to use actually a hardware as efficiently as possible.

11:44You have like new challenges coming from the fact that you have tokens flying around from one expert to another. That creates some communication constraints and you need to make it fast. And then on top of that, you also have new constraints that apply when you deploy the model to do inferencing efficiently. And that's also the reason why we released an open source package based on VLNM. so that the community can take this code and modify it and see how that works. Well, I'm definitely excited to see what the community does with the MOE. Let's talk about open source, which is an approach and a philosophy that's permeated all the work you've been doing so far.

12:19Why choose to tackle this space with an open source approach? Well, I guess it's a good question. The answer is that it's partly ideology code and partly pragmatical. We have grown with the field of AI. From 2012 We were detecting cats and dogs and in 2022 we were actually generating text that looked human. So really made a lot of progress and if you look at the reason why we made all of this progress, most of it is explainable by the free flow of information. You had academic labs, you had very big industry -backed labs communicating all the time about the results and building on top of each other results and that's the way we increased significantly the architecture and training techniques.

13:01We just made everything work as a community and all of a sudden in 2020 with GPT -3 this tight reversed and Company started to be more opaque, but what they were doing because they realized there was actually a very big market And so they took this approach and all of a sudden in 2022 on the important aspects of AI and on LLM Which are really the host but the topic and the most promising one and beyond Chinchilla that there were basically no communication at all. And that's something that I as a researcher and to Mutian Geo, and all of the people that joined us as well, deeply regretted, because we think that we are definitely not at the end of the story.

13:38We need to invent new things. There's no reason why to stop now, because the technology is effectively good, but not working completely well enough. And so we believe that it's still the case that we should be allowing the community to take the models and make it their own. And that's some ideological reason why we went into that. The other reason is that we are talking to developers. Developers want to modify things and having a deep access. A very good model is a good way of engaging with this community and addressing their needs so that the platform we're building as well is going to be used by them.

14:11So that's also like a business reason, obviously, as a business we do need to have a valid monetization approach at some point. But we've seen many businesses build open core approaches and have a very strong open source community and also a very good offer of services and that's what we want to be. That resonates. The early days of deep learning were largely driven by a bunch of open collaboration between researchers from different labs who would often publish all their work and share them at conferences. Transformers famously was published and opened the entire research community, but that has definitely changed.

14:44As a clarifier, do you see a difference between open and open source as viewed by the community or are those two things the same in your mind? Yes, so I think there's some level of open source thing in the AI. So we offer the weights and we offer the inference code. That's the end product that is already super usable. So it's already a very big step forward compared to closed APIs because you can modify it and you can look at what's happening under the hood, look at activations and all. So you have interpretability and the possibility of modifying the model to adapt it to some editorial tone, to adapt it to proprietary data, to adapt it to some specific instructions, which is something that is actually much harder to make if you only have access to a closed source API.

15:28And that's something that also goes with our approach of the technology, which is to say, pre -trained models should be neutral, and we should empower our customers to take these models and just put their editorial approaches, their instruction, their constitution, if you wanna talk now, can'tropic into the models. That's the way we approach the technology. We don't want to pour our own biases into the pre -trained model. On the other hand, we want to enable the developers to control exactly how the model behaves and what kind of biases it has, what kind of biases it doesn't have. So we really take this modular approach and that goes very well with the fact that we release open -weight models.

16:05Could you just ground us in the reality of where these models are today, just to give people a sense of where in the timeline we are? is open source really a viable competitor to proprietary closed models or is there a performance gap? So Mixtile is a similar performance to GPT 3 .5. So that's a good grounding. Internally, we have stronger models that are in between 3 .5 and 4 that are basically 3rd or 3rd, the best model in the world. So really we think that the gap is closing. The gap is approximately 6 months at that point. And the reason why it's 6 months is that it actually goes faster if you do open source things because you get the community, modify the model, suggest very good ideas that can then be consolidated by us, for instance, and we just go faster because of that.

16:48So it has always been the case that open source ends up going faster, and that's the reason why the entire internet runs on Linux. I don't see why it would be any different for AI. Obviously, there's some constraint that are slightly different because the infrastructure costs quite high to train a model with customer money. But I really think that will converge to a setting where you have proprietary models and the open source model, just as good. Yes, so let's talk about that a little bit more. How are you seeing people use and innovate on the open source models? I think we've seen several categories of usage.

17:20There's a few companies that know how to strongly find your models to their needs. So they took Mr. R7B, a head of human annotations, had a lot of proprietary data, just modifying Mr. R7B so that it solved their task just as well as dbt3 .5, but only for a lower cost and a higher level of control. We've also seen, I think, very interesting community efforts in adding capabilities. So we saw, like, context length extension, 128K, that worked very well. Again, it was done in the open, so the recipe was available, and this is something that we were able to consolidate. We've seen some effort around encoders, imagine coders, to make it a visual language model.

17:58A very actionable thing that we saw is, I think, the hugging phase. Folks first did the direct preference optimization on top of Mr. Al -7B and made a much stronger model than the instructed model we proposed at the early release. And it turned out it's actually a very good idea to do it. And so that's something that we've consolidated as well. So generally speaking, the community is super eager to just take the model and add new capabilities, put it on the laptop, put it on the iPhone. I saw Mr. Al -7B on the stuffed parot as well. So, fun things, useful things, but generalized speaking has been super exciting to see the research community take a hold of our technology.

18:34And with Mixtral, which is a new architecture, I think we are also going to see much more interesting things because on the inter -credivity field, also on the safety field as it turns out, you have a lot of things to do when you have deep access to an open model, and so we're really eager to engage with the community. Safety is an important piece to talk about. The immediate reaction of a lot of folks is the demopen source less safe than closed models. How would you respond to that? I think we believe that it's actually not the case for the current generation. Models that we are using today are not much more than just a compression of whatever is available on the internet.

19:11So it does make access to knowledge more fluid, but this is the story of humanity making knowledge access more fluid. So it's not different than inventing the printing machine where we had apparently similar debate. I wasn't there, but that was the debate we had. So we are not making the world any less safer by providing more interactive access to knowledge. So that's the first thing. Now the other thing is that you do have immediate risk of misuse of large language models and you do have them for open source models, but also for closed models. And so the way you do address these problems and come up with contour measures is to know about them.

19:48So you need to know about breaches basically. And that's the same way in which you need to know about breaches on operating systems and on networks. And so it's no different for AI, putting models under the highest level of scrutiny as a way of knowing how they can be misused and it's a way of coming up with contour measures. And I think a good example of that is that it's actually super easy to exploit an API, especially if you have fine tuning access to make GPT -4 behave in a very bad way. And since it's the case, and it's always going to be the case, it's super hard to be a bit adversarial in boost.

20:18It means that we're only trusting the team of large companies to figure out ways of addressing these problems. Whereas if you do open sourcing, you trust the community. And the community is much larger. And so if you look at the history of software and cyber security and operating systems, that's the way we made the system safe. And so if we want to make the current AI system safe and then move on to the next generation that potentially will be Even stronger and then we can have this discussion again Well, you do need to do open sourcing so today we think that open sourcing is the safe way of Developing it.

20:53Yeah, I think this is not understood widely that when you have Thousands or hundreds of thousands of people able to read team models because it's open source the likelihood that you'll detect biases and built -in breaches and risks are just dramatically higher. And I think if you were talking to policymakers, how would you help advise them? How do you think they should be thinking about regulating open source models, given that the safest way often to battle hard and software and tools is to put them out in the open? We've been saying precisely this that the current technology is not dangerous.

21:33On the other hand, it can be misused. On the other hand, the fact that we are effectively making them stronger means that we need to monitor what's happening. Best way of empirically monitoring software performances is through open source. So that's what we've been saying. There's been some effort to try to come up with very complex governance structure where you would have like several companies talking together, having some safe space, some safe sandbox for a red tumor that would be potentially independent. So things that are super complex, but as it turns out, if you look at the history of software, the only way we did software collaboratively is through open source.

22:09So why change the recipe today where the technology we're looking at is actually nothing else on the compression of the internet. So that's what we've been saying to the regulators generally. Another thing we've added to the regulator is that if they want to enforce that AI products that needs to be safe, if you want to have a diagnosis assistant you want it to be safe. Well, in order to monitor and to evaluate whether it's actually safe, you need to have some very good tooling. And the tooling requires to have access to LLMs. And if you access closed source APIs, LLMs, where you're a bit in a troubled water because it's hard to be independent in that setting.

22:47So we think that independent controller of product safety should have access to very strong open source models and should own the technology. And if open source LLAMs were to fail relative to close source models, why would that be? I guess the regulation burden is potentially one thing that could make it harder to release open source models. It's also generally speaking a very competitive market and I think in order for open source models to be widely adopted They need to be as strong as a closed source model They have a little advantage because you do have more control and so you can do if you're fine tuning until you can make Performance jump a lot on the specific tasks because you have deep access But really at the end of the day Developers look at performance and latency and so that's why we think that as a company we need to be very much in the frontier if we want to be relevant.

23:41Given the complexity of frontier models and foundation models in these systems, there just stands a misconception that folks have about these models. But if you just took a step back and we'd look at the battle that's raging between folks pushing for closed -source systems versus the open -source system, what do you think is that stake here? What do you think the the battle is really for. I think the battle is for the neutrality of the technology. Like a technology by a sense is something not so. You can use it for bad purposes, you can use it for good purposes. If you look at what the LLM does, it's not really different from programming language.