Multilingual LLMs and the Values Divide in AI with Sara Hooker - #651

25:58It's just perplexity. So perplexity is just generated every time a model is something, something's put through a model. So this is almost the crudest model signal you could possibly have. And the second one was much more expensive. It's L2 norm. So it's called EL2N. It was proposed at a different setting. It was proposed as a pruning metric. So for understanding what weights are important, so not within the data space, within the weight space, but it requires quite a lot of computationally expensive calculations. You need to see how the gradient norm changes over various checkpoints. So you have to kind of take snapshots across model training.

26:37The third one, and for that one, you have to do it. I believe the suggested one was over 15 model training. So you have to have 15 separate models that have been trained independently. And then you take this computation of like, how much does the gradient norm change? The third one was memorization score, also quite expensive, but both the ELTUN and the memorization score were not proposed for data pruning. So we were trying them out for a context that they weren't originally imagined for. And then we compare that with perhaps the grumpiest baseline, does this outperform a random selection? And so really there, what we're asking is a good method should at least be better than randomly choosing what data to keep.

27:17And we found that only perplexity did. So perplexity was the only metric that consistently outperformed, which suggests we have a lot more work to do because we do need more formally rigorous methods, but clearly the methods that we have proposed that are very rigorous, when we test them in the setting for this use case, they are not up to par with just a very simple model signal. Do you have any intuition around, you know, what may emerge in terms of kind of the next step, the next improvement in data selection method? Maybe I'll share how I see this direction in terms of what I think is really the big picture view of this.

27:56One of the reasons I'm attracted to data pruning is that in some ways, there's a very key underlying question, which is we seem to have the least efficient way for training these models right now. We show all the examples to the models the same amount of times, whether that's one or 90 times, but no matter how difficult or how simple the data point is, it's just presented to the model. It's not how humans work. Typically, when we find something challenging, we like squint in, we ask, oh, tell me more, or we ask a clarifying question. We have a high degree of saliency that makes us way more efficient because we spend more time on hard things and less times on easy things.

28:36So the overall direction that I think is fascinating, data printing fits in as a very simple example of this, where you put zero attention on some examples and 100 % of attention on others. But the future of that is adaptive training in general, where over the course of training, you adaptively identify what is salient for your learning procedure and you spent more capacity there. Right now, we allocate the same amount of capacity to all examples. So I see this from a few different directions. Data pruning is one, but even how we use the weights within a model is another. Right now, we do global updates, which means that no matter what, you're updating all of your weights for an incoming batch.

29:15But this doesn't really make sense. We need ways that we have more modular capacity, that we have kind of fusing and merging of different model sections and much more, you know, another work that came out of the lab, that was led by Ted was Mixture of Experts. Mixture of Experts for me is one of the first steps in this direction of just having much more decomposable components to your model and having this adaptive capacity where you can route. So there's two data pruning and general adaptive capacity are how I think that we're going to be creating models that are more efficient because right now we just spend the most.

29:51We show all the examples, we update all weights, We do the same number of forward passes. And if you think about the world, one of our great benefits navigating the world and efficiency is that we can choose to ignore and we can choose how we allocate attention. So that's missing right now from these models. Can you talk a little bit more about mixture of experts broadly and ultimately this work in particular? It's, I think, come up in the context of some speculation that GPT-4 is a mixture of experts based model. And so I'm curious how you explain it, where you see it going and what this particular paper looked at with regard to MOE.

30:34Yeah. So mixture of experts in general is this idea that you have many different models that learn slightly different things. And then at the end of the day, when you actually have your trained model, you route your input, whether that's language or where that's images to the expert most specialized to represent it. It's a beautiful idea. In some ways, it gets this essence of how do we create something where we have specialization instead of just having these massive models where you update everything, you have a model which might be better at coding than it is at generating or processing French prompts.

31:16And so the idea is, is that you have this specialization over time. What are the challenges of a mixture of experts? One is, frankly, that they're very hard to train. They tend to be very sensitive. So because you're essentially doing routing, so you have to decide where something goes, experts can become biased early on in training. So if an expert is favored early on, it almost gets favored repeatedly, which means that there's all sorts of tricks to try and ensure that you don't choose your favorite expert early on and then end up sending it everything. The most crude trick, which actually is widely used, is you do like load balancing with random routing.

31:51So during training, you might just randomly allocate. But I'm sure we can do better. It's one of the open questions. The second tricky thing about mixture of experts is that when you actually deploy these models, you have to hold in memory all the different experts because you have to have the expert ready if it gets chosen. This turns out to be very challenging for actually making these models usable. Just from an operational scalability perspective? Operation scalability, because of the amount of memory that you need in the machine to be able to hold all these checkpoints of fairly big models and then call the one that's needed.

32:28What's interesting is that this was a topic of our work. So we, Ted, led this work and it was about how do we make mixture of experts extremely parameter efficient? And what I mean by that is using 1 % to 2 % of the weight updates so that we can actually only update a few weights and then only use a few weights when you're holding these things in memory. And so it combined mixture of experts with these parameter efficient modules. When you're training your mixture of experts, you don't actually have to bring in the whole model in memory. You have a single model and your mixture of experts are these tiny little routers and tiny little modules on top of that.

33:06So you're adding less parameters, but it means that you have dramatic savings in your memory requirements. And that's an important continuing direction, which is how do we make these actually more usable and lower latency. But in general, mixture of experts is a really exciting direction because it's also just shown clear breakthroughs on certain tasks. So I think you alluded to it, but this idea that mixture of experts is behind some of these very performant models, which has been more or less confirmed at this point. It's, of course, showing huge promise. The tricky thing is the optimization, which takes a lot of effort, as well as the deployment.

33:43So there's also a reason why GPT-4 is considered extremely high latency by those who use it. It takes a long time. And I don't know the details of how it's currently productionized, but I'm guessing part of the issue with the high latency is this model choice structure. Hmm. When these GPT-4 kind of speculation came out, it was suggested that, you know, almost like the fact that OpenAI had to use or ultimately use MOE for GPT-4 was kind of indication that the scaling laws were petering out, right? They couldn't train a bigger model, kind of bigger monolithic model. They had to do these eight experts.

34:23But in contrast, you express excitement about MOE and what it was able to produce. Do you buy into that idea that it is, I don't know, a cop out in a sense? There's no denying that one of the reasons that people gravitate towards MOEs is this idea that it reduces the effective number of flops. So what are flops? It means that the bigger you go, typically the more flops that you have. These are these matrix multipliers, floating point operations. And when you have MOE, because you don't go deep, essentially you're going massively wide. Like you have just many models you can choose from. Your individual models can be big, but they're not extremely big.

35:03And instead you're focusing on the routing as the magic. So you reduce your flops. That almost certainly is a practical constraint because you can essentially have way more compute, but your latency or your necessary energy at serving time is much lower because you're not using all of it. I think that's still one of the motivating factors behind MOE. If I were to talk about these two components, right? One is the effective compute is minimizing flops. The other is a specialization ability. I would actually say specialization as observed property is sometimes not immediately clear with these trained models.

35:43So sometimes it's not that we see that one of these models only does code and one of these models always does French. So definitely what's motivating the interest right now is more this effective compute argument. What motivates a lot of people in their general interest and why this architecture is special is the idea if we do it better, because optimization is still very unknown with this type of architecture, we can get that specialization. But you are right that if it was feasible to train this massive model instead, there would probably be more exploration in that direction. I don't think it means scaling laws are petering out.

36:16I just think it's very difficult to train models of that size while still preserving training dynamics. The main issue when you go with a single huge model is that training is frequently derailed. You have lost spikes. It is extremely difficult. And the people who know how to manage that optimization process are just very, it's a very small group of people. Whereas if you do MOE, it's sensitive its own ways. How do you route? Do you have balanced MOEs? It's own dark set of dark art, but it does avoid some of the gradient saturation and instability that seems to happen in these much bigger models.

36:50You addressed a point that I wanted to raise with regard to your initial description of MOE. as like these semantically significant experts in some kind of hierarchical arrangement or a parallel arrangement. And that's often not the way they're practically implemented. My impression of the GPT-4 stuff is that the data partitioning that was used to train the experts was rather, you know, it was fairly naive. There wasn't any attempt to do that. It was kind of just splitting up the data and training each of those experts. are the experts in an MOE type of model, are they trained independently or are they trained kind of end-to-end with this, the multiplexer thing being part of the, you know, what's learned in the process?

37:41Yeah, so typically they're trained together. So you want to learn the routing. The routing is the magic of these mixture of experts. So, but there's a lot of tricks beyond that to stabilize your routing. One is that you could have different learning rates for different experts. So you kind of try and stabilize it by making sure experts get updated in different rates and it kind of induces the experts to learn different things. But I also think the overall goal is typically to take these experts, learn them jointly and try and induce specialization. I do think in this case, what's interesting is that we haven't observed much specialization.

38:18I don't know about the concrete aspects of GPT-4 because it wasn't published, but we do know from other papers that have published that it's very hard to induce the type of data specialization, especially if you use random shuffling, which is what tends to work the best. So for most of our models today, random shuffling of data is hard to beat. And so even with mixture of experts, we talked about the data partitioning being fairly naive. Even for published papers on mixture of experts, typically the shuffling is preserved the same way it is done for other models. So it will be randomly shuffled.

38:51And what that means is we see some differences in the ability of these models to specialize, but it feels still a bit cherry picked. So if you look at the papers, we actually explored this in our paper. we looked at the differences between if we showed the mixture of experts tokens versus if we showed embeddings. And the difference being is that if you show mixture of experts embeddings, it's much more of a heavy prior on what is the topic of the input. So we're kind of imposing this more heavy onus towards specialization. What we found is this actually hurt performance. It wasn't as competitive of just showing the tokens.

39:28And so sometimes efforts to induce specialization and result in worst performance, which you're right to point out. It's very interesting. It does seem like the driving factor right now for why people love these models. It's not the inherent goal of why these models were first designed. And I still think that is partly due to the opaqueness of what optimization works for this type of model. So I'm optimistic that we will see better recipes to induce specialization. And as someone who cares a lot about adaptive computation and the goals of it. I still think of a mixture of experts as this really interesting direction.

40:02But why do people use it widely now? It's more for this energy conservation reason, which is that you're just getting, it's a very clever technique to scale across rather than scale deep. And that has resulted in really nice gains in performance. And so a lot of people gravitate towards it for that. Got it. Awesome. I also wanted to ask you about the Grand Illusion paper, which looks at portability, like framework types of issues, right? Was this investigating a pain point that emerged in other projects? This project started because I wrote the hardware lottery a few years ago. And the hardware lottery is really this paper, which is a grumpy paper.

40:43And it's saying we're very locked into the tools that we have. And it says basically like the tools that we have dictate so much of like what ideas are possible and really the ideas that we end up pursuing. A perfect example is that GPUs really unlocked deep-door networks. Like if you think back to 2012 and why convolutional networks were basically adopted overnight, it was because GPUs over the 2000s had been very intentionally slowly repurposed from video games to work with convolutional neural networks. And that was the trick. It allowed for much deeper networks, which turned out to be the crucial ingredient that was missing.

41:20Even though for decades, the researchers working on deep neural networks have been marginalized. And in fact, I think it's one of the most interesting examples of tooling, paying a deciding factor on when ideas are recognized. That was a paper which for me had come from this place of trying to understand even some of my own ideas and their incompatibility with hardware. For example, sparsity is a very widely used compression idea, but it's widely used in research forums because it achieves the best levels of compression, it doesn't play well with GPUs. There's no gains because our whole computer science and hardware field has been built around the idea of matrices, that all your inputs are going to be in a structured form.

42:06So this idea of an unstructured form where you have some things that are sparse and some things that are not, it results in really not any game because it still has to be put into a matrix for the hardware to recognize it. So I wanted to interrogate this and And Fraser and Zung led this paper. And I wanted to ask, well, how locked in are we really? Because we know we're locked in, but it's hard to tell because we often only realize after the fact what was holding us back. So we only realized that GPUs were critical for deep neural networks after 2012. But how can we tell how bad the situation is?

42:42And so this paper asks, it says all these mainstream frameworks, so JAX, TensorFlow, Flow, PyTorch, they all are purported to be general framework. They're all Python libraries, they're workhorses of ML, they're used by millions of people around the world. And most ideas, probably 99.9 % are probably developed in one of these frameworks right now. How portable are they? Or how locked in are they with the hardware that they use? And this is a simple idea. But what we find is that these frameworks, even though they purport to be general, are just extremely locked into underlying hardware. So for example, when you try to transfer PyTorch to TPUs, 40 % of all functions fail, which means that if you need to translate an idea to a different hardware type, it's just not usable.

43:36We also find that there's just massive slowdowns in performance, which sometimes are untenable. So some functions just fail to transfer across completely in a usable way. What all this means is that there's first-class citizens for each type of hardware. So for example, JAX. JAX is actually a promising software framework because there's hardly any failures. It works. Everything works in TPUs. Everything works in GPUs. But you see massive slowdowns on GPUs. So above 90 % of all functions have massive slowdowns on GPUs. And that's because it's been developed with TPUs in mind. So what the big picture of this is, I think that we really don't have a sense of how our field is narrowing.

44:18It's happening with software. It's also happening with architectures. We have everything basically now is built on transformers. And what's interesting is that you see hardware overfit to transformers, which means it's going to be harder and harder and harder to do the next step where I shift to a totally different architecture. NVIDIA in their latest H100 even called it the transformer accelerator, even though that was a clever marketing and there wasn't really anything under the hood. There wasn't just accelerating matrix multiply. It also speaks to the what you're calling out here. Yes, yes.

44:52And it's very interesting because this is a wider. I worry a lot about this because our field moves really fast. And it's very fascinating because we forget that a mere decade plus ago, There was no one in mainstream machine learning who would have vouched for deep neural networks. And so it's kind of absurd because transformers are not the finish line. I think most people who work in this field know that. But it is becoming stickier to deviate from transformers. And so the time it takes for us to make the next leap is just more pronounced by some of the stickiness and friction. Do you propose an answer, a solution?

45:32it doesn't seem like the tools can predate the method, right? That doesn't make sense. So does it always have to be like this? Is this just kind of the way the game has to be played? I think that's partly why I wanted to focus on software portability. So typically, a lot of our research innovation is empirical. So no one wanted to acknowledge deep neural networks before they empirically worked. To get empirical evidence, you have to make the cost of iteration cheaper. So you have to make it easier for people to experiment across different settings and to understand the limitations and settings in which the methods work.

46:06So for me, software portability is crucial because the ability to experiment on different hardware is crucial. There's some hardware that really facilitates certain subsets of functions. So even documenting the latency of functions across is actually part of this wider benchmarking. And as we get even more narrow in what types of software is supported, we need to have ways to evaluate what software frameworks are getting better at this and what they're getting worse at this. So this is part of a wider initiative where the crucial thing in terms of enabling innovation is decreasing the cost of switching and increasing the benefits of doing ablations on different types of hardware.

46:48That only works if we decrease the cost of all. And this comes in different forms. But for me, JAX is promising because at least it doesn't have failures across both hardware types. For PyTorch to lose 40 % of all its functions when it switches to TPUs, that really tells me that the compiler team for TPUs needs to pay attention. If this is one of the most widely used frameworks with the most adoption, it needs to be supported. Otherwise, we're just going to see these large cliffs where people who use PyTorch are locked into GPUs and can only experiment there, can only see what ideas work there.

47:21We also point out to concrete examples where ideas have not been able to be pursued because of the software lottery. One is early exiting. So it's interesting. Early exiting, the main benefit is you don't have to pass something through the entire network and you can exit early if the example is easy. You gain a lot. It doesn't work at all with TensorFlow. Why? Because you have the initialization of the graph at the beginning. This is a graph style software program. And what that means is that because the entire network is in the graph represented, you don't gain any memory because you can't modify it once you have initialized it.

47:58So there's things like this where I think part of what we're trying to call out is some areas where compiler level developers and hardware experts need to get close into the room and see the tension points. And frankly, I often find the most important first step in these type of problems is to characterize the severity of the issue. And this work I'm proud of because it does that in a very grumpy way, but in a very needed way, because we need to start collaborating with hardware architects in particular and with software architects. In a sense, it strikes me a bit as documentation or transparency effort for frameworks akin to model cards and data cards, things like that.

48:40Data sheets for data sets and model cards. Yeah, and I think that's actually a nice characterization because part of our goal is to create a common language to bridge these different subfields. One of the reasons this is so underexplored is because hardware architects typically have a separate training, a siloed type of jargon, and don't really talk to algorithm developers. In fact, most algorithm developers, like myself, I build models, we're just stuck with the hardware we get. And we don't really influence the next generation of hardware. Codesign has become much more of a prevalent concept only recently, and it's still very awkward.

49:22The timelines are different. The time to produce a new algorithm is much shorter than the timeline to produce the next generation of hardware. So for me, part of what I see these type of efforts, it's giving us a common language to articulate pain points. For example, I might not know the jargon that's needed to describe what changes to need to happen to a compiler. But I do know now what functions are most impacted by this lack of portability. And if they're central to my idea, I now have a language to express why it's important that we support them. And that's often the first crucial piece in creating better cohesion between very distinct communities who don't talk to each other that much.

49:58because their training has been so siloed and because the cultures are very different. Hardware architects also have a very different culture. We're very, the ML and researchers tend to favor a primarily open source community. Even the social sites that are used are entirely different where hardware architects are always on LinkedIn. I do, I now have several ongoing collaborations with different hardware architects and it's very fascinating. So part of this type of work is meant to give us a joint language to prioritize some of these changes. Hmm. Interesting. Yeah. Also calls to mind visualizing browser compatibility charts, you know, for like CSS and HTML.

50:37Well, this will work here. This will work there. You need polyfills for this because it's not native to the browser. And certainly web development would be much less far along if we hadn't come up with a way to talk plainly about what works where. Yeah, exactly. I agree. Yeah, that's a great example. This is switching gears quite a bit. One of the things that you've been thinking quite a bit is what you characterize as a values divide in the AI community, talking about folks that are concerned about existential risk of AI and that kind of thing. How do you think about all that? Yeah, I think we had an interesting time because it's very rare that the ideas that you work on are adopted by millions of people overnight.

51:23And I think we're at this moment where clearly some of those technologies connected in a much more deep way with, I would say, a very fundamental emotional way with a lot of the general population, people who are not AI researchers, people who are not developers. What's interesting about that is that didn't happen with previous breakthroughs. With convolutional neural networks, it was widely applied. So it probably lives behind your phone on different apps, but it didn't connect with the everyday person in a similar way, whereas this has really emotionally connected. Most people have played with a chatbot over the last few months.

51:55Most people have experimented and probed it. I think that part of it is that it feels very interpretable to people, which is really interesting. The ability that you can probe through chat and see the responses and identify where the model fails and where it's good. I think that's connected people to the idea of playing with an algorithm in a much more visceral way than they ever felt with computer vision, which typically was for a few narrow tasks and was always behind the hood, so to speak. Like people didn't really probe a computer vision model. But what this means is that there's just been, I would say, all the emotions that come with people experiencing new technology.

52:32I also think from the researcher side, it's been the amount of interest has really forced this question of this was a research idea which was adopted widely overnight. How do we make sure this is used in a responsible way? And what are the risks? Because whenever a tool is powerful, you have both amplified risk as well as amplified impact. I think the value divide, and I'd be curious to hear what you think about it as well, is that there's a sense that this is part of a wider march towards much more performant models. And that our focus should be on making sure that those are developed in a way that is not a threat to humans and human welfare.

53:16And this is what I would call the focus on existential risk. I think that for people who are concerned about current models, this sometimes is in tension with feeling that we need enough resources to also make sure that our models, which are current deployed, don't amplify risk of some issues. closest to my mind and to my heart right now is just misinformation. I really am concerned that we've created models where the generated text is not distinguishable to a human. And so we've lowered the barrier of creating information. This is really positive. It means that in some cases, we're enabling people to communicate and to really be creative in new ways.

53:55But on the negative side, I think we've also lowered the barrier to entry for some problematic uses of information creation. And so top of my mind is actually things like how do we make sure that we're investing in directions actually could help us identify tags that's generated by a model, but also that we have good traceability for models and we know how they're being used in different places. That would be if I were to characterize the divide between those who are focused on future models versus those who are focused on current. It's more a divide about where resources should be allocated. If you care a lot about existential risk, often you care a lot about use cases like biosecurity threats.

54:34For me, this is a compelling risk. It tends to have immediacy to those that hear about it because it feels, wow, if a language model can describe to someone how to build a bomb, it feels very visceral. People can see that risk. But I often find that, is this the most frequent risk that we're going to see in terms of how people actually use these models? And if that's the risks that we care about, I think that's addressable. But does that displace the conversation about all these other risks? Because we've created universal models that can be used for many different use cases. And sometimes I think there's certain use cases that occupy a lot of the discussion and kind of display some of these more important or equally important conversations about how do we make sure that these are representative?

55:20How do we make sure they cover languages outside of English? How do we make sure that we have these protocols in place to really isolate bad actors? Yeah, I don't know. What do you think? I guess I tend to think that it all has its place. Like, glad that somebody is thinking about bio-risk. I'm glad that somebody is thinking about broad AI safety concerns. I think there's too small a set of resources going to thinking about safety broadly, but I don't necessarily see that the pie is being split in such a way that there's a vast amount of money going to our resources going to existential risk questions, as opposed to kind of short-term alignment issues around LLMs.

56:06I think that it often strikes me that these are two kind of poles that are talking past one another and tend to think that, hey, if you're really concerned about kind of this bioalignment risk, go do your thing. This is just impressions based on what I'm seeing and what I'm hearing in conversations. I think that there's definitely something to risk diversification. I always say when you have a portfolio of research ideas, research ideas, it's important that you don't have a bunch of researchers working on the same direction to the problem. In general, history has shown that the more diverse your approaches, the more interesting the outcomes.

56:44I think that definitely from my perspective, it's important to have researchers who are working on different aspects. When you talk to a lot of people who are working on existential risk, in practice what they end up doing is a bit closer to working on current threat models, which is really interesting when you kind of look at what researchers actually produce. My main goal right now is actually getting everyone into the same forums, because I think that often we've invented new language for some things that are used in the different communities, but are really the same thing. Like mechanistic interpretability is a perfect example.

57:19Why have we recast interpretability as mechanistic interpretability? But depending on if you're interested in existential risk versus if you've been working on response for AI, you'll call something mechanistic interpretability versus interpretability. So I think part of my goal is to anchor these to a similar form and start these conversations. But there is no doubt in my mind that it has skewed for the general public, not, I think, for someone that's talking to researchers every day, which I think you do. But for the general public, I do think that the framing of existential risk is gripping.

57:57And case studies like biosecurity tend to feel intuitive to a lot of people and maybe more immediate than some things. I think that we tend to weight problems which feel like they impact us. And so the focus on multilingual has been secondary to a lot of labs where these models have been created, which have been primarily in English speaking and Western countries and China where there's Chinese specific models. But part of that is that we tend to weight what we feel impacts us. And so I do think some of this focus on narrow use cases like biosecurity is also it doesn't really speak to all the ways in which these models will be used across the world.

58:39And so that I think has to be balanced. But do you feel that the kind of the public perception and the weight of these existential risk questions and among the general public and its kind of attraction from, hey, this, you know, relates to the sci-fi movies that I've seen or the action movies that I've seen? Like, do you think that that is the concern because you think that that is a leading indicator of the way resources will be distributed in years? I think so. I think that it's a very... So first of all, when you talk about existential risk in the future, you're not very accountable for it, right?

59:17Because you're preventing future risks from every occurring. So how do you measure progress on this? So I think I actually prefer the... I think that the researchers who've articulated a certain risk, like biothreat, and are anchoring it on how, what's the capability for this threat, that makes more sense to me. But But a lot of the discussion of risk has been a bit decoupled. It's been about this future risk, this future generation of models that's much more powerful. And for me, that's hard because it doesn't really give us a way in which to measure progress. How do we know if we're mitigating that risk?

59:51How do we know if we're improving that risk? When you come down to it, often the first step involves the current brittleness of our systems. And but framing it in terms of existential threat, there's no doubt that that's it's anxiety inducing for the general public, which I think we see and how people have reacted and riskfully connected with it. For me, my role as a researcher is to have technically grounded conversations about how our models are used, how they're failing and how they're succeeding. So for me, I much prefer a conversation in which we're anchored to discussions of the risks that we see every day, because that is ultimately where our models are failing today.

1:00:30I also wouldn't be really giving the full pitch if I didn't say our field is moving very fast. But it's a less precise conversation for me when I'm imagining a future model rather than anchoring to the capabilities of our current models. And I think given our current models, there's very pressing current risks. These models have been used all over the world. Yeah. And I haven't seen any data that necessarily demonstrates that some increasing share of resources is bound to go to existential risk because we're talking about them publicly. You know, one could contend that, well, it's obvious the things that people talk about are the things that get funded.

1:01:09But sometimes the conversations feel like, oh, well, you shouldn't work on this thing that you're passionate about because it's not important, you know, from one side or the other. I tend not to think that's true. I think people should work on what they care about. And, you know, I think about examples like open AI historically was very focused on AI safety and these kind of big picture existential risk-esque questions. And yet they're producing practical things as a consequence of that broader mission, right? And so I don't know that the two are even mutually exclusive from that perspective. Like you can be worried about these big picture, long-term questions and still produce practical intermediate results.

1:01:53Here's the thing. The research is too hard to do it if you don't care deeply about it. So you're right that we need to be driven by the questions that give us joy. I also think you're right that typically you can't identify what research directions are promising. And so if researchers care deeply about some specific types of risks, it's good that in general, there's much more recognition for the risks that these models can have and for the role that we should all play in safeguarding the responsible use of these models. At the end of the day, how these terms come to be used and define the public conversation, it impacts the terms of the discourse.

1:02:31And that's the truth. So if we're all talking about existential risk, to re-anchor the conversation to current risks, you do have to intentionally intervene to say, well, this should make their agenda. And that is something that I do think researchers are struggling with because frankly, current risks are not making the agenda in many places. And in many places that are going to inform how these models are developed and where they're deployed and who uses them. And so that's the only thing I will say is like as a researcher who I think my job is to add a technical perspective is that it needs, both need to make their agenda.

1:03:07And right now, I think in a lot of places where the framework and the ecosystem for how these models are developed is being decided, they're not both treated equally. Someone said to me who works in the US Senate, they said to me off the record, they said, you know, response for AI bill has been slowly marching through for the last 10 years. But in the last year, we've just seen remarkable interest in thinking about existential risk and like, How do we do auditing of national security threats? There's certain things that gravitate. And I think that the role of a researcher and an advocate should be to make sure that we don't lose sight of everything that's happening.

1:03:47That being said, it's good that we have diversity in the types of ideas that are being developed. And when you look at it, when you actually look at the implementation, I think there's more in common between all these ideas and the formulations. One of the things I'm increasingly trying to do is just convene forums so that we can see the similarities in the work and the approaches and also stop publishing at the same venues. I think that one of the things that's created this artificial value divide is that, you know, a lot of researchers who are interested in existential risk don't publish in the same venues as like traditional researchers who have worked a responsible way.

1:04:20So that's a value divide, which is kind of an artificial one, I think, in many ways. With the caveat that if they're not talking about actual models and implementations of things, a lot of the venues that the other type of researchers might publish in might not be appropriate. Right. Like there's a difference between a think tanky kind of venue and a more pragmatic venue. Yeah, that's probably another aspect which is causing this hit or miss in terms of the discussions that have been had. Yeah, I think what one thing that's really interesting that you said is that it's much more difficult to establish a degree of accountability over than, you know, this is very far think tanky, forward-looking, future model-based research, whereas folks that are working on kind of more pragmatic, more practical things, they're at least trying to beat some benchmark.

1:05:12Now, that's a whole other question, whether, you know, we've over-indexed on benchmarks and we can probably take that one someplace too. But I agree with you that it's a lot harder to hold that genre of research to accountability, to account. Yeah, we're bad at telling the future as humans, you know, so we were purposely bad at it. And I think then when we, it's one of the things that it's really interesting, even for causal frameworks, that's one of the big motivations is that, oh, we're really good at doing modeling what would happen if there was this intervention versus that. But we're only really good at that for like very short term physical simulations, like with this cup drop, if I released it, we're actually notoriously bad at telling the future for things like, Like, should I change jobs or should I marry this partner?

1:05:57That's why those types of choices paralyze us so much. So in general, I think my emphasis on wanting a technical ground of conversation is just that it does become more tenuous the further you're projecting out. Who would have known that this year would have happened? I think that many of the components were in place. Many of the technical people who worked on it can see the components, but the speed at which had happened and also the speed of which is continuing to happen wasn't foreseeable by many of the people who worked on it, which may be, to be fair, is one of the... And a lot of my colleagues are very invested in this really characterizing the risk of these models over time.

1:06:35This, to be fair, is one of the areas I do resonate. I was just going to say you're making their argument. Yeah, I do resonate with that, that we have this acceleration of research and that we do have to have some ability to at least create cross-institutional conversations around what we see in terms of risk. But whether we're going to do a good job of actually modeling that amplification of risk is a totally different question. What is a frontier model? When does a model become a frontier model? The notion of frontier model, and this is a paper which I actually collaborated on, which I find really interesting, is interesting because it's proposing this idea that there's a transition when a model becomes a frontier model.

1:07:15And that's when it can cause immense, you know, harm to human welfare. But how do we characterize that? When we actually try and anchor that to current benchmarks, what does that mean? We don't even use the same benchmarks across our industry or across research labs. There's a crisis of evaluation happening right now. So do we really have what it takes to pinpoint a single moment when a model ceases to be a normal model and becomes a frontier model? This is the trickiness. And this is where my call for more grounded conversation, because this is what we need to actually characterize and mitigate some of this risk in practice.

1:07:52It's kind of an anchoring to how do we measure progress? Like, when are we riskier? When are we less risky? And that's what's missing from some of these discussions, I think, along long term risk is that there's an inability to characterize it. What someone would say back to me then is that, well, just because it's hard doesn't mean we shouldn't do it. And that's where I kind of share your viewpoint. It's okay for some people to do it. I think the same way it's good for some people to work on many different research directions that can play a pivotal importance. Right now, though, I would say it is a lot of people who are using that framework to discuss these models.

1:08:27And maybe that's where I want people to remember that we also need to make sure that we have ways in which we anchor this to what's actually happening on the ground. And we don't forget the ways in which these models are currently failing. You mentioned the frontier model paper and that this frontier model is kind of a framework for thinking about this. Can you unpack that a little bit? The way you described a frontier model, when I think of a frontier model, I think of frontier as being new and powerful. And you described it as a model evolving to become a frontier model, which there's some dissonance there for me.

1:09:03So can you unpack that a little bit in terms of this framework? Yeah. So this was a cross-institutional collaboration. I actually think this was a collaboration where people had very different viewpoints about what risk is and how do we focus on current risk versus future risk. The paper proposed that a frontier model is a model which has just this likelihood of being able to cause serious impact to human welfare that was negative. And to demonstrate, perhaps in the vein of emergent abilities, when you scale that certain abilities emerge, that you demonstrate these abilities that are considered to be quite harmful.

1:09:42What I would say is, and what I push... And wouldn't LLM be considered a frontier model? It wouldn't under the terms of this paper. And that's where I would push, because I think this paper came out of a workshop. So it was actually really fun. And it was really fun to grapple with this, with a group of co-authors who all had different ideas of what risk is. But I would say, how do you tell? Like, what's the inflection point where LLM is not a frontier model, but then becomes a frontier model? And that's where I think it gets a bit harder. And I don't think that paper resolves that. I think that paper talks about what should happen when it becomes a frontier model.

1:10:18But the meat of the question, what's needed to actually implement this as policy, would be how do you identify when it happens? And is it a binary moment? Can a model be a frontier model, then regress and be a non-frontier? As soon as you start thinking about the nuances, you realize that this is actually a very difficult idea and practice to implement because we don't have standard ways of evaluating. And what do we evaluate for when it's a universal model, when we expect it to do everything? Let's say that you're highly performant in one skill, which might lead to risk. Does that mean that it's a frontier model?

1:10:50And if we only care about one attribute, what is it? Is it biosecurity? Well, if I know that I'm going to be evaluated as a frontier model on just biosecurity, I'll just make sure my model doesn't produce any recommendations for biosecurity. But remember, these models are like that water bottle. Like when you slosh it to one side to solve one thing, you're taking it away from another. So for me, that's the crux of this question. And when you talk about that in that level of detail, you're kind of back to current risk. If we can't even measure current risk appropriately, how are we measuring this transition and flexion point to frontier AI?

1:11:25But I like conversations like that when we're all in the same room working on a paper. And I think that that's the type of grappling we need to make sure that some of these perspectives are anchored. And that's why I like to focus on talking about the challenges we have with current risk, because I still, I think these compound when you project down. It's not that it gets easier, it's compounds. If you don't have a good way to address current risk, projecting that out and trying to identify how that would translate into these inflection points, it's almost like reinforcement learning. All the issues with reinforcement learning is that many of the issues with reinforcement learning are present with static models.

1:11:58You have the issue of you have to identify uncertainty and you have to sample efficiently. That's both true for current models, but with reinforcement learning, it just compounds everything because you have this time element. And so you end up with really precarious algorithms that are really hard to control. Do you have a sense for, you know, when you think about the existential risk community slash conversation slash direction, however you want to characterize that, a lot of the existential risk conversation that I hear about is projecting far forward. Well, I guess they might argue that it's not as far as I might think, but they're projecting to like sentience and the kind of the crux of the issue is sentient AI, whereas bio risk, infrastructure risk, all these other things, they're, you know, maybe not the near term that you're looking at, but they're more near term than sent.

1:12:53They don't presuppose, you know, some leap to sentience. Do you think of that existential risk community as kind of a monolith? anyone who thinks that, you know, and is worried about AI kind of being dangerous on a large scale, or do you recognize kind of differences in both focus and time horizon of folks on that side of the conversation? Yeah. And to be honest, it's not that side versus this side. Like I know a lot, I collaborate with a lot of existential risk collaborators. And in some ways they're my colleagues, right? We're all the same. We're researchers, super smart people interrogating what these models do and what their capabilities are.

1:13:35I would say, I think there's actually a big, you know, there's a variety of perspectives, even within existential risk, mainly because it's so underspecified as a concept, right? What do you mean by existential risk? What is a risk that's significant enough to be a threat to human life or, you know, existence? What's the timeline? I mean, the more underspecified your problem is, the more easy it is to have many, many, many different approaches and formulations that fit within that specification. The same way that I think that, you know, machine learning is specified in terms of a goal, but there's many, many approaches to machine learning, probably even more concrete AGI.

1:14:14This notion of AGI, what it means to different people. If you pull different people in AGI, they'll all say something dramatically different. And it's as much a value concept to people as it is a technical objective. And that, if I were to say, is probably my most grumpy. And I said it to my colleagues, and I think this is part of us all collaborating on this. But when you have something which can mean is very underspecified, it also means that it's hard to hold accountable. And so I think that for essential risks, there's definitely many different approaches. But also when you get down to the technical details, those are often lacking.

1:14:52Although I think there's some excellent researchers who are doing great work in the space who are building technical frameworks for thinking about some of these things. One of which, by the way, is the red teaming for biohazard risk. I think that was done in a really rigorous, really interesting way. I think that there's probably a difference in opinion about whether that's the only type of risk to focus on for red teaming or if it's a case study and how to do red teaming for that type of risk. The authors would probably acknowledge it's a case study, but like there's other risks that matter as well.

1:15:21In parallel, there's very long term efforts, which are, I would say, about things like Frontier AI, which is what does it mean? When does it happen and how does a model look like when it happens? That's not as clear. Yeah. I'm curious how much of that workshop conversation went into naming Frontier AI, because to me, hearing you describe it, it leaves out an essential, risky element that you're trying to capture in calling something Frontier AI. A lot of conversation went into that name. I think that Frontier AI has already been widely used across different, even prior to the publication of that paper.

1:16:01I think Frontier AI means very different things to many different people. Personally, I think Frontier AI in this case, it's referring to this transition to something which is perhaps capable of producing harm. But for me, names are not as important. Frankly, we've been through too many foundation models, Frontier AI. It's more important to be clear about what we mean by the definitions. And that's where I think we have some more work to do. And maybe I'm picking up on the same thing. I think Frontier AI sounds very much like, you know, Frontier models sound like foundation models. It doesn't connotate to me that the thing we're trying to get at is this ability to cause harm in a different way or in a larger scale or on a different time horizon or something, which sounds like it's an important part of what Frontier AI is trying to talk about.

1:16:52Yeah, I think you should suggest renaming it, Sam. We'll invite you to the next workshop.

1:17:01Awesome. Well, I think this may end up being a part one and part two. And I'm sure we could come up with a part three if we kept going, but we're not going to do that for right now. But thanks so much for coming back on and chatting about what you've been up to. Super, super interesting stuff. Yeah, thank you so much. It was really, really enjoyed catching up. And I love the conversation. I really enjoyed the variety of concepts that we covered. I feel like we got through a lot. So thank you so much. This is like an old school length episode. They've been a lot shorter in recent years. So that was fun for sure.

1:17:35Oh, really? Wow. We had a lot to talk about. Very good. Thanks so much, Sarah. All right, everyone. That's our show for today. To learn more about today's guest or the topics mentioned in this interview, visit twimla.ai.com. Of course, if you like what you hear on the podcast, please subscribe, rate, and review the show on your favorite podcatcher. Thanks so much for listening, and catch you next time.