On Discovery with James Manyika

0:14Welcome to the Where Shall We Meet podcast. Our guest this week is James Manyika, who is Senior Vice President of Research, Labs, Technology and Society at Google and Alphabet. He joined Google in 2022 in a newly created role reporting to Sundar Pichai and has spent the years since arguing that AI's largest contribution may turn out to be science itself. His latest essay in the new volume of Daedalus, the Journal of the American Academy of Arts and Sciences, is devoted to AI and scientific discovery, which features Demis Acebes, Jan LeCun, Yoshia Benjo, Eric Topol and others. He also co-chaired the United Nations Secretary General's High Level Advisory Board on AI, served as vice chair of the US National AI Advisory Committee, and co-wrote No Ordinary Disruption.

1:03James was born and raised in Zimbabwe and read electrical engineering at the University of Zimbabwe before going to Oxford as a Rhodes Scholar, where he took a doctorate in AI and robotics and became a fellow of Balliol College. He was a visiting scientist at NASA's Jet Propulsion lab, then spent 25 years at McKinsey, 13 of them as chairman of the McKinsey Global Institute. We talk about Alpha Fold and the 50-year grand challenge. Did it understand how proteins fold or just learn the grammar? Is hallucination a bug or a feature? Move 37 and the bitter lesson. Touring on why an infallible machine is a boring one.

1:45Is this the first time we have had a cognitive competitor? The impossible problem of alignment. AI as an invention for making inventions. Misuse and who actually benefits from a breakthrough. Let's discover.

2:04Hi, this is Umid Ashtari. And Natasha McElhone. And with us today we have... James Manjiko. Hey James, thanks for taking the time. Well, thank you for having me. I've been looking forward to this. likewise i wanted to start us off by referring to your essay that you've written which is covering a large swath of how ai is helping with scientific discovery right now we'll have a link in the show notes for our listeners and the way we wanted to move into this topic was maybe starting with alpha fold because i think it's something that people have heard and i think it's a good entry into this so the thing about alpha fold is that it didn't really simulate physics it didn't learn the world from first principles that kind of learned the grammar, so to say, and then skipped the whole theory and came up with all these proteins.

2:53I think it would be really interesting to talk about what that means for understanding. And maybe you can talk us through what AlphaFold is and how it is different from what we've done before in science. Yeah, the story of AlphaFold in some ways is pretty interesting because it's emblematic of the kinds of contributions in science that we can see from advanced AI systems. I think it's worth going back to the almost first principles on this. First of all, proteins are the fundamental, one of the most fundamental building blocks of life, of biology, of understanding. And protein structures are the result of amino acid sequences and how those amino acid sequences kind of curl up to create these structures of proteins.

3:42And it was known for a long time that, in fact, it would be nice if there was a way to computationally predict the structure of these proteins, because if you're not able to do it computationally, what you're left with is doing lab experiments, wet lab experiments, x-ray crystallography, one protein at a time to figure out the structure of each protein as you go. So back in 1972, a Nobel laureate, Kristen Ampherson, had made the case that there's got to be some way to do this computationally. This is going to be one of the most important grand challenges. So the question of how to do this computationally kind of stood there.

4:20Meanwhile, humans were painstakingly figuring out the structure of each protein at a time. And before AlphaFold, thanks to the extraordinary work of the protein biobank, we'd actually come to a protein database, we'd actually come to understand the structure of something over about 150 ,000, 170 ,000 proteins doing this painstaking way. But we know that there was something like over 200 million proteins known to science. And just give us a timescale of how much each one of those took to ratify before AlphaFold. Well, it's generally sort of understood that the rule of thumbs that it takes something like a researcher or PhD, a team in a lab, something like three to four years to work through the structure of each protein in the kind of traditional extra crystallography kind of way.

5:07So this is very painstaking work. So thanks to the work of many, many scientists over many decades, we did have the structure of something like 150, 170 ,000 proteins. So the challenge still stood there, which is considered a 50-year grand challenge to say, how could you do this computationally? So there's always this competition called CASPERS, which every year, people, every two years, people would get together to try to figure out, are we making any progress? So to cut a long story short, the first alpha fold basically learned to predict the structures of proteins from the protein structures that were already in the protein databank.

5:46Alpha fold two did something better. It actually tried to learn on its own, including making up its own predicted structures of proteins to figure out what the protein structures were. So it was able to predict the protein structures of over 200 million proteins quite accurately to the angstrom level, which is the kind of accuracy you need to have to be able to make these be useful predictions. So at this stage, something like over 3.3 million biologists around the world, in over 190 countries, are actually openly using these predictions. So now when you're trying to study a protein or trying to understand a particular disease or develop a therapy, you don't have to wait until you figure out the structure of each protein.

6:30You can just look it up from the AlphaFold database. And then off you go. What a gift from the scientists from years before and what a gift from AlphaFold to the scientific community that follows, right? That's quite exciting how this is a circle that closes. the question still remains, does it just learn the grammar or did it really understand how proteins fold? Well, that's the question, right? So that's also one of the questions at the heart of science in many ways, which is, do you just care about making accurate predictions that accurately reflect reality? If that's the question, then you'd say, yes, alpha-fold can accurately predict the structures in a way that we can validate in real life.

7:15The separate question, does that mean AlphaFold understands the grammar of proteins? That's why in the essay I put it in quotation marks, because it depends what you mean by understanding. If you said, can we now write equations or chemical processes or equations that describe how protein structures evolve, then you'd say probably not really. So the question of what does it mean to understand protein structures is still an open question. But, you know, are these predictions accurate? Yes. Can we use them reliably to do science? Yes. So, in fact, we may actually get this wider divergence here that will only get wide as we proceed where we develop models that are predictably useful and predict reality.

8:02then there's a separate question. Do we actually understand what that means and what they're doing? Because scientific discovery, historically at least, has always relied on the fact that if you make a prediction of the world, hopefully along with it, we have a deeper understanding either of the underlying mechanisms or equations or theories that are the basis of that prediction model. But in this case, we don't quite know that yet. in the case of AlphaFold. Is this still useful? Absolutely. I get that it may or may not understand and that in the short term that doesn't matter so much because the progress is being made.

8:40But if it's all predictions, surely, I mean, predictions aren't facts because facts are only things that have already happened or things that are happening now. So given that a prediction is something that's in the future, surely you don't really know whether it's going to work until quite a lot of time has elapsed? Well, if you make a prediction, there's a few ways to validate it. Either you wait for it to happen or you do a lab experiment to see is my conjecture, my structure. In fact, we've actually seen this, for example, in the case of material science, for example, where we now have models.

9:21Yeah, sure. I know that these things have led to incredible breakthroughs, But it's just the whole system being reliant upon probability or prediction. Well, it can't be, right? And I know we don't have anything else. We can't be, no. I mean, if we don't have the data, we can't really predict. So there are areas where there's really scarce data, and we can't make predictions, right? And that's still the domain of humans. But in these situations, and I think what James was saying, is that you're referring to this as the ontological inversion. You have essentially these simulations that are giving us predictions, And then what we do in the lab is validate this.

9:55Well, it was actually the other way around before, right? Maybe you referred to that. Yeah, because historically, the way, you know, science has historically proceeded is that either you have relied on these two pillars. On the one hand, you start with, you know, theories, logic, kind of some view of how you think nature works, either theoretically or because you have intuition. and then you do the other thing, which is you then empirically test that to see, does that match observations or does that match reality? And so this interaction between theory and experiment and observation was the way in which you validated things.

10:35So to the question you were asking earlier, Natasha, is that, so if you have a theory where you make a prediction, you either wait for it, see if you can observe it, or you test it in the real world. Does it match reality or not? If it does, then presumably you'd conclude that that theory must be right, because it seems to match your observations. A bit like when Einstein first came up with special relativity. For a long time, it wasn't actually validated empirically until there was a Michael Morley experiment in 1919, where the idea was to actually see if, in fact, gravity would actually bend light.

11:14And until we'd actually observed that, special relativity was just that, a theory, that hadn't been empirically validated. So if these models are making predictions, you either wait to observe them or you validate them empirically somehow in the lab, observing something in reality. So that's the question, but it still leaves open the question around, okay, just because the predictions are correct or they match reality, does that mean we then understand what the scientific underpinnings of that are? And that's an open question. So alpha-fold is validated in the sense that the prediction for protein structures do match reality.

11:55That's why they're useful. But does that mean we understand the mechanisms by which proteins fold, the so-called grammar of proteins? It's not clear we do. Right. And this goes to what I think Max Tegmark says as well. And I think you referred to it maybe in the essay as well. It's like if you can predict the motion of planets to 99 % accuracy, that may be more engineering rather than real understanding. Exactly. In fact, this creates an interesting dichotomy because people just focusing on building useful things may be satisfied with predictions that match reality. They work. They allow us to model the world.

12:36They allow us to predict the world. They allow us to design things. They're tools for engineering. But there's still a separate question about, does that mean our understanding has been enhanced just because a model is accurate? So these become two separate things. And I think this may be a question in the future, which is, because up until now, we've been able to keep these two things roughly hand in hand. When we have models or mechanisms to make predictions, we've also been able to create the theories and understand how those things actually work. So the two have kind of gone hand in hand. But what I was interested in in your paper was the suggestion that nothing's inevitable.

13:18So yes, there are these predictions and likelihoods, and then they're tested and they're verified. But when you spoke about the hallucinations, which has had hugely negative press, in this context, those could actually be very productive because they were outside of anything that we were able to predict. So I suppose for me, there was like a third possibility that you could enter into this hinterland that isn't human, but that then, as we were saying before, you need to have a sort of interlocator to be able to translate whatever it was that was being well explored, if not discovered, and then sort of fed back to us.

13:59And then that leads to greater questions around whether we then become the minions and we don't really understand anything, which is I know the big fear. And the other thing about predictions is just, are we then freighting all of our resources into something that we've decided already, you know, is important? You talk about representation a little bit as well in the paper. I can't remember who it was who had a particular paper or link to that. You know, like diseases in Africa, for example, which perhaps didn't receive nearly as much attention as Parkinson's or Alzheimer's. Yeah, no, that was a paper by Kelly Chibale who was talking about, he wrote a fascinating paper just on drug discovery in Africa and how that can enhance possibilities in Africa.

14:41But I think I want to go back to where you started, Natasha, which is hallucinations are a fascinating thing in the context of science in the following way. Because typically people worry about hallucinations in AI systems when you're looking for accuracy. So when you're looking for accurate facts, accurate data, and you don't want hallucinations. But at the same time, hallucinations can be a feature as opposed to a bug in the following sense. When you're trying to explore unrealized territory or you're trying to be creative, actually hallucinations can be a feature in that sense because they could point you in directions that are interesting, that you hadn't thought about, that might spark ideas.

15:28So I mentioned a few examples in the paper. David Baker, who also got a Nobel Prize at the same time around proteins, he actually attributes some of the leaps that he made from a few hallucinations that then pointed him in directions he would not have looked at. And then when he did look there, it was able to inspire his own creativity to come up with interesting designs. You often see this, for example, in even in creative work. when creative artists are using AI systems. It's a bit like, you know, when you're trying to have a system that makes a prediction, peanut butter and, in America, you say jelly.

16:06Now, if you're trying to write a creative sentence, you may not want peanut butter and jelly. You may want a more imaginative version, peanut butter and marmalade of something that's creative. So a system that's trying to make an accurate prediction would have probably come up with peanut butter and jelly, But as a creative person, you may not want that. That's too cliche. You want to go outside and see something else imaginative. I'm reminded of something that actually Alan Turing said, which I thought was quite fascinating. He actually made a comment that you don't want intelligent systems to be infallible.

16:42Because if they are, they're not very interesting. You actually want intelligent systems. Intelligent systems are preferred if they can make these intuitive, inductive leaps, make mistakes. and, you know, and come up with these creative things. As long as they can learn from them and explore those possibilities, that's actually what you want. If they're too infallible, then they're not going to do anything interesting. People have said the same thing in science as well, which is, in fact, there's this other tension which I explore in the volume. It is characterized by Rich Sutton, who's a Turing Award winner, described it as part of his, what he called, the bitter lesson, which is that every time we've tried to infuse AI systems with what we know, what we think are the theories, what we think is the science, sure, we get some short-term incremental improvements, but we miss out on these big leaps because we're constraining these systems with what we know, with what we understand.

17:45Why not let them explore broadly? They might even get to territory we don't understand. That's the AlphaGo Zero lesson, right? And that's the AlphaGo Zero example with Move 37, because all the humans who'd ever played AlphaGo for millennia thought, you don't play that way. You should play this way. But AlphaGo went somewhere else that human players had never gone before and came up with Move 37, which generations of AlphaGo players would never have come up with. In fact, they all thought this was a bad idea to play this way. I think the other point maybe to drill in here a little bit, which illustrates this friction to an extent, is, of course, we can make really great predictions when there is ample data available, right?

18:31But the boundaries that we're facing very often are those where we don't have enough data. And in those boundaries, so far, the Einsteins and the Newtons and other people who just ran a counterfactual had an intuition, had a hunch, which is exactly what you're saying, a high entropy state in their brain, which is maybe a high temperature hallucination, was the thing that was generative, that pushed things forward a little bit. Yeah, there's an interesting dichotomy. I think there's a whole history, including the current stage, where you can make predictions based on how good the data you have is, how broad it is.

19:07So you're making kind of within distribution predictions, within the distributions of the data sets you have. But often you want to go outside of that. And you're starting to see ways and experiments and mechanisms to go outside of the data you actually have. So, for example, that's why you see the growth of things like synthetic data creation, for example. In fact, this was a little bit of the case even in AlphaFold 2, as opposed to AlphaFold 1, where AlphaFold 2 actually made up a whole bunch of other training data sets that were not real data, but synthetic data as a way to expand its own training possibilities.

19:48So even the question of data, while it is the case that most of the progress that anybody will point to so far, most of it is based on, is in domains and environments where we have a lot of data, that is the case. But you're starting to see, you know, breakthrough innovations that allow you to go outside of that, either create new data sets or work on other kind of latent representations of things as opposed to actual empirical data. But was that also Richard Sutton, who was talking about that a truly scientific AI would have to be able to go out into the real world rather than be limited by LLMs and reaching a dead end.

20:30Yeah, so there are a number of people who think LLMs are a dead end in the sense that until you go and experience the real world and a big push about this idea of building world models, people like Fei-Fei Li and many others who are thinking about this think that the next set of breakthroughs will come when we have systems that actually go out into the world and experience the world and collect and gather and observe the world, that's the basis for breakthroughs. But some disagree with that. Some would say even the systems we have seem to have been able to learn, for example, physics just by watching videos.

21:12If you're watching videos of objects falling, that may be sufficient to actually start to understand that when objects fall, this is how it seems to work without actually having seen in the world real world objects falling this is also what demis i think demis and dario are basically on the we can scale this up all the way to um you know well not the same way not the same way yes yes i think demis says they're breakthroughs but i think they need we need some breakthroughs right yeah so demis's view i think you know he represents its world in the conversation i have in the volume with them is that I think we're going to need, his view, which is similar to mine's, I think you're going to need both.

21:53Scaling will give us some gains and we're not done with scaling yet. But in addition to that, we may need a few more breakthroughs because we know some of the current limitations of the current scaling mechanisms that we're using today. But there are others, and Dario may be in this camp, who think actually we can scale all the way. Thinking all the way. And is that enough? I mean, is that not the sort of meta question of like a lot of human beings, they don't want embodied AIs. They are quite happy for human beings to have the exclusivity on that and for AIs to remain in a machine because they're already so competent.

22:32Well, that's part of the tension. I could argue, I live in San Francisco. No, I know you're excited by it. That's why I want, okay, I guess cars and there's lots of, yeah, automated. Well, cars, I mean, you know, So I've been living in San Francisco for the last few years. We've been living with the reality of embodied AI systems, which are driverless cars. That's what they are. They are AI systems on wheels, navigating and moving through the world and doing fairly intelligent things. So that's our first real experience so far, at least, of embodied AI systems. Now, in the future, we may have other systems, humanoid robots and all kinds of things.

23:14Some drones already, right? You could argue drones are too. So I think the possibility, in fact, I actually suspect that part of the human anxiety, I think maybe historically less about embodiment and has been historically more about the cognitive part. Because I think we've been quite comfortable with artificial or mechanical systems doing physical things in the world. I mean, we've had electric drills, we've had power tools, we've had bulldozers. You could argue those are physical embodied extensions of things we can't do. But they're not multi-purpose, are they? Well, they are. But I think that the discomfort that I sense most people feel is the fact that this is the first time we have a cognitive competitor.

24:06Yeah, and it's agentic. I mean, if it can make the decisions about what it's going to do and it's embodied, then it has everything 100x. I mean, it's a comparative thing, no? Until you turn off the battery? Until you turn off the battery. Well, this is actually one of the people I quote in the volume is Joshua Bengio. Joshua is one of the Turing Award winners, one of the so-called three godfathers of machine learning or deep learning. and he's always argued that until we solve questions around alignment, we may want to restrict our AI systems to mostly epistemic tasks, Oracle AIs, where these are systems we consult, so they tell us, but we then go act in the world.

24:55Because the concern is that if you're going to go actually act in the world, boy, you better make sure we've solved the alignment problem. because otherwise you don't want systems taking actions when you're not fully understanding what those actions are and on what basis they're being made and what those decisions are being made. So alignment is actually one of the really big and important questions with these systems. Going back to, I suppose, the hallucination possibility, is that what happens if their decisions are much better than human decisions? I mean, we're not great at being aligned between ourselves because we're constantly at war and committing war crimes and even laws and parrots.

25:36Self-driving cars are not going to get drunk on the road, right? Why isn't this pioneered a little more? I constantly hear that, you know, there's lots of nail biting around this and people are very apologetic. Yes, yes, yes. We've got to make them aligned with humans. I mean, I'm being naive. I'm sort of playing devil's advocate, but we are so in conflict. We're so dreadful to one another. It's almost hard to imagine that a machine that has been trained relatively carefully would behave as despicably. The alignment question in this regard, Natasha, kind of has two aspects to it. There's alignment in the sense of, will these systems reflect human preferences or human desires?

26:17And that's a flawed question in the sense that, okay, which human desires, which preferences? Do you mean yours, Natasha, or mine, or Mead's? Or do we mean those of this community or this country or this group of people. So there's always been that millennia-old question, which is easy to state, which is to say, align these systems with human preferences, and my question would be, which ones? So there's that problem. I think there's a separate problem, which isn't really about alignment with preferences one way or the other, but is the question that often causes some existential anxiety, which is what if these systems become misaligned in the sense of coming up with their own goals, their own preferences, quite separate from us?

27:09I mean, this is, I think, my view, low probability possibility that leads to kind of the typical sci-fi scenarios of humans losing control. So I think you've got these two questions. I think the first one is a human question. It's a very hard one. I don't think we've solved it for millennia. It's really a question about, for us as humans, which ones, which preferences, which values, which desires? Should the system, even if I'm talking about just me, do I want the system to be... You're not self-aligned. Yeah, I'm not self-aligned. Do I want my AI to do what I ask it to do or what it observes me doing?

27:48Which is the better version of the environment. The truth versus your... Yes, exactly. Because often sometimes I say, the things I say are not the things I do. I want to be fit. I want to eat healthily. I don't always do that. So if my area observes me eating unhealthily, maybe he thinks that's what James would like to eat. So we'll do that. Some ice cream for James. But that's before we even get to religious differences, sensibilities. Absolutely. I mean, there's a whole set of things. Yeah, yeah. And in fact, the alignment question has another dimension too, which is, because often these kind of normative questions, which have to do with yours or mine or this community or that community or this country, that religion, this culture, etc.

28:29Then you've got questions of plasticity, which is, if we build systems that are aligned with what we think today, are we going to think that a decade from now? Imagine if we'd had AI systems built in 1950 at the Dartmouth conference that coined the term AI, and they built in whatever the values and views were in 1956. and now we're in 2026, would we still like those preferences and those values? Imagine if they'd been baited in for all time. So there's also these questions of plasticity and change over time when it comes to this. But that's why you have the self-improvement being recursive, right?

29:11But that's then what creates the risks of them running ahead and not being aligned with us. I think self-improvement has its own challenges because in the case of self-improvement, This is not a new idea. I.J. Goode kind of framed this back in 1964. It's the question of, so let's just imagine we're able to build systems that are so good that they can also build themselves. So they can build their own successes. And if they're really good, and this is often what people then worry about is you might have a runaway situation. So self-improvement has both extraordinary benefits and extraordinary challenges.

29:52The benefits are, imagine if AlphaFold was self-improving. Now it's understood proteins. It goes on to understand cells. It then builds its own successes that can understand all these other questions that we don't know how to answer today. From a scientific standpoint, you'd say, that's incredible. Let's do that. But from a risk standpoint is if you then build systems that you don't fully understand and you haven't fully aligned, and then they start to build their own successes, then you start to worry about safety and you start to worry about control. So recursive self-improvement is this extraordinary, could be incredibly amazing for scientific discovery and progress, but could also raise these extraordinary risks and challenges if we have unaligned systems.

30:40I think you point to this in a different guise in your essay, which I thought was quite unexpected. And that was the point where you said, when we become the students of the machine's outputs, when it is self-recursive improvement takes it to a place where we can't even catch up anymore, where we need maybe translators to give us lessons about what the AIs are figuring out, or whether we're just taking the observations of those things and we don't even understand the underlying things. We just go with it, whatever it says, because it's working. We don't understand the universe anymore, but our lives are getting better or something.

31:15And the question around meaning there as well. So that, I think, is bundled up with this point. Yeah, it is. I mean, I think to get to that situation, I mean, I don't think you necessarily need to have a recursive self-improvement. I'll give you a good example. And Mario Krenn, in one of his essays, in the volume, points to this where, you know, We already have examples of AI systems doing actually breakthrough things. And we don't understand how they did them. And we don't quite understand what's going on. We know they work because they match observations and predictions. So the question is, you know, is that an alien form of intelligence?

31:58So could we actually interrogate the AI systems to explain to us what they did or how they came up with that intuition and insight? So we may already have been in that situation where we now have these breakthroughs. We don't understand what's going on, but they're clearly breakthroughs. You could argue AlphaFold2 is an example of that. It clearly must have some view of the grammar of proteins. It just hasn't told us yet. And I guess we're doing this with weather forecasting. I mean, us humans, we forecast weather pretty reliably to an extent, and we don't know how it works, right? So we do this ourselves, and now AI is doing it even better, and we still don't know how it works.

32:42Yeah, no, it's a good point. It's a good point, I mean, because this is not a new problem, right? And we've had this with weather, with stochastic systems. There's any number of these kinds of systems where the predictions work. We can't explain them. So we've had these kind of black box issues, epistemic issues for quite a while. The potential here is they could get bigger and even more pervasive. And then you get to the idea that are we going to then need teacher AIs, which then take all these breakthroughs and then explain them back to us? Is there an endeavor to get an answer about how it came up with this?

33:25because you guys can't figure out how it found that solution and it doesn't seem to know. Through a series of interrogation, let's just say you got an answer that was satisfactory, then you'd feel able to move on to the next level of complexity or freedom, allowing it to explore further in a slightly unboundaried or not so limited way. Yeah, I mean, it's actually one of the big kind of areas of research, which is to try to understand what's actually going on inside these systems. It comes with so many different terms, mechanistic interpretability, and a whole range of terms. And so today, what mostly we do is to look at these so-called reasoning traces to ask the LLMs or these foundational models to say, explain your sequence and steps as you go, or tell us how you did this.

34:17Now, that's still an open research question, because take the human version of that. If I ask you, Amid, you just made a big decision. Explain to me how you made that decision. We know with human subjects, what they tell us may not necessarily be what actually happened. Often there's a bit of back rationalization and so other kinds of things. So there's no guarantee that what Amid might tell me is exactly what actually happened. Sure, but they're not human. So I would have thought that the ability to trace back would be far more predictable. But I think that's exactly the challenge, which is because there's stochastic probabilistic systems, there isn't a linear algorithm that's running that I can go and rerun to say, you know, let me rerun this to understand what actually happened in the system.

35:11So because these are probabilistic systems, and as we scale them to be much, much larger, the problem only gets worse and there's one thing to say if it does something that is language-based it's easy for it to explain but if you're doing something in the world of combinatorical um cell structure sequences or like amino acids or whatever exactly you're you're doing things in 50-dimensional spaces to compare and find patterns it's very hard to then explain that to us in some sort of language that we would necessarily understand maybe yeah so in fact you know one of the most more fascinating essays or contributions to the volume is the conversation that I had with Jan LeCun.

35:52So Jan is one of the three so-called godfathers of deep learning. He's spending a lot of time trying to understand these kind of latent abstractions and representations. If you're dealing with these large spaces, think of these as, for example, cell structures or these kind of at the things of the mesoscopic scale where we don't even quite have intuitions about those things or even ways to talk about them. You find the same thing, by the way, in quantum data and in multidimensional quantum states where our intuitions fail very quickly. So it's not clear that having an AI system explain the, you know...

36:36Would be any good to us anyway. Would be any good to us anyway. We may need a translator. So we're back to almost teacher AIs. How do you translate these highly abstract, multidimensional space back to dimensions we understand? And how much do you personally mind understanding or not understanding? I know how curious you are and that your curiosity would be chasing and chasing and chasing and trying to find out the genesis of how that happened. Would you rather have the discovery and the exploration without the so-called security? I actually like both. So the practical part of me, on the one hand, would actually like useful systems, even if I don't understand them.

37:18If something is useful enough that it actually allows me to predict the world and do something useful that benefits humanity. One part of me, call that the engineer part of me or the utility useful part of me, is satisfied if I can build useful systems. The scientist in me is not because I think that I'd like to really understand what's going on. Just because it's useful isn't quite enough for me. And so part of the motivation, I think, for many scientists is, in fact, to really try to understand what's going on. I think engineers has already answered this question, right? A lot of our technology is based on quantum physics calculations and formulas that we really don't truly understand, but they're powering a lot of the lasers and the satellites and all this stuff that's wrong.

38:08It's back to the weather predictions. It's back to the weather predictions, right? And we don't understand them, but we're using them all the time. And so, of course, you can use something that you don't understand and still try to seek to understand it. No, but my point was not either of those things. It was given that, that we've lived with this dichotomy for a very, very long time, even in a non sort of native digital way. Why is it that the general public cares so much about security and safety and guardrails? We talk about this all the time. I'm saying, would you give up the security for the sake of the discovery?

38:47Because that to you is more compelling and exciting and advantageous to human beings. you don't mind not understanding it because hopefully we'll catch up let's just let it run or do you think no we we need those guardrails like a stuart russell kind of mindset yeah i i think this is one of the tensions i think in the context of science and scientific discovery uh i really would try to encourage discovery in the sense that we know that technology advancement of scientific discovery has led to millennia of humanity's progress, whether it's dealing with diseases, understanding the way the world works, inventing and building the world around us.

39:29Everything we're doing now is a result of scientific and technical discovery. So in that sense, more discovery, please. More discovery, please. I think there's a separate question when we think about putting technology like this into society for other uses beyond discovery. So I think in that context, then I think it's important to think about the benefits and the complications, the benefits and the risks. Of like open source models or whether... Yeah, or any number of systems. I think the risks come from not just open source systems, but they come from any number of things. They can come from even people using closed systems to do cyber hacking.

40:13Milicious things. To do malicious things, to do deep fakes. So the risks aren't just the ones from open source systems. So I think when you put these things into society broadly, then the question of the right balance between benefits and utility and risks, I think, becomes so important. But if we're in the realm of science and discovery, I want more of it. I think it's progressed humanity. It's led to breakthroughs. breakthroughs. In December last year, one of the Nobel laureates, Joe Moik, in his Nobel lecture, talked a lot about, he's an economic historian, which I thought was fascinating that an economic historian actually got a Nobel Prize because he talked about just the history of human progress and how it's always been this interplay between scientific discovery and technological progress, the interplay and the bi-directional nature of that interplay, where science has advanced technology, technology, and the tools that we technically build have led to more discovery, that that interplay between those two things have led to human progress, societal welfare.

41:22So in that sense, I want more of that. I think you referred to this as the Enlightenment and Industrial Revolution at the same time. Absolutely. And I think what's fascinating about that is the idea that, you know, AI may be the first technology we've had that has two characteristics that are unique. one is that it's a general purpose technology. Yes, we've had some general purpose technologies before. The steam engine, electricity, you could argue, is a general purpose technology. It's generally in the sense that you can imagine most human endeavors using those technologies, multiple sectors, multiple activities, etc.

42:00But it also has a second characteristic which is interesting, which electricity didn't have, which is this idea of an invention for making inventions. So when I think about the impact of AI in the best sense on human society, I think we almost want to combine. We often make just the Industrial Revolution comparison. I'd say I want to combine the Industrial Revolution with the Enlightenment because the Enlightenment gave us ways of thinking, ways of inventing things. It gave us instruments. It gave us microscopes, telescopes. It gave us a scientific method. These are things that became methods of inventing.

42:40Either they gave us a way to think about the world, the scientific method. They gave us a way to look at the world, telescopes and microscopes. And as a result of that, we invented other things. So that's why I like this combination of the Industrial Revolution and the Enlightenment in the best sense. that having said that, most of these benefits and possibilities also require us that we are thoughtful about a whole bunch of complexities and risks that come with that. And a lot of those are the ones you're referring to, Natasha, which is when we put these technologies into society, how do we think, therefore, about use, misuse, guardrails, and so forth?

43:24Because even with the scientific advances, we still need to worry about misuse in the form of bio-risks, for example. Sure. People inventing viruses with these tools. Yeah. So even there, you've got misuse risks. So the kinds of risks that I worry about tend to be of at least a few kinds of any of these technologies. On the one hand, you worry about misapplication and misuse, whether in the scientific realm or in the societal realm in the sense of, I don't know, deep fakes and those kinds of things. Then you also worry about alignment risks, which we were talking about before. So both things are true.

44:03And I often worry that we too often focus on one and not the other, depending on which one you pick, the benefits or the complexities. I think both are true. And we should be solving for both maximally getting the benefits from this technology, but also taking seriously the risks and the complexities. I just think if you are truly using AI to explore unencumbered by current theories and biases and all of those things, and you really are going to the next level to try and make some breakthroughs in our understanding of why we're here or a whole host of unanswered questions that we have, you just can't have both.

44:51you're not going to be able to have these so-called guardrails. I mean, what would they even be? It's a bit like your other example of, you know, you set in stone something from 1957 and it's no longer fit for purpose, obviously. You know, morality is like fashion. It changes with each generation. So I just don't see how this tension can be resolved. But you can contain the system and have a hand on the plug, right? So it's not that it has to be open in society, running rampant and having access to everything. I mean, even before you put it into society, I mean, even at the very early stages, as you've just said, a bad actor could possibly create a virus in a lab when they were actually meant to be trying to find, you know, a cure for something.

45:34The nuclear codes. Yeah. Somebody in a level five lab, right? Yeah, but I think this is something, you know, I'm a cautious optimist in the sense that I trust human ingenuity to work this out. I don't think you should say, on the one end, everybody here, all these extraordinary tools, go explore unfettered, everybody on the planet. We don't do that with most things. We don't do that with most things. We've also historically been quite good at establishing norms for ourselves, even as a scientific community. I think, for example, back to the famous Asilomar conference in the 70s, when scientists could see where we're going with understanding the human genome and genetics and all kinds of things, before it actually happened, and set norms about what was appropriate to do or was not appropriate to do.

46:31Now, does every person on Earth adhere to that at 100 %? Probably not, because we know there have been some instances of the kind of cloning we would look down on. But for the most part, as a scientific community, in a very large measure, stayed within those norms. Yes, so I think you would have to rely on that. But even the Chinese locked up the guy who did it. What was his name? There was this one person, right? So I think you have to rely on some version of it. But that's already a different era. As you talk about the compression between the question and the answer, you know, where we're living right now, these things are almost instantaneous.

47:09True. But I think I'd rather rely on us evolving those kinds of methods than, because what's the alternative? The alternative to say, stop all progress. Well, I wonder if AI, no, actually the opposite. I wonder if, you know, whatever we've been doing to nature for millennia, you know, AI may do to us. Or maybe not. It may treat us much, much more kindly. I guess, you know, it's a radical thought, but I'm just interested in whether there's anyone in your community that doesn't have a reputation for being a... The accelerationists? Yeah. They're all like, head out to the metal, right? No, there are people who think of themselves as accelerationists, in other words.

47:51But, you know, my own view, and thankfully, you know, many people that I work with, there's the idea that we we have to both be bold and responsible. For as long as possible until you can't. Yeah, I think we have to. And I think we have to, as a community, take both things pretty seriously. And I worry when everybody comes down on one side of that and not the other. I think both things are true. So I take, you know, I'm very proud of the fact that many of us spending an enormous amount of time doing fundamental research and alignment on safety and security? How do we red team these systems? How do we track the progress?

48:32How do we monitor and evaluate whether they're getting to, you know, dangerous levels of capability? How do we keep monitoring that and evaluating that? So I think as much effort should go into that, as much as the effort we put into developing useful uses of this technology for humanity. I mean, I think there's just so much, you know, I'll give you an example. I spent some time for a couple of years, I was kind of the co-chair of the UN's high-level body on AI. And when you look around the world and in some communities and in some countries, people don't have access to doctors, don't have access to schools.

49:17and in fact I remember being struck by the fact that when we were talking about the risks of AI in the high level body we talked a lot about misapplication and misuse but then many people from the global south and other communities said no no no there's another risk we should add misuse misuse I love that term we're in places where what's the alternative compared to what we don't have doctors we don't have access to this that and the other If this technology gives us the possibility of having access to those things, access to information, access to medical advice, you know, unless somehow the world magically helps us get to those things, this technology could be useful.

50:01So I think it's those kinds of uses of technology in the public interest and to help progress society everywhere that I think make this worthwhile. think about the applications and there's a lot of this in the volume by the way in the deadless volume in health care in the health sciences uh in understanding diseases it is already the case and it's a wonderful essay by eric topple i was one of the most cited uh biomedical researchers in the world by the way i think is you know he's written so much but there's now enough studies and examples that show that for example medical practitioners assisted by this technology do much better much better than practitioners not assisted especially in diagnosis i think that's hugely celebrated and known isn't it it's i i'm i don't think anyone's disputing that what surprises me though is you can only get to that place if you've had some unbridling at some point and some ability to be wild west about it and make discovery.

51:06I mean, that's how we've always made discoveries, isn't it? Sometimes by accident, sometimes by going for one thing and then ending up discovering something else. The other question I wanted to ask you is, given that that seems to be the biggest kickback about AI is the safety, that's what I hear all the time. So how much is being invested, you know, proportionately in that compared to innovation and pushing? You'd be surprised at how much work goes into safety because so for example often the safety questions themselves are fundamentally interesting research questions so for example i was describing earlier the questions around mechanistic interpretability it's actually a fascinating question it's no different than uh cognitive scientists who are trying to understand how the brain works right so it's still about capability in some level yeah it's about capability i mean you know we do mris we try to understand which parts of the brain get activated when you are happy, when you're sad, when you're telling the truth, when you're lying.

52:11That kind of research has also been happening recently in AI systems. We want to understand when they make an accurate prediction, is there a part of the neural network that's activated? Which one? We now have these systems that are these so-called mixture of experts architectures, where we We know some part of the AI architecture is doing speech, especially in natively multimodal systems, is doing speech. Some parts is doing reasoning. So we're curious. So on our way to tackling some of these safety guardrail questions, there's some fascinating science questions. Computer scientists in this case are almost like cognitive scientists, trying to understand what's going on.

52:55we've spent an enormous amount of time trying to think about how would you even watermark these systems? The watermark in question is an incredible computer science problem. So in our case, we've ended up being able to, for example, create something called SynthID. And I can tell you the people who worked on that were just so thrilled about the breakthrough because that's a fascinating question about how do you do that. But think about the people who are working on safety in the sense of securing cryptographic systems. The mathematics and science of cryptography, particularly with quantum and quantum AI systems, these are fascinating problems.

53:39So I think it's not as if the safety-related research is boring and uninteresting. Any less exciting. That's good. No, thanks for clarifying. It's actually quite, but I think we always need more of it. We always need more of it, I think, as a community. And I guess the part, you know, to comment on your wild, wild west comment earlier, when it comes to scientific discovery, I don't think it's as much a wild, wild west as people think. So one of the things you see in the volume, and you see it in the scientific journals, a lot of the AI for science work is being played out in scientific journals.

54:19The work gets a lot of peer review. There are scientific trials. For example, we're doing a nationwide trial in the US to try to understand, for example, AI-assistive tools for physicians. So it hasn't been thrown out into the world. It's going through trials and the regulations that guide those things. You can't just put AI-assistives into a doctor's office. So I think society... Well, people already use their chat GPT as their doctor anyway. So that's been crossed that threshold. Well, for individuals, yes. I'm talking about assistive tools for clinicians in hospital settings. I think the questions around what do individuals as consumers want to do, I think those are questions for democracies to work out.

55:08I think societies and democracies should work those things out and decide collectively what they want. I think that's not a question for technologists to decide. I think society should decide, that communities and citizens should decide those questions. I think this is also something that really came out of your paper to kind of go back to it as we close. The reality also for scientists is that we now are on a precipice where a lot of the work is actually asking the right questions. It's no longer really just doing the research and finding things. And I wonder if we're teaching people how to ask the right questions and whether we have to change the way we approach all that.

55:50Yeah, I think you're right. One of the things that I try to point out in the essay, and I think it is the case in the field, I think the impact of AI in science in some ways is changing what it means to do science. because it's changing, for example, I mentioned this kind of ontological inversion that I was talking about in the volume, which is, in some ways, it used to be the primacy was all around what you do in the lab and how you validate things. And it was actually often quite hard to come up with ideas and conjectures. Now, in this case, these systems come up with a lot of them. For example, in material science, one of our systems came up with something like 2.2 million theoretical crystal structures that had never existed before.

56:44Now, the vast majority of those are not stable. In fact, when we went to validation of the 2.2 million, there was something like a couple hundred thousands that were stable enough to be synthesized. And even then, the ones that are being synthesized are more in the tens of thousands, I suppose. So in other words, you have this vast space opened up by these systems. You still have to figure out which of these match reality. And what's interesting about that is that in some of those, there's real novelty. Another example that I actually, there's a paper that I quote in the volume where an AI system is exploring vast territory around potential cancer treatment pathways.

57:31It came up with something like 36 pathways. Half of those were already known, so not that interesting because doctors had figured it out some other way. Another, you know, 12 or 10 or 12 of those were just total nonsense. Then there were another five that were kind of, oh, sort of interesting. And among those, one turned out to be actually truly novel, a truly novel discovery. but it took discerning scientists figuring out what was already known, what was nonsense, to find that novel idea that they hadn't known before. I think you want that. It's still worth it, right? It's worth it. And that one example is actually validated in a lab at Yale, that, oh, this is actually a novel pathway for something.

58:22So it changes the question often to, maybe the challenge becomes how do you validate ideas as opposed to come up with ideas it's much more about validation it also changes the picture from you know exploration because before most scientific experiments started out with the hypothesis so you're always constrained by what hypothesis you have what theoretical frameworks you had uh we're kind of almost the guardrails, if you like, of the range of theories and ideas you would have. It's a bit like the goal player. If you'd ask the goal player, what are some hypotheses about what strategies would win?

59:04They'd just be narrowly confined to the history of human experience playing Go. But now, these systems allow us to go much beyond that. Now, is there going to be some hallucinations in that? Yes. Are there going to be some nonsense in that? Absolutely. Are there likely to be some breakthrough innovations within that? Yes, there will be. And I think that's what's exciting. The possibilities are just, that's what's very exciting. Well, that's a beautiful place to wrap this up. Thank you so much, James, for spending the time with us and giving us a view into where AI is taking us on the scientific world.

59:42I think you already wrote an adendum to this paper that you wrote in January. So hopefully by the next year in January, you can come back and tell us about the addendum to the addendum to the addendum. The American Academy of Arts and Sciences has only done kind of, this was the third volume of Deadless focused on AI. The first one was in 1988. Then the next one was in 2022. And this was in 2026. So I don't know how much longer we'll wait for the next one. But they've all been asking very different questions, which is what's fascinating. Yeah, we didn't even get into culture. The other thing we didn't get into, which, you know, you mentioned Kelly Chibala's paper writing about AI in Africa.

1:00:26I think one of the key questions that's worth keeping in mind is history has shown us that even when we make breakthrough discoveries in science, it isn't always the case that everybody benefits from them. And I think that's one of the things we'll need to make sure happens because we know, for example, what happened in COVID, right? The world, you know, discovered and invented vaccines. For a while, a few countries hoarded them. So I think the question, just because we make scientific progress and even discoveries, doesn't necessarily mean everybody benefits from them. So I think that's one of the things we'll need to make sure, which is as we make these discoveries and these breakthroughs, how do we make sure everybody benefits, that scientists everywhere can make use of them, that society as a whole benefits from them.

1:01:13That problem won't solve itself. We have to actually work to solve that. So I don't just worry about, Natasha, about the guardrails. Of course, those are very important. Safety is important. How we bring the technology to society is very important. But even on the positive side, making sure that the extraordinary progress and breakthroughs and benefits, you know, everybody gets access to them and participates in them is also just as important. Absolutely. Yeah, of course. I think we have a good guardian of those values there. Thanks so much for doing your work, James, and appreciate you taking the time.

1:01:48Thank you for having me. Thank you so much. Thank you.

1:02:02You