The Simulation Within with Karl Friston

0:00So guys, I was delighted to learn all the ways that principles of physics could be borrowed by neuroscientists to try to understand how our brain works. Because at the end of the day, there's physics and everything else is just opinion. If you say so yourself. I love how you get to say that. But it's the physics of intelligence. The physics of neuroscience is really what that is about. As you've educated us, the physics is in everything. You just don't think of it as being in neuroscience. Yeah, because we've compartmentalized what people do as professional scientists have their own textbook and their own journals and their own departments at universities.

0:41At the end of the day, we are one. It's all physics, people. That's the lesson we learn. Coming up, all that and more on StarTalk Special Edition. Welcome to StarTalk, your place in the universe where science and pop culture collide. StarTalk begins right now. This is StarTalk Special Edition. Neil deGrasse Tyson here, your personal astrophysicist. And if this is Special Edition, you know it means we've got not only Chuck Nice. Chuck, how you doing, man? Hey, buddy. Always good to have you there as my co-host. And we also have Gary O 'Reilly, former soccer pro, sports commentator. Was that the crowd cheering him on?

1:29Yeah, that's the crowd at Tottenham. Tottenham, yeah. Crystal Palace, they're all. Anytime you mention my name in this room, there's a crowd effect. Gary, you know, with special edition, what you've helped do with this branch of StarTalk is focus on the human condition and every way that matters to it. The mind, body, soul. It would include AI, mechanical augmentation to who and what we are. Robotics. So this fits right in to that theme. So take us where you need for this episode. In the age of AI and machine learning, we as a society and naturally as StarTalk are asking all sorts of questions about the human brain, how it works, and how we can apply it to machines.

2:19One of these big questions being perception. How do you get a blob of neurons? I think that's a technical term. Yes, technical for sure. Yeah. In your skull to understand the world outside. Our guest, Carl Friston, is one of the world's leading neuroscientists and an authority on neuroimaging, theoretical neuroscience, and the architect of the free energy principle. Using physics-inspired statistical methods to model neuroimaging data, that's one of his big successes, he's also sought after by the people in the machine learning universe. Now, just to give you a little background on Carl, a neuroscientist and theoretician at University College London where he is a professor, studied physics and psychology at Cambridge University in England.

3:07An inventor of the statistical parametric mapping used around the world and neuroimaging plus many other fascinating things. he is the owner of a seriously impressive array of honors and awards which we do not have time to get into and he speaks brit yes so he's evened this out there's no more picking on the brit because there's only one of them okay carl fritz then welcome to star talk well thank you very much for having me i should just point out i can speak american as well please don't uh carl uh That takes a certain level of illiteracy that I'm sure that you don't possess. Yeah, please don't stoop to our level.

3:49So let's start off with something. Is it a field or is it a principle or is it an idea that you pioneered, which is in our notes known as the free energy principle? I come to this as a physicist, and there's a lot of sort of physics-y words that are floating or orbiting your work. And so in physics, they're very precisely defined. And I need to know how you are using these terms and in what way they apply. So let's just start off. What is the free energy principle? Well, as it says on the tin, it is a principle. And in the spirit of physics, it is therefore a method. So it's just like Hamilton's principle of least action.

4:35So it's just a prescription, a formal mathematical prescription of the way that things behave that you can then use to either simulate or reproduce or indeed explain the behavior of things. So you might apply the principle of least action, for example, to describe the motion of a football. The free energy principle has a special domain of application. It talks about the self-organization of things, where things can be particles, they can be people, they can be populations. So it's a method, really, of describing things that self-organize themselves into characteristic states. Very cool. So why give it this whole new term?

5:23You know, we've all read about or thought about or seen the self-organization of matter. Usually there's a source of energy there available, though, or it reaches sort of a minimum energy because that's what it, it's a state that it prefers to have. You know, so a ball rolls off a table onto the ground. It doesn't roll off the ground onto the table. So it seeks the minimum place. and my favorite of these is the box of morning breakfast cereal and it will always say some settling of contents may have occurred. Yeah. And you open up and it's like two-thirds full. Yeah, two-thirds of powder. You get two-thirds of crushed shards of cornflakes.

6:09So it's finding sort of the lowest place in the Earth's gravitational potential. So why the need for this new term? Well, it's an old term. I guess, again, pursuing the American theme, you can trace this kind of free energy back to Richard Feynman, probably his PhD thesis. So he was trying to deal with the problem of describing the behavior of small particles and invented this kind of free energy as a proxy that enabled him to evaluate the probability that a particle would take this path or that path. So exactly the same maths now has been transplanted and applied not to the movement of particles, but to what we refer to as belief updating.

6:59So it's lovely you should introduce this notion of nature finding its preferred state that can be described as rolling downhill to those free energy minima. This is exactly the ambition behind the free energy principle, but the preferred states here are states of beliefs or representations about a world in which something, say you or I, exist. So this is the point of contact with machine learning and artificial intelligence. So the free energy is not a thermodynamic free energy. It is a free energy that scores a probability of your explanation for the world in your head being the right kind of explanation.

7:44and you can now think about our existence, the way that we make sense of the world and our behavior, the way that we sample that world as effectively falling downhill to that settling towards the bottom, but in an extremely itinerant way, in a wandering way, as we sort of go through our daily lives at different temporal scales. It can all be described effectively as coagulating at the bottom of the cereal packet in our preferred states. Wow. So you're, again, I don't want to put words in your mouth that don't belong there. This is just my attempt to interpret and understand what you just described.

8:22You didn't yet mention neurons, which are the carriers of all of this, or the transmitters of all of this thoughts and memories and interpretations of the world. So when you talk about the pathways that an understanding of the world takes shape, do those pathways track the nearly semi-infinite connectivity of neurons in our brains? So you're finding what the neuron will naturally do in the face of one stimulus versus another? That's absolutely right. In fact, technically, you can describe in neuronal dynamics the trajectory or the path of nerve cells firing exactly as performing a gradient descent on this variational free energy.

9:09So that is literally true. But I think more intuitively, the idea is, in fact, the idea you've just expressed, which is you can trace back possibly to the early days of cybernetics in terms of the good regulator theorem. The idea here is that to be well adapted to your environment, you have to be a model of that environment. In other words, to interface and interact with your world through your sensations, you have to have a model of the causal structure in that world. And that causal structure is thought to be literally embedded in the connectivity among your neurons within your brain. So my favorite example of this would be the distinction between where something is and what something is.

10:01So in our universe, a certain object can be in different positions. So if you told me what something is, I wouldn't know where it was. Likewise, if you told me where something was, I wouldn't know what it was. that statistical separation, if you like, is literally installed in our anatomy. So, you know, there are two streams of the back of the brain, one dealing with where things are, and one stream of connectivity dealing with what things are. However, we are pliable enough, though. And, of course, I'm not pushing back. I'm just trying to further understand. We're pliable enough, though, that if you were to say, go get me the thing, okay, and then you give me very specific coordinates of the thing, I would not have to know what the thing is.

10:53And I would be able to find it, even if there are other things that are there. Yep. And that speaks to something which is quite remarkable about ourselves, that we actually have a model of our lived world that has this sort of geometry that can be navigated because that presupposes that you've got a model of yourself moving in a world and you know the way that your body works. I'm tempted here to bring in groins, but I don't know why. Chuck injured his groin a few days ago. That's all he's been talking about. We've all heard about it since. Carl, I hear the term active inference, and then I hear the term Bayesian active inference.

11:33Let's start with active inference. What is it? How does it play a part in cognitive neuroscience? Active inference, I think most simply put, would be an application of this free energy principle we're talking about. So it's a description or applying the maths to understand how we behave in a sentient way. So active inference is meant to emphasize that perception, read as unconscious inference in the spirit of Helmholtz, depends upon the data that we actively solicit from the environment. So what I see depends upon where I am currently looking. So this speaks to the notion of acted sensing. You went a little fast.

12:19I'm sorry, man. I'm trying to keep up here. Okay. But you went a little fast there, Carl. You talked about perception being an inference that is somehow tied to the subconscious. But can you just do that again, please? And just to be clear, he's speaking slowly. Exactly. So it's not that he's going fast. No. It's that you are not keeping up. Well, listen, I don't have a problem. okay i have no problem not keeping up which is why i have never been left behind by the way i have no problem keeping up because i go wait a minute okay so anyway could you just like break that down a little bit for me sure i i was trying to speak at a new york pace my apologies i'll revert to london um okay so let's start at the beginning sense making perception how do we make sense of the world.

13:10We are locked inside of, our brains are locked inside a skull, it's dark in there, there's no, you can't see other than what information is conveyed by your eyes or by your ears or by your skin, your sensory organs. So you have to make sense of this unstructured data coming in from your sensory organs, your sensory epithelia. How might you do that? The answer to that or one answer to that can be traced back to the days of Plato through Kant and Helmholtz. So Helmholtz brought up this notion of unconscious inference. It sounds very glorious, but very, very simply, it says that if inside your head you got a model of how your sensations were caused, then you can use this model to generate a prediction of what you would sense if this was the right cause, if you got the right hypothesis.

14:06And if what you predict matches what you actually sense, then you can confirm your hypothesis. So this is where inference gets into the game. It's very much like a scientist who has to use scientific instruments, say microscopes or telescopes, in order to acquire the right kind of data to test her hypotheses about the structure of the universe about the state of affairs out there as measured by her instruments. So this can be described, this sort of hypothesis testing, putting your fantasies, your hypotheses, your beliefs about the state of affairs outside your skull to test by sampling data and testing hypotheses.

14:54This is just inference. So this is where inference gets into the game. These are micro steps en route to establishing an objective reality. And they're people for whom their model does not match a prediction they might make for the world outside of them. And they would be living in some delusional, some world that you cannot otherwise agree to what is objectively true. And that would then be an objective measure of insanity or some other neurological disconnect. Yeah. Really? You know, I mean, is it really? Well, if you project your own fantastical world into reality and you know it doesn't sit, but it's what you want, then that's a dysfunction.

15:40You're not working with, you're working against. But we live in a time now where that fantastical dysfunction actually has a place. And talk to James Cameron for just a little bit, and you'll see that that fantastical dysfunction was a world-building creation that we see now as a series of movies. So is it really so aberrant that it's a dysfunction? Or is it just different? Well, I think he's trying to create artistically rather than impose upon. Yeah, so Carl, if everyone always received the world objectively, would there be room for art at all? Ooh, that was a good question. Yep, it really was.

16:30Well done, sir. I'm going to say, I think I was the inspiration for that question. Yes, Chuck inspired that question. So there's a role for each side of this perceptive reality, correct? No, absolutely. So just to pick up on a couple of those themes, but that last point was, I think, quite key. it is certainly the case that one application of or one use of active influence is to understand psychiatric disorders so you're absolutely right when people a model of their lived world is not quite apt for the situation in which they find themselves say something changes say you lose a loved one so your world changes so your predictions and the way that you sort of navigate through your day, either socially or physically, is now changed.

17:17So your model is no longer fit for purpose for this world. But as Chuck was saying before, the brain is incredibly plastic and adaptive. So what you can do is you can use the mislatch between what you predict is going to happen and what you actually sense to update your model of the world. And before I was saying that this is a model that would be able to generate predictions of what you would see under a particular hypothesis or fantasy. And just to make a link back to AI, this is generative AI. It's intelligent forecasting prediction under a generative model that is entailed exactly by the connectivity that we were talking about before in the brain.

18:04And it's the free energy principle manifesting when you readjust to the changes. And it's finding the new roots that are presumably the more accurate your understanding of your world, the lower is that free energy state? Or is it higher or lower? What is it? It's lower. Yeah, that is absolutely right. So actually, technically, if you go into the cognitive neurosciences, you'll find a big move in the past 10 years towards this notion of predictive processing and predictive coding, which again just rests upon this meme that our brains are constructive organs generating from the inside our predictions of the sensorium.

18:50And then the mismatch is now a prediction error. That prediction error is then used to drive the neurodynamics that then allow for this revising or updating my beliefs such that my predictions now are more accurate and therefore the prediction error is minimized. The key thing is, to answer your question, technically, the gradients of the free energy that drive you downhill just are the prediction errors. So when you've minimized your free energy, you've squashed off the tip of the arrows. Absolutely. Excellent. You're not going to roll uphill unless there's some other change to your environment.

19:41I'm Kais from Bangladesh, and I support StarTalk on Patreon. This is StarTalk with Neil deGrasse Tyson.

19:56so if we think back to early mankind and the predictability so i'm walking along i see a lion in the long grass what do i start to predict if i run up a tree high enough that lion won't get me but if i run along the ground the lion's probably going to get is this kind of evolutionary that we've born for survival yes or have i misinterpreted this completely no no i think that's an excellent point. Well, let's just think about what it means to be able to predict exactly what you would sense in a given situation and thereby predict also what's going to happen next. If you can do that with your environment and you've reached the bottom of the cereal packet and you've minimized your free energy, minimized your prediction errors, you now can fit the world.

20:41You can model the world in an accurate way. That just is adaptive fitness. So if you look at this process now as unfolding over evolutionary time, you can now read the variation of free energy, or it's negative, as adaptive fitness. So that tells you immediately that evolution itself is one of these free energy minimizing processes. It is also, if you like, testing hypotheses about the kind of denizens of its environment, the kind of creatures that will be a good fit for this particular environment. So you can actually read natural selection as, well, in statistics will be known as Bayesian model selection.

21:28So you are effectively inheriting inferences or learning transgenerationally in a way that's minimizing your free energy, minimizing your prediction errors. So things that get eaten by lions don't have the ability to promulgate, propagate themselves through to the next generation. So that everything ends up at the bottom of the cereal packet, avoiding lions, because those are the only things that can be there because the other ones didn't minimize their free energy. Yeah. Unless Gary, you made babies before you said, I wonder if that's a lion in the bushes. Let me check. But if they've got my genes, then there's a lion with their name on it.

22:06That's exactly right. I want to share with you one observation, Carl, and then I want to hand back to Gary because I know he wants to get all in the AI side of this. Yeah. I remembered one of the books by Douglas Hofstadter. It might have been Gödel Escher Bach or he had a few more that were brilliant explorations into the mind and body. In the end of one of his books, he had, was an appendix, I don't remember, a conversation with Einstein's brain. And I said to myself, this is stupid. What does this even mean? And then he went in and described the fact that imagine Einstein's brain could be preserved at the moment he died.

22:47And all the neurosynaptic elements are still in place. And it's just sitting there in a jar. And you ask a question. And the question goes into his ears, gets transmitted into the sounds that trigger neurosynaptic firings. It just moves through the brain and then Einstein then speaks an answer. And the way that setup was established, it was like, yeah, I can picture this sometime in the distant future. Now, maybe the modern version of that is you upload your consciousness and then you're asking your brain in a jar, but it's not biological at that point. It's in silicon. But what I'm asking is the information going into Einstein's brain in that thought experiment presumably trigger his thoughts and then his need to answer that question because it was posed as a question.

23:47Could you just comment on that exercise, the exercise of probing a brain that's sitting there waiting for you to ask it a question? I mean, it's a very specific and interesting example of the kind of predictive processing that we are capable of, because we're talking about language and communication here. And just note the way that you set up that question provides a lovely segue into large language models. but note also that it's not the kind of embodied intelligence that we were talking about with in relation to active inference because there's no the brain is in a body the brain is embodied most of what the brain is actually in charge of is moving the body or secreting in fact there are those are the only two ways you can change the universe you can either move a muscle or secrete something there is no other way that you can affect the the universe so this means that you have to deploy your body in a way to sample the right kind of information that makes your um your model as apt or as adaptive as possible wait so chuck did you hear what he said it means you cannot bend the spoon with your brain right so let's a uri geller right just to clarify okay so what i was trying to hint at because i suspect it's going to come up in later conversation that there's i think a difference between a brain and a vat or a large language model that is the embodiment of lots of knowledge so one can imagine say a large language model being a little bit like einstein's brain but einstein plus you know a hundred um possibly a million other people and the history of everything that has been written that you you can probe by asking it questions and in fact there are people whose entire career is now prompt engineers ai prompts Yeah.

25:38It's funny. The people who program AI then leave that job to become prompt. The people who are responsible for creating the best prompts to get the most information back out of AI. So it's a pretty fascinating industry that they've created their own feedback loop that benefits them. and now you can start to argue you know who where where is the intelligence is it in the prompt engineer as a scientist i would say that's where the intelligence is that's where the sort of behavior is it's asking the questions not producing the answer that's the easy bit it's certainly asking queering the world in the right way and just notice what what are we all doing what is your job is it asking the right questions carl can i ask you this please um Could active inference cause us to miss things that do happen?

26:33And secondly, does déjà vu fit into this? Yes and yes. Ooh. In a sense, active inference is really about missing things that are measurable or observable in the right kind of way. So another sort of key thing about natural intelligence and be a good scientist. Just to point out that sort of noting, the discovering infrared, that's an act of creation. That is art. So where did that come from? From somebody's model about the structure of electromagnetic radiation. So I think just to pick up on a point we missed earlier on, creativity and insight is an emergent property of this kind of question now as answering in an effort to improve our models of our particular world.

27:30Coming back to missing stuff, it always fascinates me that the way that we can move depends upon ignoring the fact we're not moving. So I'm talking now about a phenomenon in cognitive science called sensory attenuation. And this is the rather paradoxical, or at least character intuitive phenomenon, that But in order to initiate a movement, we have to ignore and switch off and suppress any sensory evidence that we're not currently moving. And my favorite example of this is moving your eyes. So if I asked you to sort of track my finger as I moved it across the screen and you moved your eyes very, very quickly, while your eyes are moving, you're actually not seeing the optic flow that's being produced.

28:20because you are engaging something called saccadic suppression. And this is a reflection of the brain very cleverly, knowing that that particular optic flow that I have induced is fake news. So the ability to ignore fake news is absolutely essential for a good navigation and movement of our world. Is it fake or just irrelevant to the moment? If it's the New York Times, it's definitely fake. fake news but it's not it's not so much fake it's just not relevant to the task at hand isn't that a different notion it's a subtle one for the simplicity of the conversation and then I'm reading fake as irrelevant imprecise it's like it's unusable so your brain is just throwing it out basically like don't nothing to see here so get rid of that so Neil this is this is in your backyard rather more than mine But isn't this where the matrix pretext kind of fits in that our perception might differ from what's actually out there?

29:27And then perception can be manipulated or recreated. Well, I think Carl's descendants will just put us all in a jar. The way he's talking. Carl, what does your laboratory look like? Full of jars. Well, yes. Well, there are several pods and we have one waiting for you. Yeah, and the film The Matrix, of course, which came out in 1999, 25 years or a quarter century ago, which is hard to believe. It was a very candid sense that your brain's reality is the reality you think of and understand, and it is not receiving external input. All that your brain is constructing is detached from what's exterior to it.

30:18And if you've had enough lived experience, or maybe in that future that they're describing, the brain can be implanted with memory. It reminds me, what's that movie that Arnold Schwarzenegger is in about Mars? Total Recall. Total Recall. Thank you. Get your ass to Mars. Instead of paying thousands of dollars to go on vacation, they would just implant the memories of a vacation in you. Yeah. And bypassing the sensory conduits into your brain. Of course, these are movies and they're stories and it's science fiction. How science fiction-y is it really? Well, I certainly think that the philosophy behind, I think, probably both Total Recall, but particularly The Matrix, I think that's very real and very current.

31:06And, you know, just going back to our understanding people with psychiatric disorders or perhaps, you know, people who have odd views, world views, to understand that the way that you make sense of the world can be very different from the way I make sense of the world, dependent on my history and my predispositions and my prize, what I have learned thus far. and also the information that i select to attend to so just pursuing this theme of ignoring 99 of all the sensations for example um chuck are you thinking about your groin at the moment i would guarantee you're not and yet it is generating sensory impulses from the nerve endings uh but you at this point in time we're not selecting that so the capacity to select is you know i think a fundamental part of intelligence and agency.

31:59But of course, to select means that you are not attending to or selecting 99 % of the things that you could select. So I think the notion of selection is a hallmark of truly intelligent behavior. Are you analogizing that to large language models in the sense that it could give you gibberish, it could find crap anywhere in the world that's online but because you prompted it precisely it is going to find only the information necessary and ignore everything else yes i know but that's a really really good example um so the yes part is that the characteristic bit of architecture that makes large language models work certainly those that are implemented using transformer architectures are something called attention heads so it is exactly the same mechanism the same basic mechanics uh that we were talking about in terms of intentional selection that makes transformers work.