#236 Pedro Domingo's on Bayesians and Analogical Learning in AI

0:00The Bayesians are the hardest core of the five tribes, the most tribal. they truly believe that either what you're doing is Bayesian or you're wrong and they will never die the Bayesians are a tribe that comes from statistics so machine learning of course is closely related with statistics it builds on it in many different ways because after all both of them are about building models from data and then acting on them of course statistics has a much longer history statistics and probability going back centuries even statistics in the last 100 years at least, if not more, has been dominated by what is called the frequentist school, as opposed to the Bayesian school.

0:38So the Bayesians in statistics have always been the oppressed minority, and they have a big chip on their shoulder about it. And so what is the difference between them? The difference is about what is the definition of probability. The truth is that even today, for all the thousands of papers or millions that have been written, nobody really knows exactly what probability is. Probability is actually a surprisingly slippery concept. Create an oasis with Thuma, a modern design company that specializes in furniture and home goods. By stripping away everything but the essential, Thuma makes elevated beds with premium materials and intentional details.

1:19I'm in the process of reorganizing my house and I'm giving Thuma a serious look for help in renovating and redesigning. Thuma combines the perfect balance of form, craftsmanship, and functionality. With over 17 ,000 five-star reviews, the Thuma Bed Collection is proof that simplicity is the truest form of sophistication. Using the technique of Japanese joinery, pieces are crafted from solid wood and precision cut for a silent, stable foundation. With clean lines, subtle curves, and minimalist style, the Thuma bed collection is available in four signature finishes to match any design aesthetic.

2:12Headboard upgrades are available for customization as desired. To get$100 toward your first bed purchase, go to Thuma. That's T-H-U-M-A dot C-O slash IonAI. IonAI all run together, E-Y-E-O-N-A-I. So for$100 off your first purchase, go to thuma.co slash ionai. That's T-H-U-M-A dot C-O slash ionai to receive$100 off your first bed purchase. I'm Peter Domingos. I'm a professor of computer science at the University of Washington in Seattle. I'm an AI researcher. I've been doing it since 1998, actually. I'm also known as the author of The Master Algorithm and 2040, a Silicon Valley satire. The Master Algorithm in particular is an introduction to machine learning for a broad audience that was very popular.

3:26And I guess that's why we're here today. That's right. So tell us about the Bayjans, which was one of the five tribes. The Bayjans are the hardest core of the five tribes, the most tribal. They truly believe that either what you're doing is Bayesian or you're wrong. And they will never die. Some of the other tribes, sometimes they look a little tired, but the Bayesians always keep going to thick and thin. Now, why is that and who are they? The Bayesians are a tribe that comes from statistics. So machine learning, of course, is closely related with statistics. It builds on it in many different ways because, after all, both of them are about building models from data and then acting on them.

4:18Of course, statistics has a much longer history, going back to statistics and probability, which of course, again, is closely related going back centuries even. But statistics in the last hundred years, at least, if not more, has been dominated by what is called the frequentist school, as opposed to the Bayesian school. So the Bayesians in statistics have always been the oppressed minority, and they have a big chip on their shoulder about it. So what is the difference between them? The difference is about what is the definition of probability. The truth is that even today, for all the thousands of papers or millions that have been written, nobody really knows exactly what probability is.

5:04Probability is actually a surprisingly slippery concept. But the two main definitions are the frequentist one, which in some ways is the more natural one, which is, he says, just a probability is just the limit of a frequency. For example, why do I say that the probability of getting heads on a coin toss is a half? Because if I keep flipping a coin over and over again, approximately half the time, I'll get heads. And if I toss it infinite times, it'll be exactly half, provided the coin is unbiased. So probability in this view is just frequency. But then there's a lot of problems with this. Like, for example, well, okay, but then what is the probability that Trump will win the election?

5:49You can't do infinite trials of Trump and Kamala running against each other. So somehow that definition seems to fall short. And there's a whole bunch more problems like that, which we could get into. And so the Bayesians actually have a very different reply to this question. They say, probability is subjective. It is inherently subjective. probability is just your belief like i say oh the odds that trump will win are 55 prove me wrong right so actually you're entitled to whatever beliefs you want and the bayans just say how given your beliefs you should then calculate new beliefs such that such that they're all coherent but it doesn't tell you what your beliefs their priority should be and in particular in bayesian statistics and then throw that in Bayesian learning, which again inherits all of that machinery.

6:41So Bayesian machine learning in a way is starting with Bayesian statistics and then adding all the computational power that we have today, including in some ways that weren't necessary in other methods, but are necessary in this one that we can get into. But the basic way that Bayesian learning and statistics work is based on something called Bayes' theorem, which is in fact where the name comes from which is actually ironic because the theorem is not due to Bayes it's due to Laplace so really it should be called Laplacian learning but Laplace already has so many other things to his name that adding one more might be too much so what does Bayes' theorem say?

7:20it says that the way the whole Bayesian point of view or even statistical point of view at least by some standards is there's a probability distribution over everything that could happen in the world. And what Bayes' theorem says is you start with your prior probabilities, which is the probabilities that you assign, for example, to this patient having cancer or this email being spammed before you've seen the patient or before you've seen the email. So these are your beliefs, a priority. And then as you see evidence, you update your beliefs. And the probability of what you see given your model is called the likelihood.

8:02And what Bayes' theorem says is that your posterior is the prior times the likelihood normalized. And your prior times your likelihood is now your new prior for when new evidence comes in. So I'm playing the stock market. I say this stock is going to go up. It goes up or down. I compute my likelihood up with my prior. And now this is my prior for the next day. So this is in essence how Bayesian learning and Bayesian statistics work. Can I just ask, tell us who Bayes was. Thomas Bayes, or Reverend Bayes, was an English Protestant preacher, clergyman, who, you know, in the 18th century, I believe, who, you know, like many cultured people at the time, had mathematics as his hobby.

8:50and you know he so he so probability has an interesting history going back several you know hundred years it actually started out in a very non-serious way as people trying to figure out how to win at gambling right can I predict what are the chances of whatever this combination of dice coming up and so on and it kind of went on from there so when they and what was formulated was in some ways a proto version of his theorem not in you know you can see the idea there but he didn't really formulate it as i said laplace did it but in some ways that he was the one who started things on this track of well let's think about the prior probability and the likelihood and multiply it too and by the way this is something that people often naturally gets people confused frequencies have no problem whatsoever with base theorem in fact you learn base theorem in stats 101 it's one of the first things you learn.

9:45It's a really simple theorem. It's almost just a definition of conditional probability. It barely merits being called a theorem. The thing that makes it controversial is when the Bayesians say, ah, but your prior probability is whatever you want to make it. This is where the frequency will be like, whoa, whoa, whoa, what are you talking about? You can't just make stuff up. And you can see why there'd be a lot of resistance from scientists in general to this idea that, hey, are you just going to make probabilities up? Like, what is that? science is supposed to be objective. But the point that, you know, my point of view is actually the frequentists and the Bayesians both have good points.

10:20So you can't truly ignore either of them. And one very important point that the Bayesians make is that like, yeah, but you frequentists are also making assumptions that are just assumptions. You're just not being explicit about them. So it's rare to be explicit about them. So there you go. And the Bayesians, it sounds a little bit like Euclidean geometry with the axioms and everything is built on top of that. And those are just taken on faith. No, absolutely. So the Bayesians are probably, again, of all the tribes, the one that most strongly believes in, we should solve machine learning from first principles, from a few axioms.

11:05and in particular from Bayes' theorem. And, you know, there's a few others of probability. And that's the right way to do it. Don't talk to me about the brain because the brain is a mess. Don't talk to me about evolution because evolution is a hacker. We have no guarantee that any of those did anything meaningful. I mean, you know, the optic nerve comes out of your retina towards the outside. Like, what the heck is that, right? Why would the brain make any more sense than that? So the whole Bayesian idea is that, like, yeah, you know, I'm going to tell the axioms by which you need to reason. And they even have this expression called turning the Bayesian crank, which is I have my axioms, you bring near data, and I turn the crank, and I produce the output.

11:45And anything else that you do is just wrong. Now, of course, where things get interesting, at least from the machine learning point of view, and in fact, from the statistics point of view as well, is that trying to do this in any non-trivial setting is just computationally too hard. It's completely infeasible. And in fact, a large part of the reason why for many decades, you know, Bayesians were really in the background in statistics. It's just you couldn't do Bayesian statistics. It wasn't possible. And again, part of why they're on their eyes, even within statistics and other disciplines like economics and other areas of science today, is that we now have the computing power to do Bayesian calculations.

12:23But even then, doing them requires doing approximations and whatnot. So at the end of the day, the Bayians wind up making as many unfounded or furious assumptions as everybody else, which is ironic considering where they started from. So tell me how they put this into a computer program that learns. Absolutely. So Bayesianism doesn't tell you, Prairie, what kind of model to use. and so what happens is that there's a bayesian version of everything so for example there's bayesian neural networks there's this you know guy david mckay that did a very brilliant phc thesis on how to make bayesian neural networks and for a while they actually took over the community it was very clever he kind of took it over from the inside uh there's bayesian versions of symbolic learning algorithms there's bayesian versions of analogical learning algorithms so like whatever learning algorithm you come up with there's a you know it's almost you know certain that tomorrow, the Bayesians will come up with their correct Bayesian version of it.

13:27Having said that, something that is very distinguishing, I would say, of the core Bayesian approaches is that they represent probabilities explicitly. The models actually have not just weights or some other parameters with no clear meaning, is the parameters are probabilities. So for example, the most, probably most widely used, or at least best known type of Bayesian model in AI is our work called vision networks. And Huda Pearl actually won the Turing Award for developing vision networks, so it's a big deal. So what is a vision network? A vision network is an answer. It's actually an answer to a problem that goes back to the early days of AI.

14:08We talked about when we talked about the symbolists that, like, yes, the world is full of uncertainty and ambiguity and confusion and whatnot. And in theory, the right way to handle it is probability. I think everybody more or less agrees with that. There's infinite variations, but at the end of the day, the axioms of probability are hard to get around. And in fact, that's what the Bidians keep pounding on. But the problem is that it's computationally intractable. And the first reason why it's computationally intractable is that if I give you, you know, let's suppose I have 100 variables of interest and they're all binary.

14:42Yes or no. Do you have a fever? Do you have whatever, diabetes? Do you drive a car? Et cetera. right? Let's just keep them binary, Boolean, for the sake of argument. There are two to the hundredth possible states of those variables. So in order to naively represent that, I need two to the hundredth probabilities. And there is no computer in the whole. In fact, if you used an atom in the universe to store each one of these, there wouldn't be enough atoms in the universe to just store that distribution. It's just completely impossible. So you have to find some way to summarize that distribution.

15:19And Bayesian networks take advantage of this thing called conditional independence, which is really the crucial property. And so, for example, so backing up slightly, we say that two things are independent when the probability of one doesn't depend on the probability of another. So the probability that I smoke probably has nothing to do with the probability that somebody will observe a shark tomorrow in the Pacific head for a year. They're independent. Now, independent is not very interesting. The interesting notion is conditional impassence. For example, A is conditional independent of B given C, not because A and B are independent, but because once I know C, they become independent.

16:04In other words, all the information about B that A contains is also contained in C. and if you think about it the entire universe works by this principle because at least according to physics as we understand it today what happens to me is conditionally independent of what happens far away condition on what happens close to me so like the light from distant stars can hit my retina but before it hits my retina it will hit a point in space very close to my retina so without conditional independence life would be completely intractable and cognition would be impossible and so on. So what vision networks are a representation that tries, in the knowledge engineering days, people go on, for example, vision networks are very good for things like medical diagnosis because you really do want to precisely quantify those probabilities, right?

16:57These are important decisions at stake. And you can say like, okay, what are the things that, what are the illnesses that could cause a fever? And then you draw this graph, the vision network, with an arrow from all these different illnesses to fever or to high blood sugar. There's one from diabetes, but there isn't one from, I don't know, brain cancer. And then you draw a big diagram like this. And then the only parameters that you have to learn, this is the key, are for each node in this network, given its parents. I only need to represent the probability of fever given the things that cause it, which is a vastly, vastly simpler problem than letting everything depend on everything else.

17:41So the amount of memory. So conditionally independent, such as lets you build a reasonable sized model. Then there's also the problem of doing inference, which is a lot of what, you know, who the Perl worked on in a way that that doesn't blow up. And also of learning this from a finite amount of data, because again, if I need to learn to the hundred probabilities just by measuring frequencies, I will never be done. But once you have a Bayesian network, a lot of this starts to be feasible. It can still be very computationally expensive. And in particular, Bayesians have this thing in which they're different from everybody else, in which they don't just try to find...

18:18Some people would even say that this is what defines Bayesianism, not even just having prior probabilities, is this notion that there is no right model. There is no one neural network or no one decision tree. They are all possible. They all have their probabilities. And what I have to do is not find the best model, unlike what all the others do, is find the probability of each and every possible model, and then to make a prediction, I average over all of them. Now, you can see that from a theoretical point of view, this is very attractive, right? Of course, I don't know. This is induction. I don't know for sure what's going to be the right model, so I should just average over them.

18:56The problem is that there's often a doubly exponential number of models. not just exponential, but like you take all those states, which are already exponential, and now there's an exponential number of models over those. So you have not just two to the n, but two to the two to the n, which is just absurd, right? And Bajins tie themselves in just trying to compute some approximation to this average overall models efficiently. And it's a massive waste of time. I really feel their pain. But just to give you an example, Google, before deep learning came along, their whole ad placement system, which is what made all their money, was a big vision network.

19:38They just had a massive vision network with, I don't know, probably millions of notes that were things like words, right? The words that appear in the page and on the ad that you want to show and various other things and how these things depend on each other and ultimately what they want to do is like predict the probability that you will click on this ad if it contains these words and these links. And that was a huge vision network. So, you know, there's many important applications even today of vision learning. Yeah, just one question on that. How do you take the, what was it, cancer and all the various causes of cancer, was that the example you gave?

20:25How do you compute the probability for each of those nodes? Let's start by taking a simple example. Suppose that I had just two symptoms, fever and blood glucose. What I want to do is predict the probability that you have diabetes. Then what I need, this is a very small model, is this little table that goes like this. If you have no fever and low blood glucose, then your probability of diabetes is low, 0.05, let's say. But if you have a high fever and low blood glucose, that slightly increases your probability of having diabetes. By the way, I'm not a doctor, so I'm just making this up. But the main thing is you still have low blood glucose, so maybe your probability of diabetes is 0.1.

21:15Now, if you have high blood glucose, your probability of diabetes jumps up to whatever, right? And finally, if you have both of them, it's, let's say, you know, 0.8. So you just need to specify probability for each combination of these variables. Now, in a Bayesian network, you only need to do it for a combination of the appearance of that variable in the model, because given those, it's conditionally independent of all the others. That's the big whim, right? I don't need to condition on a hundred things, provided that if I know these two, the others become irrelevant. If I don't know these two, is actually the beauty of vision networks, is that then I can infer the probability of those two from the others, and then from those, the probability of actually having diabetes.

21:54So I don't actually need to know them. But knowing the structure of the world is actually what makes me, you know, lets me have a table with two parents and therefore four lines, as opposed to a table with two to the hundred lines. Yeah. And you said there are, you mentioned the Google example, but what are the other applications that use Bayesian networks today? And is there still progress being made in the Bayesian school into other networks or other structures, I should say? Yeah, so I would say that, for example, medical diagnosis, as I mentioned, is important. So there are some really good, very sophisticated vision networks for medical diagnosis that exist today.

22:45And by the way, they've actually existed for decades now, and they're better than doctors. We've been, you know, like machine learning has been better than doctors for, I don't know, 50 years. The reason it's not widespread is that the doctors won't let them be used. They're the gatekeepers. Why would the gatekeepers unemploy themselves? So there's a very interesting sociological aspect to this. But like next time you hear this big polemic, oh, does Chad GPT do diagnosis better than doctor? Even expert systems pre-learning in the 70s already did, you know, medical diagnosis better than doctors at the things that they were designed for.

23:20It turns out that beating doctors is not that hard. And in particular, famously, humans are very bad at Bay's theorem. They don't understand that you need to take the prime probability into account. And doctors make that mistake too. So if you have a disease that is extremely rare, but you see these symptoms that are consistent with it, the doctor will tend to overestimate the probability, not realizing that even with these symptoms, it's still very unlikely. So doctors really screw this up all the time. I have Bayesian colleagues who say, I went to a doctor and I was just shocked at how stupid he was, telling me that I probably have this and that, and I'm like, oh my God, this guy doesn't even know stats 101.

24:00one. So, you know, visions also have a certain superiority complex, which is not entirely unjustified. But anyway, so medical diagnosis is another class of applications. I would say that in general, and this I think is actually what is important for people to know, is like, what are the types of problems for which these different types of machine learning make more or less sense? For vision learning, I would say it's typically when quantifying uncertainty precisely is very important, right? If just making a prediction, which is what your typical neural network does, it just says, well, it's going to be this, right?

24:38If that's not enough, if you really want, like you're a doctor, you know, to go back to that example, you don't want to tell your patient you have to get, you know, you have to have surgery. That's the patient's decision. What you need to say is, look, you know, there's this probability that you have a tumor. If you don't get surgery, there's this probability that you die. and et cetera, et cetera. And then the patient, according to their own utilities, as they call, can make their decision. A very interesting type of application, which there are some famous historical examples, is let's say, for example, America just lost a submarine in the Pacific.

25:16The submarine has disappeared. Pacific is big, right? You can't search every square mile. And if you just start doing things randomly, you might not get anywhere. Now, Bayesian learning is really good for this because it says like, okay, let me start with my prior distribution over where the submarine would be. It's not uniform, right? It's more likely to be near the routes that they usually take, blah, blah, blah. And now let me start, you know, bringing in all the relevant evidence, right? Was there a storm here? Did I hear some whatever sonar ping there? it's those pieces of evidence might be very weak, such that, for example, a symbol is looking at it would be like, I can't conclude anything from that.

25:56But as you accumulate these pieces of evidence, you know, the probability of each square in the grid starts going up or down. And with enough pieces of weak evidence, you often wind up with a very strong belief that, hey, the submarine is right here. And then you go there and you find the submarine. So subagianism is very good for stuff like that. I would argue that it is, you know, in many ways, the right approach to problems like that, or at least the best one we have at this point. Yeah. One question. I mean, we keep using the term machine learning, but this sounds like a predictive system, which is different than a learning system.

26:34No, absolutely. So very good point. And again, we are guilty of conflating the two. So if you were to do what I just described, let's say you are an admiral in the Navy trying to find the missing sub, right? And I say, let's do this, right? And one of your frequentists would be like, but where do all those probabilities come from, right? And this is where I say, well, you can impede them. You tell me what they should be. A frequentist would never do that. That's just not allowed in frequent statistics, as a result of which there are a lot of problems that they just can't touch. But you say like, no, you tell me, what is your prior distribution over where in the Pacific this submarine is going to be, right?

27:15And I'm just going to believe you. And I'm going to tell you how to then consistently and soundly go from these estimates, right? Now, what happens in practice, as I alluded to earlier, is that like as much as possible, we don't just depend on our prior probabilities. We also use data. but the data they you know again the data about the routes that submarines take and blah blah blah right but what those data do is they cause us to update our prior and again when you have a lot of data in many ways visionism becomes superfluous it's a cost that you don't need to pay when you have very little data unless you use your prior as your host so this is genuinely like so for For example, a famous example is, let's say I'm going to flip my coin and I only flip it once.

28:06Frequency statisticians will use what is called the maximum likelihood principle. To them, that's the right thing to do. But the maximum likelihood principle says that if I flip a coin once and it comes up heads, the probability of heads is 100%, which is stupid, right? And then if the coin comes up heads twice, it's still 100%. And if it comes up heads and tails, then it's 50-50. But the Bayjans don't suffer from this. They're like, no, a priori, let's say you believe that the coin is unbiased. So a priori, the probability is 50-50. And now when the coin comes up heads, that makes it, you know, depending on how exactly you do this, let's say 60-40.

28:42And then if it comes up heads, right? So this is actually a much, I mean, honestly, you would be crazy to not do this when you have a small amount of data. And in fact, what the frequentists do is that they have a bunch of ad hoc techniques, right? This is part of the Bayesian criticism is correct, that they wound up making all these assumptions. But often the assumptions are incoherent, which really annoys the Bayesians. And in fact, one analogy that might be helpful is that these are two churches of statistics, and the Bayesians are the Catholic church. They have one set of true beliefs, and the frequentists are the Protestant.