#133 Michael Jordan: How Deep Learning Has Completely Changed Industries

31:39I don't think it'll autonomously make the decisions, but that overall system will feed that information through a doctor to you. And it's a big supply chain of information, as well as all the products that are needed, that allows us all collectively on the planet to be more healthy. And even though we're pretty healthy now, way more than in past history, the pandemic shows us how unhealthy we really are in some sense. Millions of us died. and looking back, one could have done it much better. And a lot of the solutions there are, like machine learning systems thinking and hidden information thinking, incentives kind of thinking.

32:16And some of it is just old-fashioned politics or legal barriers that prevent people from sharing. But some of those are also based on lack of information. I don't know what consequences will be. If I give out some data, maybe I'll get thrown in jail or something. So working out the overall system, which now will start to include legal people and medical people and advocates of stakeholders of all kinds. That's the modern technology. Those are the issues to focus on. And it's not just a one company will do that. It won't be locked up inside a proprietary company. It'll be an overall societal effort.

32:56And so the supply chain knowledge we have from a company like Amazon, it's already clear. A lot of that is actually fairly open. And in fact, a lot of their methodology has been brought to the fore, like in SageMaker and the other platforms that these companies have built, because if they keep it bottled up, they don't have a very good business model. They need to make it open source so people will actually buy it and use it. And so there's not that many huge secrets stuck inside. The detailed data, maybe. but it really is not a just locked up in the company and people that work in these fields are very much aware of this people in health care people in commerce people in finance that oh i got all this data and oh i've got the platforms and the algorithms and if i get good engineers and i work hard on this i can start to do things you you on health care and you this The example of my going into a doctor and having them take a scan or collect data from my current state and then relating that to all of the other people that have provided data in the various outcomes for various treatments and that sort of thing.

34:15That's one kind of network, right, that you're talking about. where every node is informing every other node. So my state of health, my history and treatment and outcome would go into the pot and inform everyone else.

34:46How do you see that being managed? because for that to happen, don't you need a central authority to manage that? Or is, does it become, is there a network effect that you have all these different systems that are tied together that are learning from each other and collectively get better? I don't think anything I said required any kind of central authority. And there's lots of federated learning methods. There's lots of concern about long tails. There's lots of peer-to-peer kind of methods that allow systems of all kinds to be built that mostly don't require central authorities. And again, just think about economies.

35:36The system, the technology that brings food into any city in the world, not just New York, there's no central authority that runs it, certainly. And there's certainly no central authority that designed it. But there were principles that were put in place and there were incentives created and there was roads created because they knew that would help this overall process and so on. So it's more of that level of thinking. And, you know, central authorities are, there's roles for them. Again, economists have talked a lot about this. The tragedy of the commons kind of issues. Free riding kinds of issues.

36:14How do you structure systems so it's not the kind of negative externalities? And when you think about those issues in the new context of all this data flow and all these nodes kind of sharing data and all these competitive issues, it just becomes exciting as an academic and as a person thinking about public policy. Because now, over the next few decades, it's all going to change. The previous systems we had for medicine, the previous markets we had for the economy, for entertainment and all are going to have to complete, will completely evolve in this new world. And if we're smart as human beings, we'll do like, you know, previous engineering disciplines did.

36:55Civil engineering or chemical engineering will organize it as a big set of principles that allow us to think about these overall systems and allow them not just to be a big black box, but allow them to be a working, breathing system that has pieces that will have all this machine learning part inside of it is the technology, but it's not just the big black blocks solved our problems or that we created an artificial entity that solved our problems for us. It's us as kind of creators of principles that figured out how to design systems that we want. And then, you know, the future engineers of this field, whatever you want to call it, will learn all these tools and these principles and then they'll go into particular domains.

37:36I need to design a education system in Southeast Asia, or I need to design a climate mitigation policy using adaptive measurements, whatever. No two systems will be exactly the same. It'll be just like no two chemical factories are exactly the same. There will be an engineering process that brings all these things together. Yeah, that's interesting. And maybe when I said central authority, I didn't mean a government or a company. No, I didn't take you to mean that, but I see you seem to imply that, yeah, there had to be a computer or an entity that would collect all this stuff and organize it.

38:13And like I say, that's just not the, most economies don't have that kind of thing happening. And, and you don't have to think of systems that way. And like I say, we already are all over that in our research field. Things like peer-to-peer or federated or all that already are very much focused on issues like that. Yeah. You, can you talk a little bit about your work in multi-agent learning and how that might relate to this? Sure. Maybe just let me give an example of, I mean, there's a, maybe to give a couple of examples, but one of the topics I'm real interested in right now is asymmetry of information, like I was alluding to earlier.

38:59And economics is something called contract theory, or that's the solution methodology. And in our federated learning situations, we often have this model, this idea that data resides at the edge of a huge network. People have their cell phones or there's cars. And it would be, wouldn't it be nice if you'd collect all that data centrally so you could build a huge mega model and then everyone would profit from that. That's the original vision. And for some cases, that's appropriate. For other cases, you really have to worry about the incentives. Why would anyone willingly allow their data to be sent in?

39:43And if they wouldn't allow their data to be sent in, can at least some of it be sent in? And now you start to talk about trade-offs. You say, I'm going to lose a little privacy, but maybe I'll get some benefit from that. So I like to think about things like recommendation systems for real world goods. Like I go into a brand new city, Singapore, I've never been there. It's 6 p.m. I finished my meetings and I'd like to get dinner. I'd like to be now in a system which knows something about me. I like Szechuan cuisine. I have a certain price point. I'm located in a certain location. And here's the kind of things I've tried before when I'm in a, in Southeast Asia, whatever.

40:27It knows some things about me, but it didn't know everything about me. It doesn't know lots of stuff. So know some limited things about me, the overall system. It takes that information then, and it broadcasts it in a market that there's also on the other side of the market, there's restaurants or whatever that see me as a possible client. And they see other possible clients. Maybe there's about 10 ,000 people in that moment with their cell phone out looking for a dinner. And that overall two-way market does a kind of a matching, but again, based on all the data that's being provided by some history.

40:59And so I now I'm glad I gave up a little privacy because suddenly on my cell phone, there's a bean for a nice restaurant that is perfect for me. And it was matched to me. And it's not just that I got a search engine that gave me a long list of things or some advertisements or whatever. I got something that, which actually was matched to me for good reasons. And the matching means that they're not going to send 10 ,000 people to the same restaurant because different people are going to have different preferences and they can get placed to other places. So I like that kind of way of thinking. It shows that privacy is just part of the trade-off, that sharing data is part of the overall system that I want to get some value out of.

41:40And so people now work on that. How do you value data? How do you think about building systems that value data? And now in situations where like two people have the same data, should I value them equally? Should I, if I have one, is that enough? How do I pay people for supplying data? These are issues that are really touched on lots of fields, not obviously computer science, but statistics, sociology, really, law. And this is what excites me is to think about how to build such systems. The multi-agent learning, what role does that play in this? So let me just, so multi-agent learning is a buzzword.

42:26It just means any learning where there's multiple entities. I like to think of these entities as strategic. You're not just collecting passively from agents and they're just not willing to share. So let me give you this concrete example that helped us think about contract theory a little bit. So the FDA, the Federal Drug Administration, every year has got, well, ongoing, has this problem of trying to figure out what drugs should go to market. But they don't develop the drugs themselves. They test the drugs. And a test costs tens of millions of dollars. It's a clinical trial. So we've seen that with the vaccines.

43:06All right. So the FDA is running a statistical test. So it's collecting data from the clinical trial and trying to decide whether a drug is actually good or it's actually not good or not effective. A big, important decision. It's a statistical decision, though, because there could be false positives. There's data being collected. All right. So now the agents are the pharmaceutical companies, and there's a large number of them, and they're all sitting out there and they are trying to build, instruct new molecules or new proteins or whatever. And they're trying to, and they test them internally and start to learn a little bit about what's promising, what's not.

43:47But at the end of the day, they're going to send some drug candidates to the FDA and the FDA is going to test those drug candidates. And if it passes the test, they're allowed to go to market and they can make billions of dollars. So clearly they're highly motivated to send lots of candidates into the FDA. All right. Now the FDA is sitting there saying, I'm getting all these candidates and for people hoping to get false positives or maybe true positives. Wouldn't it be nice if I could ask those drug companies, Hey, that candidate you just sent me, that's a really good candidate or not. Internally, maybe you don't know if it's really good.

44:26You've done some internal tests. If you put your best engineers on it or not. So if you ask the drug company that, hey, is that a good candidate or not? They're not going to tell you or they're going to lie. And that's what humans do. A lot of times when we ask humans for data, if you ask me, if you're an insurance company, you ask me how much I exercise, I'm going to lie. And so lots of data has that strategic flavor to it in the multi-agent setting. So now the FDA is sitting there and saying, okay, I can't just ask them. and that's called, that's the problem of moral hazard or adverse selection in economics language.

45:03There's people that possess information. They're not just willing to provide it and they won't provide it. And if you ask them, you push them, they'll provide you false information. Okay, so contract theory is a way to set up what are called contracts. And these are menus of options. If you pay this amount, you'll get this. If you pay this amount, you'll get this and so on. Just like when you go on the airplane, you've got business class fare and you've got economy class fare and so on. and you got different services that allows you to effectively charge people different amounts based on their self-selection.

45:33And so contract theory is a mathematical theory of how to do that in some generality. All right. But again, it wasn't ever statistical. There was not a statistical contract theory. You didn't collect data as part of the process and do a hypothesis test. All right. So that's what we've done. We've created this statistical version of contract theory, where the FDA now offers the drug companies a menu of options. If you pay this amount, I'm going to do this kind of test and you'll get this kind of outcome. You pay this amount, you'll get a different outcome and so on. And the drug company will look at that and now you set it up so they are incentivized to mostly send in their effective drugs, the ones they think are most effective.

46:11And now see the FDA is collecting things and they have to pay out a little bit of money. They're making bets really, but they've set it up so that most of those bets are likely to be good ones. Just like the airplane has set things up so they fill their airplane and most people are happy. And that's not being done because this problem hadn't been thought about before. But now I hope you can see that this is a very common situation. I've got lots of agents who possess private data and there's no way you're going to get that data because they're not incentivized to get it, to give it to you. But you'd like an overall system to run effectively by supplying some aspects of that data.

46:50And so you got to set up these options and now you're using data to inform the menu of options. So it becomes a statistical thing, okay? That's probably a little bit of a technical description, but when you really say the word multi-agent, this is what you got to start thinking about. Strategic agents who possess things that others don't possess, it's decentralized and it has to be because people hold onto the value they have internally. They don't want their competitors to know, but they also don't want decision makers to know. And so this is leaked into the machine learning world. When you do things like ask people for loans, when you ask a bank for loans and you're trying to use machine learning internally to assess whether to give someone a loan, when you collect data from people, if they know a little bit about the algorithm being used, it's some neural net or some logistic regression, they're going to fake their data a little bit.

47:45So they seem more likely to be loan worthy. And the system should know that a priori. And machine learning is now starting to work on this a little bit and treat these as fairness issues or adversarial issues and all. But really, a lot of these frameworks already exist if we take the more broad perspective. We're designing economic systems that have got contractual and strategic issues and valuations being made and so on. and how does data inform all those kinds of issues? So I have a lot of work on that. And I would, this to me is the field of multi-agent. It's critical that the agents be strategic for me because that makes it really interesting.

48:24That makes it, there's opportunities for cooperation. There's opportunities for competition. And in that view, agents are human agents or organizations. No, they don't have to be any, they could be all the above. And so for example, think about a system that we're already close to traffic control. If I want to get to the airport, I plug in from here to the airport and I get the fastest route, right? It's the fastest route based on historical data. And in the moment, it's not clear that it's the fastest route and it's getting, the calculations are getting better and better. But the point, the problem is that as soon as you start making this into a decision-making system, not just pattern recognition from past data, you can send lots of people down the same route.

49:21Okay. And you will create congestion and that'll show up pretty quickly. You know, other people coming will see the congestion. They may avoid it, but you've created a congestion. Okay. And these kind of smart pricing systems like you have now in some cities start to feel like a solution. Yeah. I might want to really get to the airport quickly because I got a flight that's leaving and I got someone sick. I got to go see. And so I'm willing to spend a little money or some currency, whatever. Whereas you are, Hey, I can go slowly or, Hey, I wouldn't mind taking a longer trip and see the mountains.

49:59I've never been here before, or I got to stop off at the pharmacy or whatever. And so there's all these human little decisions that kind of decide who gets to go the on a certain route and who would go maybe on other routes. It shouldn't be an arbitrary top-down centralized decision. It should be more like an economy and a market. And now who's going to, are you going to sit here in your car and actually run an auction for people to see who values the certain route the most? No way. What'll happen instead is there'll be like avatars and those will be like computing systems who are like acting on your behalf.

50:33And so think of these like brokers. and so they will look at the situation, gather all these bids, and they will run a little auction on your behalf, just like we have with search engines and ads. And then people will get routed and it has to be verifiable and all that. But this kind of system, we're not that far away from that. But the last step that I mentioned, creating any kind of economic version of the current data gathering systems and prediction systems is still not, we don't have it yet, but I bet in 10 years we will. And the overall effect of this will be social value. You will have much better flow that we, where this economic part gives it to you.

51:19So yeah, lots of great problems of that kind that are multiple strategic agents interacting via data-informed markets. Yeah, although the agent isn't learning through that interaction, it's just drawing data from the agents, right? When you think of autonomous cars in a community where all of the autonomous vehicles are communicating with each other and learning from maybe one, that's what I think of as multi -agent learning, where one car encounters some side case in the long tail and then informs all the others about what to do in that case. that could also apply to the traffic problem you when you get in an autonomous yeah i mean i don't see the distinction we're talking about the overall transportation problem getting huge numbers of people from one point to another and some of that is recognize what's happening on the street currently and that should feed into this system but i'm just saying on top of that should be also a who gets to go where kind of decision uh if everything was literally autonomous and it was just sharing data, then you would still have the problem of you sent too many people down the same street.

52:38And you'd still have the problem of, moreover, you just learned the street is blocked or that some kid had just run into the street, so people shouldn't go there. That's great. But then how do you then percolate the decision-making to who goes where from that? You can't, again, just send everybody one place and all. So the overall system is, and so the one I'm referring to as this little economic layer, but that's a learning system too. When you run auctions, those are learning systems. When you run contracts, those are learning systems. They base how they make their decisions on past decisions and on past interactions.

53:16And if I go into a new city and people drive differently, my avatar should know that or learn that quickly. And people have certain valuations from that should be learned and should come into the way my avatar interacts. I was just trying to emphasize that it's not just humans doing all the decision-making, it's also avatars, but that's why we need the theory of decision-making that includes valuations. And then eventually the valuations have to be okayed by the human. I want my avatar acting on my behalf, not wildly or weirdly. Yeah. Yeah. That's fascinating. You work with Amazon. How do these ideas find their way into the real world?

54:12Very directly. But I do a day a week, I'm an Amazon scholar. And so partly I just go in and it's for my own benefit. I see emerging problems. And a lot of the things I'm talking about here was informed by experiences. Also, I spent time at Alibaba in China watching real merchant producer interactions. I've also spent time in a company that's changing the music business by create a direct consumer or a listener to a producer to brand relationships and all that. I go into companies, I think like any classical engineer or scientist saying, what are the new problems that are emerging? And you see the kind of things I'm talking about.

55:03Logistics that you see uncertainty quantification, you see planning, you see what if experiments, causal inference. You see all kinds of data flows of various kinds. You see the attempts to create market-based mechanisms. And not just Amazon, but just most company has a lot of this going on. And I just wish our academic world would be a little more tied and responsive to that.

55:30And a little more aware of the problems that are emerging there. but yeah there's no secrets that all these companies have got especially a company amazon what i why i chose amazon was because they literally do the end-to-end problem they're taking products made by anybody anywhere roughly and they're sending it all the way to the doorstep of other people and there's human preferences all along there's scale issues there there's uncertainties it's not just they're creating a website or a platform or a service on a web which is fine but then it's a little artificial you have to monetize it with advertising and all that and i'm kind of more enamored of situations where you monetize via people actually pay for things because they're getting value out of the overall process and how do you make that fair how do you make that efficient what are the new issues that arise when you look at that yeah i'm up against an hour but i just want to ask on the music my son is in the music business and is looking at applying AI to to servicing catalogs that have future life can you talk about the company that you're referring to and what it is that they're doing or that yeah no the company is amazing it's called United Masters and I'd highly recommend having a look at it on the web it has over two million musicians mostly young people who are really good at making music but who are never getting paid directly.

57:04You know, when they were never really in a market for their music, they were like uploading songs that were getting streamed by other people who were selling subscriptions. So now, for example, the National Basketball Association is one of the brands that United Masters has signed a contract with. And so music being streamed on the NBA website is actually coming from these United Masters musicians who are often 16 or 18 year olds and they get paid. when their music is streamed. And so it's a three-way market. You've got the musicians and they create music and also you have listeners. And so now the data, it starts to inform you about who's listening to what.

57:45It's a recommendation system in a two-way market. But then you also have brands who want to be associated with certain kinds of music. And so it becomes a three-way kind of entity here, all informed by data analysis. All are formed by adaptively connecting people. And it's actually a thriving economic entity. It's not the old record company business. It's different. It's, I think, revolutionizing the industry. The record companies were not acting in the favor of musicians, of young musicians, especially. And so a new market needed to be created. And I think United Masters is the best example of this.

58:26I'm on the board and I've been involved in this for about four or five years. And the CEO is Steve Stout, who's a legendary figure in the music world, the hip hop world, especially. And he's had the vision to create a company that really turned out to be a multi-setting market. And I've been delighted to be involved in that. And I think it's a healthy thing. So it's AI in the sense that it's networks and it's data and it's preferences and it's all this, all the stuff we've been talking about. But it's also AI that has created new jobs. roughly speaking, you have 2 million people who've got some form of a job.

59:03And some of these jobs are not a billion dollar salary to make your music, but it's maybe$100 ,000. And if you do that over 2 million people, you've added a lot to social value. So I think that's a vision of AI or whatever you call it that I really want to push. Wow, that's fascinating. And yeah, I will look into that. The incentive on the end users and is that it's it's cheaper than streaming than the traditional streaming services or yeah what's the incentive an end user can choose any number of places to listen to music okay the advantage here is that the music that united masters produces and streams you have a real connection between the musician that made it and the people that are listening to it.

59:58And the musician can actually know how many people listen to my music in each city in the United States. That becomes data for them. And they could now discover, for example, they're popular in Dayton, Ohio. 10 ,000 people listened to their music last week. And they can then offer the venue owners in Dayton, Ohio to come give a show there. The venue owners say, yeah, you're popular. And moreover, we know who's listening to you, or we can ping them, say, hey, so-and-so is coming to town, and they'll show up at the venue. So everybody's happy, the venue owner, the people get to see their favorite young musician that no one ever knew about, but they like this person.

1:00:35And the musician's happy because they get to go and maybe make$20 ,000. So that triad of kind of data flow allows a market to emerge. And then And you can have things like, I give a go to this show and I say, look, I can broadcast to the people who listen to me. If you want me to come play your daughter's wedding, I'll do it. That's$10 ,000, whatever. That's what markets do. They create new possibilities when you start to actually link the people who are making the music to the people that are listening to the music. So the listeners are, that's probably the third leg in this in a certain sense.

1:01:11but they get more direct access you can also imagine a version of this i think as it goes forward you'll imagine a version where there's a web presence of the artists where they go and they're they could buy merchandise they could wear a cap with their favorite musician it's not like just beyonce or the beatles it's musician x who lives in inner city baltimore who's really good and I love that musician, I want to wear a cap. Right now, I can't do that. But if I could do, that would be an economic value to me as a cap wearer, and certainly the musician who would get the money for it. And if you do that at scale, this starts to feel like a really powerful use of new data and technology and networking.

1:01:59And this is not, this is no wildly new vision. Sometimes this is called the creator economy. And you don't see that terminology used much in academic circles, but you do see it nowadays in Silicon Valley. And I think it's real. I think people do create, they want to create, and they want to be connected. They get value out of the things they create via all of our technology and not just hand it over to the big companies to be sold on their behalf. Yeah, except what I don't understand and what you've just described is how that's different from Spotify or somebody like Spotify. I'm not saying there can't be an intermediate system that collects things and sends them on.

1:02:37But, and I don't know that much about the Spotify business model, exactly how it's done and all that. But if I'm a musician, I just blindly put my songs up there on the web and Spotify streams them. Okay. I don't have any clue who's listening to them. I can't monetize that. Okay. and I'm not in a market where my songs could be played by Pepsi Cola and I'm connected directly to them, that there's a bidding process where the artists are made available. That level of economic integration is not present in the Spotify model. If I understand correctly, Spotify is really more of a subscription model.

1:03:19So they're making a fair amount of money by people just giving them money to listen to songs. and then Spotify, I think, sends some money back to musicians, but that's not really the business model. They're not incentivized to do that. Maybe they do it out of their goodness of their heart, or maybe they feel they have to do some. These companies give money back to the influencers, partly because they just feel they have to, partly because they have to create a community of influencers, but it's not a very strong business model where the influencers themselves should have economic agency. Economic visibility should be directly connected to listeners, be directly connected to brands.

1:03:55And so that's not the Spotify model. Again, from a pure computer science point of view, bits are bits being streamed from one place to another, but that's not the right perspective. It should be an economics perspective. Who knows what and when, and how can they monetize that individually, not via the company? Yeah. Wow, fascinating. Okay, I'd like to keep talking, but I've taken up too much of your time already. I hope I can have another conversation with you in a year or so. I enjoy talking to you as well. So hopefully we'll talk again. Okay. Thanks, Michael. Hi, I wanted to jump in and give a shout out to our sponsor, NetSuite by Oracle.

1:04:35I'm a journalist and getting a single source of truth is nearly impossible. if you're a business owner, having a single source of truth is critical to running your operations. If this is you, you should know these three numbers. 36 ,000, 25, 1. 36 ,000 because that's the number of businesses that have upgraded to NetSuite by Oracle. NetSuite is the number one cloud financial system, streamlining accounting, financial management, inventory, HR, and more. 25, because NetSuite turns 25 this year. That's 25 years of helping businesses do more with less, close their books in days, not weeks, and drive down costs.

1:05:25One, because your business is one of a kind. So you get a customized solution for all of your KPIs in one efficient system with one source of truth. Manage risk, get reliable forecasts, and improve margins. Everything you need, all in one place. As I said, I'm not the most organized person in the world, and there's real power to having all of the information in one place to make better decisions. This is an unprecedented offer by NetSuite to make that possible. Right now, download NetSuite's popular KPI checklist designed to give you consistently excellent performance, absolutely free at netsuite.com slash IonAI.

1:06:17That's IonAI, E-Y-E-O-N-A-I, all run together. go to netsuite.com slash ionai to get your own KPI checklist. Again, that's netsuite.com slash ionai, E-Y-E-O-N-A-I. They support us, so let's support them.