What it takes to trust AI | Kanjun Qiu

23:56He was like, I can't really justify charging for this. So I think I'm just going to open source it because someone else is going to replicate it. So I think there are great forces toward openness. And from our perspective, there's some additional work needed to like maybe enable business models around open source. There are a lot of open source licenses where you can't really discriminate between personal use and company use. And so maybe that's an easier license or better norms or something like that. On explainability, some of our work at MPU is about how do you like is it possible to ship software that can explain itself to you?

24:30Like, yeah, it seems very feasible. well, maybe all software ships with something that explains itself to you. And like maybe, and then on modifiability, on explainability, there's other pieces that we're doing where we have a lab called Feature Lab that is doing foundational deep learning theory. And that theory is really aimed at understanding, can we make these systems fully engineerable rather than, you know, kind of probabilistic in the way that it is? Is there like a mathematical foundation for deep learning that can allow that? And if that is possible, then it's potentially possible to create these systems with a lot less resources.

25:06And so that would allow more proliferation of models and like base models and also kind of like the training and guidance needed for them to like be good at instruction following, etc. Then on modifiability, I think we're just going to kind of mostly get that for free. Like people want to modify software. You see a lot of people already like making a ton of stuff. And so I think we're going to get that for free.

25:32Aria Finger:There's one more piece on openness, which is about data. So we talk about open software, open agents, and open data. And I think for data, there's like a, you know, I talked about local data, how like agents that have access to our local file system are actually really powerful and kind of like can work with our local data. Like Cloud Code, or Codex CLI, like we really like these local models. And I actually think we might see like MBU is going to invest in an ecosystem of more local tooling that can work with your local data, maybe work with data in your Dropbox. So it's like backed up, but it's still yours.

26:05And so I think better supported ecosystem around things that like work locally on your computer, agents that work locally on your computer and don't necessarily like they're not kind of black boxes in the cloud that you can't really change. I think that's really interesting. And then the second piece around data, so the local data is maybe more private or stuff I want to work with. There's a second piece around data, which is about public data. So earlier I mentioned that quality information really shapes our sense of reality and what decisions we make. And there's a lot of public data out there right now that maybe is not necessarily the information that we'd want to see.

26:41And we also lose a lot of data because it's stored in a server somewhere. But I think there's something really interesting and kind of like a public data backbone for AI agents to work with the Internet. And I think Blue Sky and their AT protocol is an interesting kind of first take at this, where basically the way it works is all of the posts on Blue Sky are in this public data repository. And you as a developer can make any interface you want. And so now you can kind of remix like different pieces of software on top of this public data. And I think that there's an opportunity to have a lot more like public commons data, data as a public good that maybe harkens back to like the old internet, 90s internet, and maybe away from this like Web 2.0 centralization and private servers.

27:28So all of that to answer like those are some components of things that I could see helping us become systemically more set up to have trustworthy agents that work for us.

27:38Aria Finger:I love that you're bringing up public data. Just I'm like real deep on the Yimby Twitter threads. And especially in New York, there's like big questions about cars versus city bikes. And so someone was spouting some nonsense about city bike and someone immediately was like, well, I can access the data in New York City on every single city bike station. How many bikes were dropped off? How many bikes were picked up? Oh, someone left that station at 259. And I know that 50 people used it that day. And when people park, it's only five. And so one of the things I'm excited about for public data is obviously the vast majority of people are going to do nothing, are not going to build an agent, are not going to look at public data.

28:16Aria Finger:But if we can get – like, first of all, there's so much government data out there that someone who now can utilize that can do amazing things with, you know, CityBike and more. And there's even more that we should categorize. Like with our new abilities, like we could supercharge so many government services if we actually had the data instead of it living on PDFs in like the, you know, middle of DHS files that no one ever sees. If we can bring it to light, like we can do so much more. Yeah, I think this is super interesting. One example recently is the UK added a tax on pubs or something like that.

28:53And somebody made a map of all pubs that would be affected. And that helps all of these local business owners figure out, like, how does this tax affect me, et cetera. Similarly, we had the thought when Trump defunded a lot of projects, oh, maybe it would be helpful to, like, make a list of all these projects and, like, do that programmatically and make it so that it would be easy for people to collectively fund them. I think there's something really interesting here about data and specifically what you said about kind of a person being able to correct the like public discourse. Because, yeah, in the theme of this podcast possible, like right now we have an Internet that has a lot of fake news on it and like false information.

29:37And in theory, with language models, we should be able to with language models and a better public data record system, question mark. Like we should be able to have a system that tends toward truth and tends toward people able to being able to have more truthful understanding of reality. And so that's like starting to become possible. And I'm like super excited about that path. Well, I think part of what you need, and I agree that like lots of data being more publicly and generally available is good. As you just mentioned, there's a lot of that which is worse than garbage. on the internet because it's deliberately misleading.

30:19Right.

30:19Aria Finger:Garbage is just neutral. Yes, exactly. So a lot of the anti-vax stuff is in this vector. But another thing that is important is how do you weave the logic of understanding this data together? In addition to the data layer for making a more humanist set of things, What are the things that you think need to be done on the reasoning side and on the understanding of the quality of data side? Yeah, this is a great question. And it's actually something that if somebody has a project that is trying to work on like excavating truth from a lot of data streams, I would love to fund that project or have them work on it and imbue.

31:05Like I am not sure what the answer is. I think there need to be really thoughtfully designed, like a really thoughtfully designed layer kind of on top of the Internet. An example of a problem that cropped up recently is there was some fake chat GPT generated news that was published. And then another model cited that, like something else cited that news and something else cited that news. And then now it became real. And then the models thought they were real. And you're like, OK, that's not good. Like we need some way to like deeply investigate this whole chain of source citations and to understand like is it actually real?

31:39And so maybe that looks like agents that are investigative agents that then report their findings just like a scientific meta review into some central database. Like I guess decentralized but like central database. Centrally accessed even if decentralized in terms of where all the data is and the links between it. That's right. And so yeah maybe that's what that looks like. I'm not quite sure what works, but there's something interesting in the scientific kind of process that's been developed. And I think these are actually like social process innovations plus technology to support those social processes.

32:14I'm not quite sure what it looks like yet. One of the frequent things that I find challenging, and RFK Jr. is a poster child for this, which is people misunderstand to say, well, science, that broke there. Yes, science is a continually learning mechanism with reproducibility. The answer is not it's always right. The answer is it's on a trend towards being right in the process. Because like most people are not good at understanding statistical argumentation. What are the ways that we get to common knowledge, like common sourcing? What are some of the ways that if we're steering AI systems to get to this, and I deliberately use the term, possible future, What are some of the things you think we need to be doing as technologists and as society in order to get there?

33:03Yeah, on the like possible futures, I think this is one of the most, my most optimistic views of what progress looks like with AI is that AI by its nature synthesizes a lot of conflicting information to present to us what it seems to understand as the, like what its best understanding of the truth is if it's trained well. I actually have a maybe hot take here where like I think one of the most important things I think there's like a systems thing and then there's a moral thing so there's a systems thing which is like anthropic making AI systems that are truthful as truthful as possible and I think anthropic culturally really cares about that and like I love that they exist I think there's a moral thing which is as technologists we build incredibly powerful things that change people's lives and affect their day-to-day but we don't have a variant of the like Hippocratic oath and like something I've been kind of noodling on is like what would the 10 commandments for technologists look like one example is I've been thinking about why some things feel more fake than others for example like if we're scrolling Instagram at 3 a.m that feels fake but like watching a great movie until 3 a.m feels kind of more meaningful more real or for eating processed food it feels kind of fake but like if we're eating really nutritious food it feels like more real um or if we see someone beautiful in a beauty magazine but they've been photoshopped that feels fake but like in real life they're like really vibrant that feels real what's going on and i think there's this thing that can happen uh to humans as we get more powerful technology that i'm calling sensory hijacking so technology hijacks these sensors that we have to determine actually how to live really meaningful and healthy lives.

34:49We have really good sensors for that. But they're very, very hackable. And so these systems that capture our attention, hijack our attention, are hijacking some of these sensors that exist for that, you know, that determining a meaningful life. And so like one example of a commandment might be like, thou shalt not hijack other people's sensors. Another one might be like, thou shalt not build systems that propagate false, you know, reality. Or like, that's another kind of sensory hijacking. You're giving people wrong information on purpose.

35:22Aria Finger:When the rubber hits the road, how do you make this happen? Is it a culture change? Is it government intervention? Is it all of the heads of the AI labs getting together and being in conversation with each other? I think so many people, sort of in this room certainly, but in our position, want the better future. They want to drive towards these sort of humanistic goals. But we don't want to stop technological progress because we see how it makes people's lives better in so many ways. There's also geopolitical implications. But I think people are still struggling for then like, how do we cross that chasm?

35:57Aria Finger:Because you're talking about some of the harms, but how do we get to the other side? Yeah, I think some of the things I talked about, like, I think it requires a lot of invention, actually. Some invention and like some kind of like internalization of morality and like excavation of what is moral technology. Invention at the systems level, you know, how do we invent systems that actually do tend toward truth? How do we invent systems that allow for more openness and trustworthiness, like software systems, digital systems? How do we invent economic systems that allow for more agency and more authorship, especially as, you know, AI becomes more powerful and is able to do more of people's work?

36:39Like, this is an open question I have. I don't really know how that's going to shake out. So there's, I think, a lot of invention required over the next few decades to get here. And I think it's, if I knew what exactly to invent, I would have done it already. Yeah. Well, it's a process. It's a process. That's right. It's a lot of what we're trying to figure out at MBU and why we're such a strange company is like trying to invent these systems that allow for a future that is like systemically more robust and more trustworthy and more agentic and not just, you know, not fragile. Of technologists, you are at the far edge of philosophically sophisticated, right?

37:22There are many technologists who are not. Coming to a growing common perspective on morality is also just like common knowledge, or common source, there's kind of like the, okay, how do we get to that common morality, and how does AI play into that? It feels like, as humans, we're always trying to figure out what leads to a more meaningful life. Why do we exist? What is a good life with the amount of time we have on this planet? And some of us figure it out early, like you. Some of us figure it out late like me you know quite sure of this but that's okay it's taking me a long time to be like oh finally i can do the things i want i don't just have to do what people expect of me um there's something really strange and interesting about

38:16meaning agency slash authorship i've been using this term authorship and being controlled there's like some dynamic here i'm going to try to point at it I think we are controlled often by the world that we live in. Not necessarily badly. We don't murder other people. It's a form of being controlled. I think we do internalize a lot of maybe like, I'm going to use this term, it's not quite right, oppressive or controlling narratives. And we do exist in some kind of controlling systems Like my phone is a controlling system. And so I think if I were to answer like what is human good or like how do we build a positive future for humanity?

39:09It's something like we want a society that's able to continue to make progress on how like what a meaningful life is. I think there is real kind of like thinking and quote unquote progress to be made there. I think we understand a lot more about it today because we understand our world and ourselves and our mind and like our what makes us human and what you know why is that valuable. So I think we want a society and systems that like let us continue to make that progress. And at the same time, so that's at the like social level, collective level. And then at the same time at the individual level that like protects us from negative things, negative, that's like negative liberty and also kind of like induces positive liberty, positive freedom, like induces agency and sense of authorship and like protects us from things, systems that might control us or tell us false things.

40:02And so the combination of these two, I think, in the context of technology is some of what we've talked about today, but not comprehensive. Like in the context of technology, being protected from some of the negative control might be like being protected from a system that might manipulate us, being protected from easily ingesting false information, being able to trust agents that operate on our behalf because they're systemically trustworthy. Those are like protective stances. And then there's like a positive freedom aspect of it. A lot of what you do, Reed, I think, to kind of like give people examples of what's possible.

40:35Like it's really helpful to see what somebody else did. Like when you wrote this book, Superagency, I was like, oh, my God, I can do this. Whoa, that's so cool. And it like actually unlocked a lot of writing for me. That was really cool. Thank you. That is great. Yeah. Yes. So like more examples of people like doing stuff that's possible. And then also updating some of our social narratives that kind of keep us operating in like, oh, we're just this way or just that way and business as usual. My co-founder has a really interesting thought experiment, which is let's imagine that 10 ,000 years from now, there are some human descendants who live on a faraway planet orbiting a faraway star.

41:16And they see Earth through a telescope. In the case where they have forgotten where they came from, Earth is just another planet. It has no meaning to them. And that's actually really interesting. In the case where they have remembered where they come from, Earth has this deep meaning to them. And so it actually says something interesting about meaning. Like, what is meaning to people? There is something in this thought experiment that I don't quite understand that says something about meaning.

41:46Aria Finger:Well, I was just thinking about a hypothetical where you had your personal AI agent and we've solved the misalignment. We've solved the trust issue. You trust it. It has your best interests at heart. And so if you're feeling down, it knows you. So maybe it tells you about meditation or maybe it tells you go to church this Sunday or maybe it tells you to get a new job that has more purpose. There's some universal ones and some go get your flu shot. It's going to be a really bad flu season. And so like it does all the right things. And so I guess I have two questions. One is like, is this the utopia where everyone has this personal AI agent that we trust and is aligned and we are on board?

42:26Aria Finger:And then you made me think, how does that, like the good news is that helping others and having a purpose is actually good for you. But is there some misalignment because it's good for you, but it's not necessarily looking out for society. Is there a trade-off between the personal and the collective if we have this magical AI agent? I think this is a really interesting question and kind of speaks to the amount of power someone who trains these models has, which is, you know, you can think of an agent that has both your values and societal values at heart. And maybe the agent that has societal values thinks, okay, what is best for society?

43:05Well, if you buy an SUV, here are the tradeoffs for society. However, you value freedom and personal liberation. And so maybe you want an SUV. So an agent that really strongly considers both might say, you might want an SUV, but know that the tradeoff is that you're making things worse for other people. However, an agent that doesn't consider that and only considers your values might be like, you should get an SUV. Like, this is best for you. And so I think I'm confused about like I personally would prefer the agent that informs me of the tradeoffs of my decisions. I think that's good and kind of like helps me become more collectively oriented.

43:43But not all agents are going to do that. Well, and part of the challenge when it gets to human nature is that many human beings prefer certain kinds of self-deception. They prefer to, as opposed to saying, hey, when I get this SUV, I'm actually having the following kinds of impact on the world. They go, no, no, no, no. I don't want to know that. I don't want to know that. I'm not going to buy this agent. I'm just being a good parent.

44:08Aria Finger:I'm keeping my kids safe. Exactly. And my first identity is as a protector of my children. Exactly. So it's really easy to just say it's totally fine and it's what I should do as a moral human. Exactly. Because my hope is that with AI and its ability as the best learning tool that we've constructed in human history is that helps us all become somewhat more truth seekers. Yes. It's a definite hope. It's a definite trying to steer in that direction. But it's a very complicated path. Yeah, that's my hope also. Like, you know, I think maybe that a lot of us lose faith in education because we're not really met.

44:48We're either overly bored or overly stimulated. It's like too much on either side. And like these systems can meet us where we are. So maybe that will help us get closer to being truth seeking. I think humans do want to at our nature understand what's going on. And that like we only lose touch with that when we're really distracted by all this stimulus. or like distracted by hurts, old hurts. And so maybe like, you know, part of it is like AI being better tutors, helping us understand things better. Maybe like part of it is like everyone meditates with this new method that's really efficient and like we'll all see the true nature of our experience as like a field of compassion and love and then that will solve all the world's problems.

45:32I think it'll be like a slow progression toward something like that, hopefully. And just so that our listeners don't think it's unduly optimistic or Pollyannish, the answer you just gave actually, in fact, is deeply within the Western philosophical tradition because it's Socrates. And it's deeply within the Eastern tradition because it's Buddha. It's true. And both of those two figures would say the thing you said was exactly right. And we just need to help people on the dialectic towards it. Now, I think there's complexities to it, but it's a good, it's not an unconsidered objective that actually, in fact, leads us into better places.

46:12Yes. I love that. Yeah, we have an opportunity to kind of really lean into Socratic method, Socrates, and really lean into, like, what Buddha discovered about the fundamental human experience. And maybe that would teach us something.

46:24Aria Finger:So we've been talking a lot about sort of philosophical power in the future and how do we point to this humanistic future. but economic power is going to be critically important and how do creative people retain those economic rights? How do people even have jobs in the future? And so what do you think needs to change about ownership and monetization in the future as we think about AI? You run a research lab and you need some feedback signals. And so what are the things that we should use day to day as we sort of hurdle towards this future when we think about the economic piece of it? I think I personally am most confused about the economic aspect of this entire situation.

47:04I have this model of capitalism that I'm not sure is correct, but it's something like there are two core mechanisms. One core mechanism is the markets, which is like a coordination and resource allocation mechanism. And the markets are amazing because they allow people who are like more self-interest, like they tap into human self-interest and allow that self-interest to be super beneficial to other people. And then there's a second mechanism, which is like the idea of return on capital and kind of being able to invest in something and get more returns on capital. And I think when people talk about capitalism, they often combine these two mechanisms.

47:37They're like, oh, capitalism is like this whole thing that like, you know, people invest and like get money back and like, but reading Adam Smith, Adam Smith is like only talking about the first mechanism. And he's actually very concerned about like monopolies and rent seeking and the kind of perpetual growth machine that we currently live in. I think that the perpetual growth machine combined with AI that's really, really smart causes serious problems. I could see AIs that do a lot of kind of hijacking or hacking of our existing system to accumulate capital or resources. So like people who get really good at high frequency trading, I was a high frequency trader a long time ago, and like AIs can definitely do this well.

48:22But is high frequency trading actually connected to real production, like real production value. And that was one of Adam Smith's other concerns is like financial returns that are disconnected from real value of production. And so I think we currently live in this economy where like a lot of our financial returns are disconnected from real production. It's like kind of hard to evaluate exactly how or in what ways they're disconnected. And so for me, the only part of it I know what to do about is that I think I want to see if we can make software go from a renter's economy to an owner's economy.

48:58Right now, we're in a renter's economy of software. Pretty soon, I think we can be in an owner's economy of software. Seems possible. Seems like, you know, the production costs are getting really low. And so that'll change some things, but I'm not sure it'll address much of the problem. You know, I don't know how to think about the rest of like, yeah, how do people get paid? Who gets paid for what? Who makes the money? Where does the money go? Does the money accumulate? Does it distribute for some other reason? Like maybe people are super creative and then somehow we end up with like substack everywhere for everything.

49:34So now rapid fire, which is, is there a movie, song, or book that fills you with optimism for the future? I love The Beginning of Infinity by David Deutsch. I think it is one of the deepest excavations of what progress is that I've ever read. I read it in 2020. And before that, I was like, I don't know if progress is real. There's this kind of meme among historians that history is cyclical. What's the point? What is progress? I can see that our standard of living has improved. Is that progress?

50:09Aria Finger:It seems like progress, but like, what does it actually mean? The idea he puts forward is that the construction and layering on of good explanations is what results in progress over time. And so a good explanation is an explanation that explains the phenomena and has reach. Like it can also explain other phenomena. For example, he gives a really interesting example. Like the explanation that the sun rises because Helios pulls it in a chariot every day is like, so his first criterion is that it's hard to vary. Well, you can vary a lot of things about Helios. His hair color, his eye color, how tall he is, what the chariot looks like.

50:51And the second thing is that it doesn't have any reach. Like it can't explain anything else in the world. Whereas the like axis tilt theory of the sun explaining why the sun is where it is in the sky has a lot. It's very hard to vary. And it has a lot of reach. It also explains seasons. And it also explains like why seasons are different on different hemispheres. So the idea that the kind of layering of good explanations is a sign of progress, I think is actually the most convincing argument I've heard for what progress is. And this book fills me with optimism. You know, even if we fulfill Maslow's hierarchy, like what cultural values make sense?

51:27And like how do you there must be some like more fundamental underlying explanation about why love and like kind of like altruism is better than hate and anger. But like what is that explanation? And like I think his argument would be the excavation of these good explanations is what leads us to these progressions in moral philosophy and in economics and these other things.

51:51Aria Finger:What is a question that you wish people would ask you more often? I think a thing I don't talk very much about because I'm like an AI person, quote unquote, I don't have very many opportunities to talk about it, is I actually have systematically removed my anxiety over the past 10 years. And like I used to be a super anxious person and being a founder was really hard. And like I had all these voices in my head and they were really critical. I have Asian parents and Asian culture and like very critical voices in my head from my childhood. And I think people assume that anxiety is just something you have to deal with.

52:22And the voices in your head are just like something you have to learn to befriend. But it's false. I think that trauma is basically or like these voices are basically overfit models. And as we grow up, our data distribution changes. And what we have to do is go access those models and like systematically update them by giving them new data about our lives. And so that's what I've done with all of these different modalities. And it's like really effective. Do you have advice for someone who's experiencing that and wants to fix it? Say. No, no, not me. Asking for a friend. I don't have anxiety, so I don't know about it, but I know it's crippling for people.

52:55Asking for a friend. Truly. Truly, you don't have anxiety. No, none. That's amazing and unsurprising. That makes so much sense. All right, so where do you see progress or momentum outside of AI that inspires you? One of my best friends works in longevity. and I think like the medical sciences and our like what AI is doing for and not even AI like outside of AI what we're starting to learn about how our bodies work and like turning things like drug discovery or vitrification of organs into like freezing of organs so that they can be preserved for organ donation like turning them into engineering problems is just like super interesting.

53:41And I'm really excited for the next five to 10 years. I think we'll learn a lot here.

53:45Aria Finger:Can you leave us with a final thought on what you think is possible to achieve in the next 15 years if everything breaks humanity's way? And what's the first step to get there? I think it's possible to have a digital world that we feel like we can change it like anything in our house or anything in our lives or anything in our kitchen. Like maybe our digital world feels more like our kitchen where we like know how to make food and have the tools for it. We can rearrange our kitchen. We can design it exactly how we want it to. This idea is not unique. It's from Glenn McDonald on our team. I think it's very possible for all of us to feel empowered in the digital world in the way we feel empowered in our kitchens and for that environment to kind of nourish us the way our kitchens can nourish us.

54:38As always with great conversations, Kanjun, I have questions for another 10 hours. So one hour leads to 10, but I look forward to the next step. Me too. Thank you both. Thanks so much. This was really fun. Possible is produced by Pallet Media. It's hosted by Ari Finger and me, Reid Hoffman. Our showrunner is Sean Young. Possible is produced by Tanasi Delos, Katie Sanders, Spencer Strasmoor, Emo Zhu, Amon Suri, Lexi Kevin, Danny Garrison, Trent Barboza, and Tafadzwa Nima Rundwe.

55:09Aria Finger:Special thanks to Surya Yalamanchili, Sayida Sepieva, Ian Alice, Greg Beato, Parth Patil, and Ben Rallis. And a big thanks to James Campsey, Brandi Hagel, Ekta Batija, Eric Pavia, Ben Kasnoka, and the Village Global team.