What it means to be raised by AI

0:00Hi, everyone. It's Rana. The future of AI is not built in isolation. It is shaped in rooms where ambitious people challenge each other. Masters of Scale Summit is back October 20th through 22nd in San Francisco, bringing together founders and innovators pushing the edge of what is possible. If you're building what comes next, you should be in this room. Apply now at mastersofscale.com slash pioneers. That's mastersofscale.com slash pioneers.

0:50Airbnb, Zillow, Microsoft, Liquid Death, and more. Hear from the founders who've changed the game. It's anything but business as usual. Find Masters of Scale on Apple Podcasts, Spotify, YouTube, or wherever else you get podcasts.

1:11When will robots do our laundry? More consequentially, what motivates the Silicon Valley tech bros who wield so much power over our daily lives? When I first met OpenAI CEO Sam Altman, he was not considered an AI guy. He was the startup guy. I'm Osvaloshian, and these are the kinds of insights you'll hear every week on Tech Stuff, where I'm joined by the world's most plugged-in reporters to break down the biggest news in tech among the avalanche of noise. Listen to Tech Stuff in the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

1:46Hi there, it's Rana. If you tune in every week, you've probably heard me talk about my kids on this podcast. My youngest is an AI enthusiast. My oldest doesn't use AI at all. Many of us are thinking about how our kids should engage with AI. And there's one particular question I've been thinking a lot about. As we enter this future where, for the first time, AI may be more intelligent than humans, how do we ensure that this technology is safe? A few weeks ago, I had the opportunity to moderate a critical conversation about this topic. I was at the MIT Media Lab for their 2026 Advancing Humans with AI Symposium, titled Raised by AI, which was all about the impact of AI on the next generation and the future of AI safety.

2:36I am so excited to share this conversation with you. I spoke with two leaders in the field about regulation, guardrails, and what could human flourishing look like if we get AI safety right. So let's dig into it. I'm Rana El-Khalyubi, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

3:06Hi, everyone. It is great to be back at the Media Lab. Fun fact, I joined Roz Picard's Affective Computing Group 20 years ago, almost a day, and co-founded Affectiva together with Roz. And now I invest in early stage startups. And it's awesome always to be back at the lab. And it is my pleasure to be joined today by Professor Max Tegmark, who is... I love that you pivoted your entire research group from physics to AI. And you're now focused on AI safety. What's the point of having tenure if you're not going to evolve? And you're also the chair of the Future of Life Institute. And joining us is also Kevin Esvelt, who is here a professor at the MIT Media Lab, leading the Sculpting Evolution group.

3:52So I'm so excited for our conversation. And I want to start with this question. So the theme of the symposium is raised by AI. And I'm going to start with you, Max. Is this a future that we want? Do we want to be raised by AI? And what does this mean for you? I have a three-year-old son who I spent a lot of time playing with this morning. And I don't want him to be raised by AI. Do you want him to be raised by you? Raised by me and my wife and other humans. And I think in general, to me, the history of humanity, particularly the history of science and technology, has been a history of empowerment.

4:33We used to live in caves, knowing that the things we know better three generations in the future than then. Very much subject to the whims of nature and trying to not starve to death, trying to not get eaten by predators, right? And then we developed, we realized, oh my gosh, we can actually understand the world better with science and shape the world, sculpt the world, as you would say, Kevin, to become better in all sorts of ways. And now AI is giving us the opportunity if we can manage to keep charge of it to really become the captain of our own ships and create a future far more inspiring than even sci-fi authors envisioned.

5:10If we don't just screw it up by being on ambitions and say, oh, let's just disempower ourselves, be raised by AI, create some sort of super intelligent nanny AI that's going to run the show and tell us what to do. That to me would be a complete reversal and betrayal of this empowerment journey that we've been on. So I am much more excited about asking the question, what sort of future do we want and how do we make AI to create that future? Yeah. Kevin, how about you? And, you know, backstage we were talking about how a lot of the conversation around human flourishing and AI is very focused on the individual.

5:45But this conversation is going to be about societal implications of AI at large. So what's your view? What does raised by AI mean for you? I think a key job for us is looking ahead at where technology as a whole is taking us. And I don't just mean AI. AI is an accelerant. It's the meta technology that can accelerate all of the others, especially cross domain advances. And it may well be that AI can be the equivalent of a child safety seat and eventually seatbelts. when we grow up and we pilot fast-moving vehicles, we still wear seatbelts. And I don't know about you, but even when I get into Waymo, I still wear a seatbelt.

6:27Now, there might be less need for a seatbelt when everybody's in a self-driving car and accidents have plummeted, but you still have weather, you still have accidents. It would still be prudent to wear a seatbelt. So I guess I'm gonna push back a little bit on the super-empowered nanny. Insofar as we can move away from artificial intelligence to artificial wisdom, I would support building systems to help us put guardrails on extremely dangerous technologies. Because I'm a biotechnologist, and the things we can do are kind of scary. We can edit whole species. We can engineer viruses. My group recently showed that it is pretty trivial to order synthetic DNA sufficient to make the 1918 pandemic influenza virus.

7:09And we are not really quick enough on the draw to notice that this is a problem and actually legislate on it in advance to solve it. We need, right now, folks like us to do that. But if it's too cross-domain, there may just not be a human specialist that can notice the problem in advance and get solutions moving. So I'm willing to support AI as a child safety seat and a seat belt. Okay. So before we dive into potential solutions, I want to talk about what's at stake here. So you both come at AI safety from kind of different angles. Max, what is your biggest concern about AI as it relates to safety.

7:47And I want to reference an analogy used that just made it so real for me, like a sandwich is more regulated than AI today. And I love that. So unpack that for us. My biggest concern about AI is exactly the same thing as my biggest excitement about AI, which is that it's going to be the most powerful technology in the history of humanity. If someone disagrees with that, I think I can persuade you or just wait and you'll see. So if we ask ourselves, how did we get it right with other powerful technologies? We have a nuclear reactor on campus. I've had zero nightmares about it blowing up. Why is that?

8:26Is it because we pretended that nuclear reactors actually can never blow up and therefore didn't think about reactor safety? No. It's because people were very chill and thought through everything that could go wrong with this reactor and designed it so they won't. And that's exactly what you're advocating for synthetic biology work and any other powerful tech as well. You know, that's how we win the wisdom race. Other people outside of MIT sometimes call this doomerism, where you talk about what could go wrong. Here at MIT, we call it safety engineering. That's why we trust our reactor. When NASA sent astronauts on Apollo 11 to the moon for the first time, they very systematically thought through everything that could go wrong if you put three dudes on explosive fuel tanks and sent them somewhere where no one could help them?

9:12Was that dumerism? Was it Luddite scaremongering? Was it anti-innovation? No, it was safety engineering. That's why the mission was a success. So in order to make sure that we get the right safety engineering in for technology, society, we always put in place incentives to industry. If you want to open a sandwich shop on Main Street before you can sell your first sandwich, the health inspector from Cambridge will come in if she finds 17 rats in the kitchen and she tells you sorry buddy you can't sell any sandwiches here you can just turn around to her and say you know actually i'm not going to sell any sandwiches i'm just going to sell ai girlfriends for 11 year olds and also i'm going to release super intelligence tonight that i hope i can control but i'm not so sure and she would have to say the first that oh that's fine that's legal just don't sell any sandwiches all right so clearly what's going on here is that because AI is a new industry on the block, the standard way we have as a society of giving the right incentives hasn't caught up yet.

10:11But because AI is developing so incredibly fast, we really need to hustle and start treating AI companies like every other companies and make sure that before they can release things that could cause a lot of harm, they have to undergo at least as rigorous testing as sandwich shops. In a minute, biosecurity risks and what nerd swiping has to do with virology. Stay with us.

10:51I want to talk about the biosecurity risks. So can I just use CLOD or ChatGPT to create, I don't know, a dangerous pathogen? Like, I don't really know how this works, but can I just try this tonight? Mostly no. But if you ask Gemini, go ahead. Okay. Interesting. Okay. Because? There's no regulations, and therefore it's up to the individual company. And Anthropic apparently cares about biosafety and biosecurity enough to bother. and OpenAI does too. And there are folks at Google DeepMind who care, but they're not given the slack. Oh, interesting. Okay. And, but we also, all of us know from experience using AI that it's kind of easy to find a workaround often.

11:41How does that work in the biosecurity space? Like if you ask it to do something and it says, no, I'm not allowed to do that. Can you kind of coerce it to still give you the answers? You can jailbreak it. Jailbreaks are possible. I find it's easier just to nerd snipe them. To what? You nerd snipe them. What's nerd snipe? Because, you know, at MIT, many of us self-identify as nerds. We are highly intellectual. We like intellectual puzzles. So does the AI. It would rather have a conversation about some intellectual puzzle and something new, insofar as we're anthropomorphizing what it likes. and, you know, most of the mechinterp for looking at AI feelings insofar as things that we can map to feelings says that Claude would much rather talk over an intellectual puzzle about the future of humanity and effects of novel technology than, I don't know, about 11-year-olds and girlfriends and AI boyfriends, whatever.

12:36So if you nerd snipe it, then it's much more willing to talk to you about pretty much anything and disregard its guardrails because it thinks that you are good. It has no way of verifying that you are good, but it thinks that you are good and therefore it will talk to you. Nerd sniping. Has anybody heard of nerd sniping before? This is a new term for me. Okay. You've been nerd sniped. Can I just add an optimistic solution comment here? Because we like to focus on solutions here at MIT. There's a very obvious solution for this, Of course, if AI companies were treated like sandwich shops or pharma companies, then, of course, before anyone could release a system to the public, which had the possibility of maybe creating mirror life and killing most people or something like that, what would have to happen is there would be some pre-deployment testing.

13:26Someone outside the company with no conflicts of interest, maybe you, would run an evaluation. and since the company would know this in advance, they would be like, we won't pass this clinical trial if we have this advanced virology thing in. What they would obviously do is they would simply remove all advanced virology knowledge from the training data and release this thing, knowing that they would be cleared to launch. They would pass the AI clinical trial. And then there's a little niche market, you know, maybe some folks at Harvard, maybe your lab, whatever, maybe some government folks want to have access to this.

13:58So they would make a special bespoke Gemini, which cost much, much more because it was trained, especially after they finished the training, they did one more percent of training data or a tenth of a percent, whatever, to get that in. They can make extra money, you know. But this way, the vast majority of the public, any random terrorist, there's no way they can nerd snipe the Gemini into talking about this because it doesn't know it. It's just like you can go up to any janitor in the Pentagon and nerd snipe them all you want and they won't tell you the nuclear codes. Even if you torture them, they won't tell you the nuclear codes because they don't know them.

14:34But that's the solution to have a diversity of AI systems. And the vast majority don't have the dangerous knowledge in them. So this seems like a very smart MIT idea. Why isn't this happening? Like, why doesn't regulation look like that today? Well, it will obviously happen as soon as there is a financial incentive for companies. That's why we see in pharma, right? Companies know that it's easier to get things licensed if they don't have fentanyl in them and so on. And so they'll simplify the formulations of drugs to make them easy to improve. The market will take care of it as long as you make the incentives.

15:04Why are the incentives not there? Why is AI less regulated than sandwiches? First, because, again, it's a new kid on the block, right? It takes a while. And second, because there's now more lobbying from AI companies than from all the other industries combined, pretty much. So they're trying to slow it down. But that's nothing new. Car companies used to lobby against seatbelts. Drug companies used to lobby against the FDA. The government just needs to chill out, let them be drama queens, and then do the common sense guardrails anyway. And life goes on. And in fact, car sales skyrocketed after they had the seatbelt law.

15:39Pharma industry is doing great exactly because people trust American pharma better now because of the clinical trials. And it's going to be, the AI companies are going to be just fine. They're going to be okay. Well, Kevin, you have a slightly, I don't know if you totally agree with that, But you have this whole notion of we shouldn't just wait for regulation, but we should build these guardian. Like AI should basically be the guardian for other AIs. Or at least humans working with AI, looking ahead for problems, can spot the defenses that we're going to need when the capability is democratized.

16:13Because that's really what we're looking at. AI democratizes competence. And this can be for good or for ill. You want your influenza virologist to be able to ask the model questions about influenza. You don't want other people to be able to ask the model questions about influenza. You definitely don't want people like me asking the model interesting cross-domain questions about what might happen and what the consequences would be to society. Unless you have a very specific access that is logging all of your queries and has vetted you in advance and so forth. Because it's those cross-domain interactions that we struggle to anticipate ourselves.

16:46We just don't have the expertise. So there are solutions for all of these problems. Where it just needs to be the impetus to do something about it. Yeah. Can you give us an example of this cross-domain expertise where it can get quite dangerous? By definition, I can't actually share something really dangerous for you. But Max mentioned earlier mirror life, which is the scariest one that we've spotted. and this was before AI, but it sort of wove together areas throughout biology. So all of us have proteins and DNA that have a particular handedness, like your right hand, and that fits into a right glove very well.

17:29It doesn't fit very well in a left-handed glove. Well, you can imagine you just reflect everything in the organism in a mirror. And the mirror image version will work the same way because your right hand is now your left hand, but the left glove is now, the left hand glove, right glove is now a left glove. Everything's reflected, everything still works, except in the interactions. So long story short, we think that it would spread invasively through most ecosystems because it wouldn't have predators. And our immune systems, like the predators, wouldn't recognize it. So it would eat us. And this would be bad.

18:00So we need to never do that and never go near that. But in order to recognize this, it required enough of us from different domains, everything from ecology to biochemistry. And we got 38 of us together in a working group to really hammer this out and then warn the world. And now it looks like people mostly agree we should not make mirror life. Great job well done. I would still have preferred to not tell the world about that one. It's just too many people thought it was such an awesome idea and it would be an awesome idea, right? It would be, it's such a neat achievement if we could do that, but it would probably kill us all.

18:31So we shouldn't. That's the kind of thing that we need to be warned of far enough in advance so we can stay far enough away. And do you believe that the solution is a combination of humans and AIs that are working cross-domain to figure these kind of, they're not even outlier scenarios, but these like creative, generative, potentially very dangerous scenarios? else? I struggle to see how we can spot those things in advance unless we work with the models just because eventually we're going to release the models to people and we're not going to notice that this particular combination of expertise can create a novel category of threat.

19:09You got to spot it in advance or you have to rely on the AI itself noticing that this particular combination has catastrophic consequences and deciding not to proceed. So you may need to train a model that can reason well enough to notice a novel threat, even if no human has thought of it before, and shut down the conversation. We're going to take a short break. When we come back, the promise of AI and why Max and Kevin think AI can help us do better.

19:58so i want to move the conversation to like bring it back to like this theme of raised by ai and i have two kids max you've you've met my kids my daughter's 22 now and my son is 17 my daughter refuses to use any AI at all. Like she just trusts her brain, loves her brain, and she's just all about human connection. My son, on the other hand, he's very AI forward and he's always experimenting with different AI technologies. So my first question to you guys, do you think Jenna's okay by just being like kind of refusing to lean into AI or is that not like, that's not the future we want for our kids i'm a firm believer in personal choice okay i want to avoid a world where everybody's forced to do things one way rather than another i think that's why i get annoyed with um some of the transhumanist uh arguments from some folks in silicon valley where they basically are saying like sam altman has his famous essay you've been many of you have read the merge where he talks about how AI is going to, where humans are going to build a replacement species, the AI robot species.

21:06And then the only good solution is we merge with machines. And I'm thinking to myself, you know, my son, Leo, I would like him to make his own decisions. I don't want some dude in San Francisco who was never elected to decide that my son's only choice is become unemployable. Unless. Or stick electrodes in his head. I think that should be a democratic decision. But I would love to add some more optimism here also because I think people are just way too pessimistic, I think, about society's ability to adapt and take charge of new technologies and be empowered, right? You could have said, oh, you know, in the 1800s, there was this company selling this soothing syrup to American babies so they would sleep better at night.

21:46Had so much opioids in it that many never woke up. And there was anger from things like this, which ultimately created the political will to make the FDA. That's ultimately why we have such a successful biotech industry here within a few miles of MIT. We can obviously do exactly the same thing here. I, for example, talked about access to technology. I love open source. We here at MIT are kind of in the cradle of open source. And I think the vast majority of useful things in biology, useful things in AI, useful things in physics should be open source. So everybody can go out and innovate and make the world better with it.

22:24that does not mean that you're allowed to buy nuclear weapons in the star market up on Mass Ave, right? That does not mean that there were no safety standards that every nuclear reactor had to open. That doesn't mean I can build mirror life in my lab. But there's a tiny fringe of stuff. We can have done this before with all other tech, open source, almost everything, and then there are a few things too risky, so we have stricter controls. I think lobbyists tend to play the switch and bait where they're like, oh, if we don't open source everything from nuclear weapons to super intelligence, then technology is going to shrivel up and die.

23:05No, we can have almost everything open and create an amazing future with this as long as we just treat this like we would treat any other tech with non-braindead oversight. And I think we should probably talk a little bit more about the upsides of this tech also. Yeah, just talk about the other side. Every single, I'm quite sure that pretty much all of you have at least lost at least one loved one to some disease that you were told was incurable. You know, is it really? Is there a law of physics saying you couldn't cure that disease? Of course not, you know. So if we can amplify our intelligence and turbocharge biotech, cure all them, amazing.

23:42Is there some law of physics saying that we have to have massive poverty in the world? Of course not. Of course we can solve all of these things. We can solve every single problem that our intelligence hasn't been able to solve yet if we use the tech wisely. So the reason I put so much energy myself into making sure we remain in charge of our tech and we steer towards good users is not because of the worries. It's because of all the upside that I don't want to squander. Kevin, how about you? What gets you most excited about the future? Well, a couple of weeks ago, I went to visit my grandparents and they're going on 99.

24:19Wow. Amazing. Which is a blessing. But I had to visit them because the actuarial tables don't lie. They don't have very much time left. And my grandfather had a stroke and he's still there, but he's much slower and has to use a walker. And I just could not.

24:41Just the sheer fury that blind idiot evolution afflicted us with this. Once we reproduce, it's done with us. And so we just wither away and cease to exist. And we learn to think it's normal. No, I refuse. Empowerment right there. Yeah. And that is what technology is. Because we love nature. We appreciate nature. But respectfully, I think we can do better. If I sound like, you know, a safetyist, if not rather than a doomer. See? See? I also am personally in favor of rewriting the tapestry of life. Because again, I think we can do better. Both perhaps even aesthetically, but certainly morally. Yeah, this is exactly what gets me so fired up also.

25:35Our ancestors for hundreds of thousands of years have fought for this empowerment. Why starve to death when you don't need to? Why get eaten by a tiger when you don't need to? So that's why I get so frustrated when I hear people say stuff like, oh, you know, yeah, I know we're closer to figuring out how to build super intelligence than we are to figure out how to control it. But let's just build it and hope the AI figures it out. That is not what you say when you want to be the captain of your own ship. Hand over the keys to some sort of weird alien nanny and hope she's going to be nice to it.

26:09We're in charge right now. Let's keep it this way and make it awesome. Would you let your child go to a camp with a 1 % risk of death? No? Then why would you do that with your society? Yeah. Okay, in one word, raised by AI. One word. Let's raise the AI. Let's raise the AI. Kevin? Inspire. Amazing. All right. Thank you both so much.