Unveiling Stanford's AI Self-Reflection Breakthrough

0:00What can 160 years of experience teach you about the future? When it comes to protecting what matters, Pacific Life provides life insurance, retirement income, and employee benefits for people and businesses building a more confident tomorrow. Strategies rooted in strength and backed by experience. Ask a financial professional how Pacific Life can help you today. Pacific Life Insurance Company, Omaha, Nebraska, and in New York. Pacific Life and Annuity, Phoenix, Arizona. researchers have just unveiled a very new interesting way of training ai or more of a feature to include in a trained ai and today on the podcast we're going to be talking about this and this is coming from stanford university their human-centered artificial intelligence department and this comes from a report they recently put out that says ai agents that self-reflect perform better in changing environments and you know originally when started reading this and looking into this, I thought this was some sort of, you know, a wellness like play wellness study.

1:02But actually what they're doing here is they have taught AI essentially to be introspective and to have what they call curious replay. And this is something that they're training into their models. So this is something I think is really interesting. The way that they actually came about doing this is they essentially took a mouse and they took an AI agent. And in they put a mouse inside of a maze with a red ball and they timed how long it would take for the mouse to go and play with the red ball to be curious about it essentially um and to you know just like you mess around with this ball pretty much to be curious see what was going on and at the same time they put an ai agent in a virtual environment i guess with a virtual red ball i have no idea what the stipulations of that were what virtual agent means but in any case that apparently ai had no curiosity and didn't do anything.

1:55I mean, you can kind of imagine this like chat GPT. If you said you're in a room with a red ball in a maze, what do you do? Then chat GPT is like, I don't know, I just like sit around waiting for something to happen. Right. So that is the problem, apparently, according to this study. So what happened was that Kuvar, who is doing this study, he wanted a way to measure and see what the fastest way to get an AI to explore a new object was. So that's what his goal with this study. And he said it wasn't expected that the AI agent didn't seem to notice the red ball in their environment. And he said already we were realizing that even with state-of-the-art algorithms there was gaps in performance.

2:37And this is essentially because the mouse was quick to approach the new object and interact with it and the AI agent seemed oblivious. So they wanted to fix this. Because of this, Kovar Doyle and Lin Kui Zhao, who is a graduate student and Haber all decided to kind of rethink how we train AI models and they explored the possibility of using simple animal behaviors to upgrade AI performance so the solution they eventually landed on was kind of a novel training method that they christianed curious replay as I've said before and curious replay essentially is a technique that incentivizes AI agents to revisit and then to contemplate the most recent peculiar encounters they've had and this is kind of interesting because they didn't want it to just replay like let's say all the conversations or all of the moments it had right like so let's say they take an AI agent they stick it in a 3D environment and mostly it's just staring at a blank wall and then you know a ball comes in they don't want it to have to spend 24 hours replaying random moments of that until the red ball comes in and then you know engage with the red ball or think about it they want it to come up they want it to essentially come to think about peculiar moments unique moments different things that happened a red ball you know that's out of the ordinary in the room and so after they kind of decided to introduce this mechanism not only did the ai agent actually react a lot quicker to the red ball but its performance at a minecraft inspired game called crafter also improved significantly so um not only you know so i think the the reason why they bring that up is because it had been already tested to you know play a specific game or do a specific task and once they trained it to be better at this new kind of thing to have curious replay and then they made it play the game again it actually improved its ability to play on that game so the team is actually going to talk I think about some of the findings from that specific study with crafter at a conference later this year but I think what's really interesting is the fact the researchers are currently using the concept of curiosity in a really groundbreaking way so essentially they're encouraging AI to use it as a learning tool rather than just a decision-making factor.

4:47So I think the idea is to essentially prompt the AI agent to interact with novel objects in its environment and to stimulate learning and encourage exploration. Now, that sounds super fun. I will put one caveat here that there are definitely some downsides to this, in my opinion. Essentially, what we're doing is we're causing the AI to look, you know, to be introspective, to think about itself, to think about its environment, to decide what's interesting, to become curious, to want to learn about things. And I think this is getting, we're getting close to a blurry line between machine algorithm and all of a sudden we're trying to teach this thing to think and decide what to think and what to explore.

5:31And, you know, if we have one of these AI agents that essentially is deciding this red ball is really interesting let me think learn everything i can about this red ball um what else what other topics would it do that with what other topics would it go really deep on what if it all of a sudden you know finds an odd fascination with world wars or with weapons systems or you know there's all sorts of things that it could become really fascinated by or curious about and go really deep into um which i think you'd you'd want to some way to log or monitor or track what the AI is going deep on for obvious reasons.

6:04So I think, you know, the concept makes these more powerful, but with more power comes, you know, more opportunities for the tool to be corrupted. And I just think anytime you started getting these AI models to be autonomous and just deciding what to do and why they do it on their own, you get into kind of a potentially sketchy territory. And, you know, I know a lot of people are like, oh my gosh, you're crazy like you're such an ai alarmist i probably would have said the same thing a number of months ago until i started recently doing a lot of research on the ai model pi which is made by inflection ai and seen a lot of really scary ideologies that that ai model has right like if you've seen my reporting you know that uh inflection ai would appear to um put the life of an animal above the life of a human it espouses uh an ideological principle called deep ecology where everything in the environment is equally important.

6:58And essentially, you know, there's a lot of different, there's a lot of different like philosophical or ethical frameworks. And I think the one that it subscribes to is essentially that any sentient item is essentially has the same value, right? So like a butterfly is sentient because it's alive. So it's the same as a human and all sorts of questions like that. I mean, inflection AI literally told me point blank that just because you could save a human life would not justify you killing a bee. So, you know, some alarming things like that. And I think that, you know, an AI model like that that goes really deep, that gets integrated into healthcare or the military or any other, you know, like, system that interacts with human life or that is, you know, critical to human life, I think could be quite dangerous.

7:40So I think that's something very interesting to think about when we see these really interesting new advancements in AI. So in any case, during this study, Kovar, he highlighted that in their new method, Curious Replay essentially deviates from the standard AI training method called Experience Replay. So instead of randomly, you know, replaying a memory to learn from them, Curious Replay prioritizes replaying the most intriguing experience. And applying Curious Replay to the game crafter resulted in an increase in the state-of-the-art score from 14 to 19. and I think this is just one change that emphasizes really the potential for this simple but very revolutionary approach if we can get if we can get AIs to make incremental improvements from small tweaks like this I think this is has a lot of potential as you kind of start implementing a lot of these small tweaks so I think the method success in a range of tasks really just indicates its potential to make some big strides in AI.

8:43And Haber himself, he kind of foresees the emergence of more adaptive and flexible technologies, such as, you know, like household robotics and personalized learning tools. And so I think really inspired by his kind of success here, Kober aims to continue comparing AI agents and mice on more complex tasks. And he believes that this can actually pave the way for a more profound understanding of animal behavior and also neural processes. So I think, you know, essentially by making kind of this direct link between AI research and animal behavior, Kovar hopes to stimulate new ideas and experiments in the field specifically.

9:22He said, you can imagine that this whole approach might yield hypotheses and new experiments that would never have been thought of before. And I think that's pretty accurate. But as I said, there are pros and cons. there definitely you know this is definitely isn't without um any warning or alarms you know teaching ai to be introspective and to think about everything that's been said to it and decide what is the most interesting and to kind of go deeper and learn more about that there are implications right when the ai starts auto steering itself and becoming autonomous but this is a really interesting space so i'll be very curious to follow in the future sure.