Best of: The future of AI coaching

0:00Hey, everyone. It's your host, Russ Altman, from the Future of Everything. I hope you're enjoying the holiday season with family, friends, and loved ones. We'll be releasing new episodes again in the new year. In the meantime, we're running a fascinating episode on the future of AI coaching. You know, the past few years have seen an incredible boom in AI. And one of my colleagues, James Landay, a professor of computer science, thinks that when it comes to AI in education, things are just getting started. He's particularly excited about the potential of AI to serve as a coach or tutor. I hope you'll take another listen to this conversation and come away with some optimism for the potential AI has to help make us smarter and healthier.

0:41Before we get started, another reminder to rate and review the show. Give us a five. It'll help spread the news.

0:54Large language models like ChatGPT and many others have started to sprout all over and people are using them for productivity. We're writing letters, we're editing our text, we're answering questions. There's a lot of stuff people are doing. But what we don't always think about is can we use this AI as part of a coach or a tutor to help us get from where we are to where we want to be? Well, James Landay is a professor of computer science at Stanford University, and he's a co-director at the Stanford Institute for Human-Centered AI. He will tell us that he's built a prototype health fitness coach that uses AI.

1:29He's also created a tutoring system for elementary school students where they learn about the planets, about global warming, about fires, all through a mixture of large language models getting outside and interacting. James, you were on the show in 2019, May of 2019. It was great. But now it seems like you're working a lot on using AI for teaching, for coaching. How has your work evolved in the last few years and what are you really excited about? Well, a lot of stuff has happened in the world since 2019, Russ, as you may recall. But most of my projects I see as more of 10 or 20 year efforts. They're big problems that if you solve them can have a big impact on the world.

2:13So problems like how do we improve the education system or how do we improve people's health to me are not things we solve in three years. They're things we work on for a long time. And then along the way, we come up with new ideas that might improve it. So, for example, in health, when I spoke to you before, we talked about work on how you can give people better awareness of what they're doing towards their fitness goals by having what we call ambient displays. So, for example, story or images on the lock screen of their phone. What we've been working on since then is how can we give people more of the sense of having a personal coach, even if they can't afford to have a personal coach?

2:51So, you know, just like personal tutoring, personal coaching is highly effective for individuals who are trying to change their health or fitness. But many of us can't afford to have a personal coach because it's quite expensive and there's not enough coaches out there to handle all of us. So combined with what's going on in AI, LLMs, for example, we're able to use that. Large language models. Large language models are, as we like to say here at Stanford, foundation models, because we're going to build other applications on top of those foundations. We can use that type of model to get at people's real needs for coaching.

3:27So you can think of all these fitness apps that you might have to have out there, whether it's Apple's fitness app or Android from fitness app from Google, Google Fit. They tend to be very quantitative in form. So you maybe you have an app where you can put some goals in quantitatively, but it's the qualitative issues that often trip people up. You know, the app's telling me, hey, you know, go running every morning. But, hey, I need to drop my kid off at preschool and I can't fit that in to my schedule. You know, their user interfaces. I'm sorry to interrupt. The user interfaces are terrible. I have a Garmin watch and sometimes I do a five-mile run and at the end it says unproductive workout.

4:10That's all it says. And I just want to frisbee it into a brick wall. So you've already gotten me, but please continue. So what's nice about these large language models is they're really good at processing text, right? And a lot of the ways we might express some of these qualitative issues of what's worked for us in the last years when we've done our fitness activities or where we might have barriers. These models are actually good at being able to process that kind of information if we have a conversation with it like you might have with a coach and then help you develop a plan for your fitness that accounts for those issues.

4:52So we've built an application called GPT Coach where we used a large language model. And, you know, it's much harder said than done to actually get these things to do what you want. But essentially we can create a bunch of what we call agents that use different pieces of the model. And this way we can check what kind of things we're telling you. And we can keep the model on track because we use a technique called motivational interviewing, which is popularized by researchers here in the School of Medicine at Stanford for coaching. And we can have this coach pretty much use that style of interviewing to find out what's worked for you in the past, what your barriers are, and then together come up with a fitness plan.

5:36And we tested this in the lab with people and they were blown away by how good it was. And in fact, when I first tested it, I said, this could be a product right now. That's how well I thought it worked. Okay. So like so many, so much exciting things there. So first of all, I take it, this is not out of the box chat GPT. Not out of the box. We have to, we have to essentially break it up into a bunch of different agents who have different tasks that they're trying to do in terms of this interviewing and also keep it on track and check where we are in the process, what kind of information we've gotten.

6:09And we also get information out of your, you know, you have a Garmin watch. A lot of people have an Apple watch or just an iPhone. We're able to actually take three months of their prior data and use that to also drive the conversation, understanding what they've been doing. So another really intriguing thing you made a quick reference to is that you've been kind of working, it sounds like, with the School of Medicine experts at fitness or health. Tell me about that, because now you're trying to get this language, large language model to do things at a very kind of virtuoso level. So what is the role of the health professionals and how do you get their knowledge into this model?

6:48Yeah, I mean, most of my projects tend to be interdisciplinary working with folks outside of computer science. So sometimes even in computer science in a different subfield. So, you know, as you mentioned, the last time I was on the show, we talked about buildings and smart buildings. And I work with people in civil engineering and education and sustainability on that as well as health. One of the other projects I've had going for a while that I talked about last time was these ambient displays for health and fitness. Now, as part of that, we collaborate quite closely with people in public health in the School of Medicine.

7:21And, you know, that team has a lot of experience on training coaches and what are the best methods for coaches. So in this case, they actually created a manual. And so we were able to take that manual and understand how it's structured and information and actually feed that in to the large language model as well to help our system kind of do it in that style. And we also consult with them on this. They haven't been as much of a close collaborator on this project, but we've consulted back and forth with them, and they may become closer as we get to the next stage. Because all we've done so far is be able to replicate that first 30 minutes or 60-minute meeting you might have with a coach.

8:05They're meeting them for the first time, and they're trying to understand what your goals are and what your barriers and what's worked and come up with a plan. But the next step of the project is, okay, now what does it mean for me to have a coach on my phone over the next six months as I'm, you know, partaking in my exercise program? And so that interface is a harder design because this first one really is a conversational interface. It's like we're having a meeting. But now I'm going to have this application running on my phone. And we don't think language is the interface that you always want to use.

8:38where you want to have graphical elements that you would see. And there we learned from some of that prior research I've done with a ambient display that's showing some kind of visual story as you go. But now you might want to use language, for example, when you might want to say, it might remind you, hey, Russ, you committed that you were going to go running this morning. And you might want to say, oh, but I have a big meeting with James Landay and he's really important. I can't miss it. Really important. Right. And maybe, or you're sick. And so, you might use that and then the coach might be able to kind of adjust your plan.

9:12So changing your plan or having exceptions or being able to, you know, highlight when you're not feeling up to it and maybe it's going to, you know, work on some other way to motivate you or give you an easier goal that might help you get there. So we'll have a kind of combination interface now with traditional graphical interface and ambient display, but also with the part where you might then still converse when you want to. So that's what we're designing right now. And the plan then is to do a short-term study of that, like three weeks just to get the bugs out. But then the true goal in all this kind of work is, can we run a study that might run over three, four, six months and show that people are doing better at hitting their goals or changing their behavior than they would otherwise without control?

10:00Yeah. So this sounds very exciting. And by the way, sign me up for that. So, but here's a question. When you have a coach, I know some people who've had coaches and I know there's a process of interviewing them because you're always trying to find the coach who's like perspective on life and whose perspective on health and wellness kind of matches yours. You know, some people like the drill sergeant, you know, we've all seen, we've all seen these ridiculous videos on YouTube where they're like, go, you can do it. You know, work harder, work harder, work until you drop. And then there's other people who are like much more like, Let's do whatever you want to do.

10:34Do you imagine that you're going to have to, or do you think the LLMs already can modulate their tone based on the preferences of your users? Yeah, that's a really great question. So in fact, as we're designing the visual version of this, we see a kind of avatar-like character that is representing the coach. And so we are doing an online study to just test these different designs for those coaches and also the personalities. And that's what we want to discover, whether different people have a different preference for types. And we're trying to design it in a way that we could probably flip in different personalities and such.

11:17Now, my caution to my grad students is we probably don't want to do that in the first version of this because then that just makes the study results harder to interpret. Right, good point. Was it because you had the different coach? But I think for a product, you're probably going to want both different visual themes. Like, you know, we have a space theme and a beach theme, you know, different people. You know, maybe you have a theme. Dude, we have to exercise, dude. Right. So there's different themes in the visuals, but there might be different themes on some people want a coach that's really tough and pushes them.

11:52I'm like, I had a coach once who could make me cry, literally, because he could push me beyond where I could push myself. And some people are okay with that, but other people would quit if they had that. So understanding that is something that we're looking at, but we probably won't push it into the first version because, again, it just adds too many confounds for a study. But I think for a further long-term thing, yes, you'll probably have different personalities. And yes, in terms of the LLM, the LLM, you can do that. but we also see the need for it in the visuals as well as the personality of the avatar.

12:26So we're looking at that in all aspects. We probably won't see that in the first version of this. And I'm very aware that I'm giving you all of these features and that you need to walk before you can run, but that won't stop me. My next question is, what about group activity? So a lot of people get a lot of their sustenance and kind of support in there, especially in health, like they're part of a running group or they're part of a rowing or a workout group. Do you have it? Maybe not today, but is in your vision some sort of social support from other humans or or whatever? It's not in the initial version.

12:58I've worked in this area for a long time and I have to tell you, people think, oh, yeah, social, you got to add social. But we did some of the early studies on this and we found that social can actually also backfire and make it worse. So I remember this when I was in Seattle at the University of Washington and running a research lab for an Intel. we've done one of our first studies like this and the people were in a group and literally one woman drove by one of the women who was walking in seattle up a hill and the lady said why are you walking up the hill you can get credit just for walking the flat so you know the group even was discouraging her from doing exercise so it is something you have to be very careful about how you design it because it can also cause the opposite result so we're not looking at group right now, but it is something obviously for, again, if it was a real product, you might think about where might you integrate that.

13:48And my next question is a little bit about a definition. Even in this conversation and definitely in your writings, you've talked about this idea of ambient awareness. And I just want, could you define that for me and let me know why it's an important thing, because that's not the kind of thing I usually attribute to a computer or even to an LLM. Like usually my LLM is in a little white box on my computer. It has no idea if I'm in a rainstorm or at a spa. So what is ambient awareness to you? So when I think about what is an ambient awareness or ambient display, it really comes out of this idea that a lot of times we're attending to something else in the real world.

14:27We're not like staring at our phone, doing something on it, right? And how do we take advantage of those glances at our phone to actually communicate information to us in the background, even if it's not the primary task. So a lot of these fitness apps and things like that, for you to know what's going on there, you need to turn on the app and look at it and go, oh yeah, I've walked this much today. I've run this much. But only if you go check it, are you going to be aware of what's going on. And so our research is based on this idea that people who are really good at sticking to their goals are people who are just much more aware of what they've done.

15:09They're tracking it more. Either they're actively checking or they're just aware. Oh, I know I parked my car over there. I've walked this far. But those of us who are less aware have a harder time, you know, understanding, are we doing well today? Do we need to do more? And so the idea of ambient awareness is, can we have a display that you might just glance at and see that gives you a sense of how you're doing? So we take advantage the lock screen of the phone or the wallpaper when you unlock as a way of just seeing some kind of display that gives you a sense. So the one example I think you see out there today is on an Apple Watch, you might see those rings if you use that display.

15:48I think, you know, they got that probably from our research because we've been working on this for years, but it's a little too subtle. Most people are like not even aware, well, what do those rings mean? And is that good or bad? It's kind of small on most people's watches. And so for us, it would have to be something you really see. And by just a glance, you have a sense of, hey, I'm doing pretty well today or I'm doing well this week or no, I should bring my gym bag because I really need to do more. And so we want to take advantage of those glances, even if you're not running the app explicitly to get a sense of how well you're doing.

16:20No, I love that idea because in addition to my phone and my watch, I have like literally right now I have two monitors in front of me. There's a lot of real estate not being used. a little corner telling me, you know, you're not acting healthy or whatever. That could be a really important thing. This is the Future of Everything with Russ Altman. More with James Landay next.

16:50Welcome back to the Future of Everything. I'm Russ Altman, and I'm speaking with James Landay from Stanford University. In the last segment, we talked about James's work building health fitness coaches, and they seem to work and they look very promising. But he's taken some of the same ideas now and is using it on elementary school children to help them have a more rich learning experience. It involves getting them outside, interacting with the world, and using large language models, which he will tell us gets them to write more. Maybe large language models are not the end of writing as we know it.

17:24But I know, James, that you're also looking at education and specifically elementary school education, which is arguably the most critical in the same way that pediatric medicine, that's the future. So tell me what's happening in AI for elementary students. Yeah, so again, one of my long-term projects is this project called the Smart Primer. And the whole idea there is, can we use narrative stories with activities embedded in the narrative as a way to get kids engaged in their education? And the high level motivation of this is that, you know, many of us do well in the school system. It's kind of a factory school system.

18:05And anyone who's here at Stanford probably did well in that system. But there's probably a lot of talent out there that just never fit in that and didn't get really motivated and excited by school. And then you know, in some ways they don't meet their potential in society and probably end up in careers that are less satisfying and less economically productive. And so one of my goals was, is there a way to motivate kids outside of traditional school to learn? And maybe that will carry over into their other educational outcomes over their life. So the Smart Primer is a series of projects trying to explore the use of narrative in a personalized tutor.

18:45So we've built a variety of these over time. But now that the AI part of this is starting to work better by having these foundation models and LLMs, we can do more. So a couple of summers ago, we built this application we call Moon Story. It runs on a smartphone. And as part of this, kids learn about the environment. They learn about the planets and the sun and the scale differences between the planets and each other and the sun. And they learn about the scale difference of the distances between the planets and the sun. So in fact, to use this, they do mobile AR on the phone. And we had these kids doing - You said something, mobile AR, just define what that is?

19:33Mobile augmented reality. So not having to wear some goggles, but instead on your phone, we can see through the camera and see objects in the physical world, but we can overlay data on top of it. So for example, we had kids come over here to Stanford. We have on our science and engineering quad, these huge, like I call them big marbles. It's a big artistic installation. There's something like 10 or 12 of these huge marbles. I love those things. I love them. They look like planets. They really do look like planets. So I had this idea of, oh, could we map those to the planets and the distance between a subset of them to kind of scale down distance the planets.

20:14And what do you know, we were able to put the sun at one end of that quad and go through the inner planets pretty accurately of being the right distance between them. And so the kids see the sun. And then as they go to these planets, they actually can see the scale of the real planets relative to the sun and the distances they walk, they learn are kind of the relative distances. And then they get through all the inner planets and then we get them go from Mars to Jupiter. So Jupiter is the first outer planet. They have to walk all the way from the science and engineering quad to Stanford's Memorial Church, which is maybe a half mile or third mile away.

20:54And that shows that's how far from those inner planets that you were just going like, you know, you know, 20 meters to now, you know, you're going something like 400 or 500 meters to get to that one. And if you want to go to Saturn, by the way, it's off past that, you know, other end of the campus. So, you know, that whole story about the planets was also put into a story based on an ancient Chinese story about the moon goddess and her husband, who was an archer who had to shoot down these seven orbs that were making the earth too hot. And so we have this story that they learn about global warming, as well as the planets, and it's all embedded in the story.

21:35And then as part of this, the key thing that LLM let us do is they had to write about what they learned and what they were thinking about changing in their everyday life is about sustainability. The LLM allowed us to tell what they had written. And sometimes you get a kid who just writes almost nothing. And we were able to use the LLM to encourage those kids who didn't write much to actually write more. And that was one of the big outcomes is you were able to actually have kids write more. And the kids thought the writing was one of their favorite parts of this experience because they got feedback from the moon goddess on what they wrote.

22:13So you had the characters from the story embodied within the LLM. Right. And it was all personalized to what you wrote. And that was the only place we used the LLM in this. And we got learning gains that we test by doing a pre-test and a post-test and a test a few weeks later. But the big surprise to me, you know, there are a lot of other results, but the big one was, hey, we got kids to write more. And that's really hard. Especially since people are saying LLMs are going to be the downfall of writing by humans, and you have a counterexample there. Just to understand this a little bit more, is it that they're interacting with the LLM and it's prompting them like, hey, what did you do today?

22:53What did you think about that planet thing? Like how does the LLM get them to write more? So in this case, the LLM asked them some specific questions about what things they might change in their lives with respect to sustainability and about what they had learned. And then if they didn't really write much, it kind of encouraged them to write more, ask follow-up questions. And even if they had written something, it could then respond relative to what they had written and also encourage other response. So it's like you really had a person who read what you wrote and gave you feedback that was really relevant to what you wrote rather than a canned response that a computer program might do in the past.

23:32The other thing that you didn't stress, but that I have to note, is that this was presumably outside. The kids were moving. The kids were not in a classroom. And this really does, you kind of delivered on the, your introduction was that some kids don't, the classroom environment is not where they excel. And you could imagine that by putting them outside, putting them in space, having them move, that this created a whole different set of skills and interests. And it just seems like you delivered on that promise. Yeah. So one big idea of this, and this was also, you know, when I got the original idea comes from really the science fiction author, Neil Stevenson's 1995 novel, The Diamond Age.

24:12Like I give Neil Stevenson full credit because this is where this idea came from. And I've been thinking about it since 1995, though not really seriously till 2010 when the iPad came out. And I thought, oh, that's the device he was describing. but the idea was kids today they're sitting inside on a screen they're not outside playing you know stickball for you new yorkers baseball for us skateboarding things in the outside like we did when we were growing up and parents lament that their kids are just inside so as part of this i didn't want to just create another thing that was just going to force you to be inside obviously there's parts of it that you might do inside like reading a book but there's other parts where we wanted you to go out and do an activity in the real world, whether it's in your backyard, on your block, or maybe with your parents down on a trail.

24:59So we've done a previous one, actually, where you look at eucalyptus trees, and you take a picture of one, you smell the leaf, and you actually learn about the fires with eucalyptus in the Oakland Hills, and you learn about kind of the controversy. Are they not native? Should we get rid of them or not? And you have to kind of debate. So that was part of that story. But since then, we've doubled down on what we could do with the LLM. So this summer, we built a new system. Now in this system, which we call ACORN, again, it has an environmental theme. Kids learn about trees, again, that are local here to the Bay Area because that's what we're using.