Best of: The future of computer-aided education

0:00Chris Piech:Hey everyone, it's your host Russ Altman from the Future of Everything. You know, commencement season is here and as students are closing one chapter and opening up another, it's a nice moment to ask, what did learning look like for these students and what will it look like in the future? With these questions in mind, we're rerunning a conversation I had with computer scientist Chris Piech, who studies computer-aided education. Chris studies how computers can and will help students learn. His message is not that teachers are obsolete. Far from it. His message is that AI will combine with teachers to create a better learning environment.

0:35Chris Piech:So whether you're a new grad, a lifelong learner, or an educator wondering what's coming next, this one is well worth another listen. Before we get started, please remember to follow the show to ensure that you get alerted to all of our new episodes and never miss an episode on the future of anything.

0:58Chris Piech:Computers are everywhere in our lives, but one of the places we don't think about them as much is in the schoolroom. We have teachers. We look up to our teachers. They teach us stuff. Then they give us grades. That's how we've always done it. Well, guess what? Things are changing. We're starting to see sophisticated computer programs that actually make a difference in education. What do I mean? They make students learn better. The students may even enjoy the experience more, and they help teachers think about how they're going to present information and with grading and assessment. Well, Chris Peach is a professor of computer science at Stanford University and an expert on using AI and computers to teach.

1:37Chris Piech:He focuses a little bit on teaching computer science, his chosen area, but he's had a wide array of applications that may surprise you. Chris, you focus on teaching people how to program, how to do computer coding with computer assistance. For people who haven't ever coded, can you just define what that's like and why it's challenging? Yeah, a good way to describe it could be. Imagine you want to give instructions to a computer. Maybe you want it to open a file, read through and then do some work for you. Or maybe you want to make a beautiful game. All of that, everything you experience on your computer, on your phone, it's all been created by a person who's written some code.

2:17Now, the experience of writing code is maybe surprisingly mundane. It's a little bit like opening up a Google Doc and writing down your instructions. You need to learn to speak the language of these instructions. And that's what I teach in my class.

2:30Chris Piech:Great, okay. So that's what we're trying to get to. And now I know that you are committed to trying to understand how we can kind of use computers to improve that experience. That will include AI, and I'm sure we'll get to AI. But what's the problem that we're trying to solve here? Are there not enough coders who know how to teach? Are we unable to create the environment for people to learn how to code? What's the problem? Yeah. So, you know, I'll actually broaden it a little bit in that I love teaching. Like, I think I was one of those people who was just born to teach. It just brings me so much joy.

3:02I would probably teach anything. I just happen to love coding as well. But I think a lot of the challenges I think are more broad. It's like, what are the challenges in learning? And if I had to name one, I'd say motivation is a pretty big one. Like, how do you get joy in education? I think there's plenty of kids in K-12 education. There's plenty of adults who need to get retrained. And they're not necessarily finding the experience to be that motivating or joyful. And part of my job is to come up with tools and things that can bring that joy into education. OK, so let me just ask.

3:35Chris Piech:It is not my instinct when I think about setting up a joyful learning experience to go to a computer screen and interact with a computer, right? I think about a group of people, maybe a teacher who is inspiring, who relates to me and also gets me thinking about how I can improve myself and learn stuff. So tell me how this turns into we should make computers better at teaching. Yeah. So actually, the question is really well posed. And I feel like I approach it from a similar perspective. You know, I'm an old man now. I've done a lot of experimentation. You have no idea, but we'll get to that later.

4:12You know, the single thing that I've seen have the biggest impact on learners is that relationship building with a teacher. And I think we can all appreciate and relate to that. So surprisingly, or maybe not surprisingly, but when the pandemic came and I had my chance to try and build my own version of joyful education, I had all the AI tools at my disposal. I sit in the AI lab at Stanford, but I actually turn to a rather simple idea that I think might really be important for education, which is people who have recently learned actually can be remarkable teachers if they're set up properly. So, you know, when I had my chance, I set up a classroom with 10 ,000 students, but 1 ,000 teachers.

4:59The teachers were all just a little bit beyond the students, and they're all leading a group of 10. I think learning is so relationship-driven.

5:08Chris Piech:You know, I love that because I tell when I'm giving kind of mentorship to new teachers, and I've been teaching for a long time as well, I say to them, you don't have to know everything. You just have to be about an hour ahead of all the students that you're about to teach, right? Because the class will be an hour. And so if I can get them to everything that I know, they'll never know that the next minute I would have been clueless. Yeah. To play it really safe, we get teachers who are about six weeks ahead of our students. Right. So that's very powerful. And how does that work? So there's a whole bunch of things there.

5:40Chris Piech:First of all, you have a whole range of students coming in at different levels of skill and knowledge. These teachers who are recent enthusiasts and recent acquirers of knowledge, it's easy to believe that they're enthusiastic. But how do you set it up for their success? Yeah. Yeah, you know, there's the program called Code in Place. It's a Stanford program, and folks are welcome to join if they find it interesting. So the way we set it up is we make sure that the teachers have good training. We kind of have an application process, and we select for who we think is ready to take that step. And here's one of the interesting things.

6:22By this point, Russ, we've had about 4 ,000 teachers. And in that experience, we've really gotten to learn what makes for a good teacher. And you would be surprised how great amateurs can be. You know, there's one way of talking about it is if they can learn to be humble and not overstate what they know, they have this real advantage, which is that they know the struggle. They know what it's like. And it's been recent. Yeah. And so, you know, as I said, we're learning a lot about what makes for great teaching. And certainly one of the big findings is we are really underestimating how great amateurs could be.

6:58Chris Piech:So you really gave a great answer because I was asking about how are we going to have those inspiring people in front of us? And even if it's through a screen or through a Zoom, it sounds like you've addressed that. And that's part of the plan. Let's zoom out. I want to ask about the status of automatic coding and AI grading. And there's just so many topics that your work touches upon. I guess I just want to, for people who aren't aware of it, computer programming has kind of been revolutionized, even among professionals. in the last couple of years because of AI. And could you just describe for us what the status is?

7:32Chris Piech:Because I'm sure that impacts the ways in which you think about how to get a new coder into the field. Yeah. So for those of you guys who are interested, there's this big revolution that's happened in the last few years. You maybe could call it the generative AI revolution. Basically, you realize if we throw an insane amount of computer power at a thing called a neural network, It can start to do crazy things like produce language. And you might have interfaced with something like ChatGPT or heard about it. What you might not know is that ChatGPT isn't just able to write text or poems about pirates.

8:10It can also write Python and computer code. Python is one of the languages that we program in. And I would almost argue it could be, I would say it's probably better at coding than at language. So if you've experienced these things, maybe a good metaphor is whatever you're thinking about its capabilities to produce fluent text, I would say it's doing a pretty good job of coding as well.

8:31Chris Piech:Okay. So that is now a thing. And if I understand that, even professional coders are taking advantage of these tools. I program all the time. I use them all the time. I mean, I don't know if you need to talk about this, but I've never had more fun programming. Okay. So it has not taken the joy away from programming because as a youth, I did a lot of programming myself and it was super fun. It was literally the only thing that could keep me up all night was a thorny, exciting programming challenge. And well, that's a whole different story. Sorry. So how does that change how you want to teach people the principles of coding?

9:08Chris Piech:It's a lot, you know, it's I'm sure it's very analogous to the problem that English teachers are having right now about writing is that they have the students have access to these powerful tools. It forces you to rethink how am I going to teach them to write an essay? And what's the equivalent question in your world? Yeah. Well, OK, so let's start by how fun programming is and let's talk about why it's so fun. And then I think that will help answer all your questions. So programming, it's a joy if done right. And one of the reasons a joy is you're creating things out of just what is in your imagination.

9:40You're like, I dream of this particular game and I can turn that dream into something I could share with my loved ones. And that creation is such a human experience. Like you have just made something that you can give. And if a loved one says that what you made is valuable, it's like the best feeling. Right. Right. And that's one of the reasons that I'm having more fun now is because actually these tools have elevated what I'm able to create. You know, there used to be limitations to how quickly I could read documentation and learn about new APIs and how quickly I could pick up new languages. And now with these new tools, I'm just having the time of my life.

10:13And speaking on that, you brought up English teachers having to teach essays. Right. As I said, I think a lot about motivation and education. And I'll say that there's this opportunity for creation all over. And it's not just encoding. Funny story of a three-year-old. And I did the funniest thing with my three-year-old. Of course, I want her to be able to read and write just like every other father. But also, I got bored one morning. And you know what we did? We just pulled up these large language models. And we just wrote a book. It was producing the images. And it was helping up with a story when we got stuck.

10:54And then once we got that book written, I just spent 30 more minutes and I got it printed with Google Photos. And then I had her holding a book. Oh, my goodness. It was the coolest thing. You can imagine my three-year-old, like she likes books, but to hold a book that she had helped craft, it was just a neat experience for her.

11:09Chris Piech:And she was fully aware of being part of this creative process, it sounds like. Oh, yeah. A little bit too aware. Now, if you come to my house, it's like the first thing she'll show you. And like we're trying to like tone that down, but that's a different story. But, you know, like that joy is cool. So, hey, so, Russ, what am I getting at? This moment for education could really be different. And if you talk to a lot of smart people, you'll hear a lot of people say this moment will be different because AI will be a great tutor. And I'm not saying that's wrong. But I'm saying one of the reasons this moment will be different is not because of what AI can do.

11:39It's what humans can do. It's using these tools to expand what we can create could be a way to unlock making learning more fun. You could go much faster from I'm learning the intros to I'm creating something of value. And I'm excited about that.

11:53Chris Piech:I can't tell you how reassuring it is to me. And I'm sure lots of people to have someone who like you, who's working on computer aided education. And I know that's an old fashioned term to have you say those things about joy and creation and the human aspect. It is so reassuring because as a child of the 1960s and 70s, you probably have studied as an archaeologist the kinds of tools I was exposed to where I remember as like a seventh grader saying I will never sit in front of a computer and try to learn anything and this is you know I became some something of a computer professional but they were such atrocious experiences that it probably delayed my open-mindedness by at least 30 years okay Do you want to hear a story on that?

12:41Okay. So, you know, I'm a scientist as well. And as you said, I'm a teacher, and I make tools, but I also like to do some science to see what's working, what's not working. Last year, as we mentioned, I had this big class, 10 ,000 students, 1 ,000 teachers. And we actually ran two experiments. In one experiment, we had early access GPT-4. Half the students got early access. Half the students did not. we do this a b testing when we're not sure if a tool is going to help people we ran a second experiment going on at the same time in this second experiment we'd kind of had a breakthrough in how we could do one-on-one teaching it's hard to do with 10 000 people we don't need to get into the details but we'd had a breakthrough and it allowed us to do one-on-one teaching we did a similar experiment where some people would get access to one-on-one teaching and then some people would eventually everyone gets access to everything but at the beginning we would like to learn what's working okay i'll first tell you about the one-on-one teaching in the one-on-one teaching 15 minutes with a near peer so somebody who's like six weeks ahead of you 10 percentage point improvement in your chance of completing the material of the class huge i've never seen a result like it that's the biggest result ever seen so i'll invite you to think what do you think the ai did did it achieve that human level of 10 percentage point improvement

13:57Chris Piech:So this is a human who has just learned, who's enthusiastic. You already told us that you filtered them for a bunch of characteristics. I can't imagine the AI was as good. It not only wasn't as good, it has a four percentage point less likely to finish the class if you had access to GPT. So we looked at the conversations. They were healthy conversations. They were talking about concepts. I think you're not the only child of the 60s. There's a lot of us who, there's something about the human that I don't want us to lose. Hey, there's something about the AI that's really cool. We'll talk about how we can bring that into the future of learning, but we better not let the human part go.

14:39Chris Piech:Good, good. Okay, I feel validated, and thank you very much. Okay, so let's go into, I know one of the areas, you know a lot of stuff, and I Google stalked you last night, so I know about all your papers. One of the things you've looked at is assessment of students. And that's getting serious, right? Because this has impacts on their future, on their job potential, on their ability to get into the next level of either education or job. So it's very sensitive and people are, I'm sure, worried about it. tell me how you approach computational assessment of learners how should we think about it and i'm guessing you've done humanistic things in that direction but i have no idea what they are no no it's such a good question so like you know um for a while there one of my main quests was to help us understand humans based off their work so you're learning physics you're learning english you're learning programming while you're doing this learning you're producing work um and you I think it's a grand challenge in algorithmic education to understand you from the open-ended work.

15:46Hey, if you're doing multiple choice, by the way, boy, can I really model what you know. It turns out I've got this algorithm. It's called deep knowledge tracing. We use it in Duolingo, but I don't dream of people doing multiple choice questions. I dream of people doing more complicated things. So that's the first bit. No one likes assessment, but But what we're able to grade is the assessments we can give. And if we can grade more complicated things, we'll be able to, as teachers, have more interesting learning experiences. So if the only thing we can grade is multiple choice, you're going to get a lot of multiple choice tests.

16:20If the only thing we can grade is you programming, you're going to get a lot of programming. You know, the dream, Russ, though, is that you could be doing an open-ended project, doing something that you really find motivating and enjoyable, and then we can give you feedback. Okay, so that's just a framework. From there, we could talk a lot about the state of the art.

16:38Chris Piech:Well, yeah. So I would like to know where are we? One of the things that comes to mind and it's terrible is and I don't even want to go there yet because we haven't talked about grading and assessment. But there's also the issue of cheating, which is kind of intimately tied in with all this. And I don't even like saying that word. Right. It's an ugly word. It's an ugly idea. So maybe we'll put that aside. But how do you approach grading fairly? and what do you do about the variety of backgrounds and cultural assumptions that students come in and then when you try to reduce all of that either to a number between 1 and 100 or a letter from A to F or whatever it is or even a paragraph you know our friends at UC Santa Cruz they write paragraphs describing how the student did how do you how are you thinking about how to do that and where is the state of the art yeah how do you want your three-year-old to be graded when they get to first grade oh i love the question you know um and i'll bring the same sensitivity like who am i to judge anyone you know i'm a curious human you're a curious human like i don't i don't presume that i don't really like this position of power and you mentioned how it could affect people's lives and that really makes me nervous um but it's a big deal and there's a lot of nuance to it you know the cheating nuance is an interesting one uh the effect on people's lives interesting one But maybe a safe place to start is actually the demographic fairness.

18:04So, you know, Russ, I've actually never used – I might be one of the few people in the world who's spent a decade studying this. So we've got a bunch of algorithms that you could call state-of-the-art. There's almost zero algorithms I've used for real assessment. And you know why? For me, grading is a different thing. Grading is the proof that an algorithm understands students. So a lot of the algorithms I want to make, they're going to help students in some way. They're going to help teachers in some way. And the central piece is, can you understand a student? It just so happens that grading is one of the few numerical things that I've got to see if an algorithm is doing a good job understanding.

18:49So I have done experiments. Were we to use this grading for a real assessment, would it be fair? As I said, we haven't actually used it for real assessments. And the answer was yes. Now, what was I grading? We took 10 ,000 students writing an exam for Code in Place. Now, right now, the exam is, we call it a self-diagnostic. You know, there's no feedback. You just take the exam and then you're done. Just the experience is the experience. But then we thought, what if we ran an AI grader on this? And then we ran the AI grader and we compared it to human graders. And you might not find this that surprising, but human graders are not all that accurate.

19:28Chris Piech:And they're a little biased in their own special ways. Yeah, yeah. And then we could see that we were a little bit more accurate. We then actually gave it to the students and said, like, hey, this is not a grade, but here's some automatic feedback if you find it interesting. And then we spliced in the human feedback and the AI feedback, and the students in my class were actually preferring the AI feedback. So that just gives you an idea of what's possible. I will say that the answer probably would be different depending on the subject. I think coding, you know, we have people from 150 countries, Russ.

19:57So like, I don't, I got to find out, is it biased against Nigerians? And the answer is no, you know, coding is something like a universal language. There's not too much about your demographics that leaks out when you're doing a little coding task. It could be really different if you're writing a personal essay. Yes, yes. It could be, and actually it could be really different if it's evaluating based off your resume, like all sorts of biases. if you're interested in these large language models that are trained off the internet that is a wild wild world

20:28Chris Piech:this is the future of everything with Russ Altman we'll have more with Chris Peach next

20:48Chris Piech:Welcome back to The Future of Everything. I'm Russ Altman, your host, and I'm speaking with Professor Chris Peach from Stanford University. In the first segment, we talked about the promise and possibilities for using computers in education. In this next segment, we're going to talk about how can computers grade less structured information, things where there's a lot of creativity and spontaneity involved. Also, how can you use computers for other kinds of tests, not just academic tests and grading? And finally, what is this idea of generative grading? We've heard of generative AI. What's generative grading?

21:23Chris Piech:I know that you've also done some work recently looking at feedback and assessment in a less structured environment. So tell us what's the state of that kind of work. Yeah. So if you're interested in giving somebody feedback, there's different degrees of difficulty. Multiple choice is the easiest. Short answer is the next easiest. A step up from that is something like coding. but then there is this particular type of coding which represents a goal we're shooting for, which is giving people feedback when they're just having fun and doing something unstructured. And so this does express itself in programming and it expresses itself when people are making games or web apps, particularly ones where it's just like, hey, you go make a great application, use the concepts we've learned in class.

22:10This is a really difficult grading task. we use a particularly neat idea. Have you ever seen a program learn how to play chess?

22:19Chris Piech:I think so. Yes. Yeah. So it kind of like learns by playing. It just like plays a whole bunch of games of chess and then it gets really good at it. So what we do when we're grading those open-end things is really different than multiple choice. What we do is we make a program. Now it's called Dream Grader made with a bunch of wonderful colleagues. And what Dream Grader is going to do is it's going to play your work. It's going to interact with it. It's going to, you know, if you made a game, it's going to move the paddle. If you made an app, it's going to try and press the buttons. And through interacting with its work, that's how it's gaining its understanding of what you're doing.

22:55And it's really wonderful to see. It's like the power of these chess engines in the hands of graders. I'm excited about it. Not just technically. Okay, I'd say like nerdy, very fun. But also as a teacher, because I want to give those things. I just can't because grading them is so hard.