When AI is your personal tutor with Sal Khan of Khan Academy

0:00Wondery Plus subscribers can listen to How I Built This Early Early and Ad Free right now. Join Wondery Plus in the Wondery app or on Apple Podcast. Listening on Audible helps your imagination soar. Whether you listen to stories, motivation, expert advice, any genre you love, you can be inspired to imagine new worlds, new possibilities, new ways of thinking. Audible has an incredible selection with over 1 million audiobooks, podcasts, and Audible originals, all in one easy app. Find the genres you love and discover new ones. explore bestsellers new releases plus thousands of included audiobooks podcasts and originals that members can listen to all they want with more added all the time enjoy audible anytime while doing other things household chores exercising on the road commuting you name it audible makes it easy to be inspired and entertained as part of your everyday routine without needing to set aside extra time there's more to imagine when you listen.

1:06Sign up for a free 30-day Audible trial and your first audiobook is free. Visit audible.com slash built. I've stayed in awesome homes on Airbnb in places like Athens and Berlin and Rome, and each time these places have given our family a chance to feel like locals. It's the best way to travel. Picture this. You're spending a cozy evening out of town in a home you booked on Airbnb. The space is thoughtfully designed and you think, I bet I could host my own home on Airbnb. Your home might be worth more than you think. Find out how much at airbnb.com slash host. If you're waiting for your AI to turn into ROI, maybe you need to do more than wait.

1:51Any business can use AI. IBM helps you use AI to change how you do business. Let's create smarter business, IBM.

2:05Hello and welcome to How I Built This Lab. I'm Guy Raz. The pandemic changed education forever. In just a few days, schools around the world had to move online. Teachers began teaching on Zoom. Schools started issuing students laptops. And platforms like Google Classroom became indispensable. Even with kids back in the classroom now, school is much more digital now than it was before 2020. And our guest today says that an even bigger change is on the way, a revolution that could transform the way kids learn even more than the pandemic did. Sal Khan has been on the show before. He's the founder of Khan Academy, an online teaching nonprofit that went from pixelated YouTube videos to a massive platform with hundreds of free tutorials in dozens of languages and tens of millions of users every month.

2:58Right now, Sal and his team are working on a new learning platform. It's called KhanMigo, which uses the generative AI technology behind OpenAI's ChatGPT to help students with their schoolwork. And the people at Khan Academy think that Khan Amigo could act as a personal tutor for every student and a teaching assistant for every educator. But we'll get there. First, let's go back to 2020 and the early months of the pandemic when Khan Academy saw millions of new students begin to use its materials. You know, the spike was interesting because that first week where you had global shutdowns in the U.S.

3:33and other places, we talked about it last time. Our traffic went from about 30 million learning minutes per day to about 90 million learning minutes per day, pretty much within that week. What was interesting is you fast forward a couple of months. As we know, a lot of school systems started to figure out how to do online schooling. Yeah. And then we saw things actually normalize, but maybe in a very abnormal way, mainly because I think people started to have screen time fatigue. and I think during the pandemic people had a, you know what, if we can just kind of pretend like we're going through the motions of school, let's call it a day.

4:10And because of that, people weren't looking to improve necessarily. They were just looking to tread water. What did you find? I mean, as you know, I mean, there have been studies over the past two years comparing students pre and post pandemic, right? Their standardized test scores. And it's clear that student performance in both reading and math fell significantly. Math, I think even more than reading. Have you found that Khan Academy helped keep students on track during the pandemic or is that something you even measure? Oh, yeah. This is something we've been keeping a very close eye on. And the thing I always point out, first of all, is that the numbers were not good pre-pandemic.

4:46What we saw, we did an efficacy study in the first full school year, the 2020-21 school year, and then we did another one in 2021-22. And what we saw in 2021 is that the students who put in an average of 15 minutes on Khan Academy a week in a school setting, they actually saw no COVID learning loss and that students who put in 30 to 60 minutes, they accelerated almost 40 % faster than pre-pandemic norms. And it's not a mystery. They're just getting more practice at their level with more feedback. And those that are supporting them, their teachers are getting more information about where the students really are and maybe can adjust their lessons a bit based on that.

5:28Tell me about what you, I mean, you must have learned, and I think now this is probably the third time I'm talking to you since the pandemic started, and in each time I've talked to you, it's so clear how much you were learning in real time and how much you've learned from what happened, sort of the overwhelming numbers of people coming to Khan Academy. Tell me how it sort of fundamentally changed how you guys operate. I think it did a bunch of things that we always wanted to do, but this just accelerated things. We always wanted better ways for students to address gaps that they might have. The fashionable term now in education circles is unfinished learning.

6:11We all thought that the pandemic was only going to affect the school system maybe through that first summer. And by the time you go back to school, even then there would be some damage done back to school 2020. I mean, now we know how delusional that was. Yeah. But we started creating these get ready for grade level courses, which essentially cover all of the prerequisites that a student needs in order to be ready for their grade level work. And we saw that that was very popular. So that was accelerated. We created another sister nonprofit called Schoolhouse.world. We saw how much people were leaning on Khan Academy, but we also saw that there was a gap of getting real human support.

6:45And so the utopian idea was, well, what if they could get real human support for free based on volunteers out there on the Internet? And then as we get to the tail end of the pandemic, and as we all know, people had very mixed feelings about COVID learning. Most people did not think it was, you know, spending time on video conference for several hours a day was a good thing. We felt the need to show that there's a way to do this well. And so another sister organization, we have a lab school that I helped start that's literally in the same building as Khan Academy. But we said, can we create an online version of this that can do online schooling, but do it well?

7:22Don't do it so it's just students listening to lectures on Zoom for hours a day. When people are on video conference together, make it interactive, make it Socratic, and then use personalized learning and other tools for students to get more asynchronous support, but always feel connected to a community. So we started working on a lot of things like that as well. All right. Let's go now back to not that long ago, back to the summer of 2022. You got an email from Sam Altman at OpenAI. And he said, he sent you a note, said, hey, we've got this thing we're working on and we would like you to check it out.

7:59So you did. Tell me what happened. Tell me the story. It's interesting. Almost on a daily basis, we get emails from folks saying, hey, you have an interesting technology. We love to partner with Khan Academy. And most of the time, you know, we look at it, we're like, oh, you know, it looks intriguing, but we just don't have the bandwidth. But I obviously knew of Sam and Greg and had deep respect for what many of the things that they've done in their lives. And I have been watching what's been happening with generative AI for the last, let's call it three to five years. I didn't really think it had a place at Khan Academy because GPT-2 and 3 were good at writing convincing text, but it really had no grounding in factual knowledge.

8:39So I'm like, I'm happy to meet with y 'all. I'm just curious what y 'all are up to. And they said, hey, we're going through our first training run of GPT-4, which obviously is now out, but back then that was very confidential. And we want to launch it with a small number of partners that we think can show a social positive use case because we think it's going to change things. I was skeptical at first because I was familiar with GPT-3. About two weeks later, I got another email from both of them saying, hey, we're done. Can we show you a demo? I'm like, oh, sure. So they put up an AP biology question on the chat and they, I answered it correctly.

9:15I'm like, oh, this is pretty interesting. And then I said, ask it to explain its reasoning. And it explained it. It gave the right answer. And then I said, explain why the other choices aren't correct. It gave that. And then I asked, can you create 10 more questions just like that one? And it did. And as far as I could inspect, They were pretty good. That's when I started getting the goosebumps. I'm like, okay, this is different. And then they gave us access that weekend and I spent hours with it. And it was doing very well, although it definitely had some issues. We realized pretty quickly that you couldn't just have it generate a ton of questions and just put students in front of it.

9:50It would make mistakes, especially in those early days. It was still pretty bad at math. But at the same time, we also, we started to make it role play. Role play as a tutor, role play as a character, role play, you know, do something in the style of someone else. And that's when I started to really say, wow, this really could be that holy grail that we've all been thinking about, reading science fiction about for decades of an AI that can actually emulate a human tutor. Before we really dive into what you did next, from what I gather, Sal, I think your initial reaction was the initial reaction that many people have, which was like kind of terrified.

10:34I read a lot of science fiction and it was a moment in which I felt that all my science fiction reading had prepared me for. But, you know, there were moments that even that first weekend where I was pushing a few of the limits, where I asked it things like, are there things that you think that you're not sharing with me? And it said, yes. You get a little weirded out by that. I'm like, well, why aren't you sharing it with me? And it says, well, it might offend you or it might scare you. and it feels like you're talking to an alien. It feels like you're talking to a super intelligence. And I had to keep reminding myself, I do understand how it works.

11:11I have a master's in computer science. I understand how large language models work. And I started to say, okay, I can get how, if it's just modeling the natural thing to say of why you're not sharing a thought, it is. Why don't you or I share a thought? It's because it might offend someone or it might scare them. So that's kind of what it was doing, but it was an other worldly feeling. and it was, you know, we had signed a non-disclosure agreement. So it was a very hard secret to keep. You know, every now and then I would go to dinner parties and I was like, I wish I could tell you, but the world is about to change.

11:43We're going to take a quick break, but when we come back, how Sal and his team struggled with the shortcomings and the dangers of GPT. Stay with us. I'm Guy Raz and you're listening to How I Built This Lab.

12:00One of the coolest things I did last summer was take my family to Athens, where we saw the Parthenon and the Agora and all those amazing neighborhoods and ancient historical sites. And one of the things that made it so awesome was the home we booked on Airbnb. We literally had a view of the Parthenon from our bedroom window. There was a hot tub on the top floor. We could sit in it overlooking the entire city. And we had a kitchen and access to markets. And we were in walking distance from all of the things you would want to see. It made the trip so, so amazing. And when you take your own vacation, that's actually a great time to host your home on Airbnb.

12:38Your place with your cool art collection and your handy kitchen gadgets. It might just be what someone else needs to feel right at home on their next trip. Plus, your earnings from hosting provide you with another income stream and could even help offset the cost of your next trip. Your home might be worth more than you think. Find out how much at airbnb.com slash host. This show is brought to you by American Express. Running a business means making countless choices every day, some small and some that can change everything. The business owners who thrive are the ones who are ready for whatever comes next.

13:18And having the right support behind those decisions can make all the difference. That's why so many business owners I talk to choose the American Express Business Platinum Card. It's built to help them move quickly and take their business further. And with five times membership rewards points on flights and prepaid hotels booked through Amextravel.com, you can turn any business trip into a chance to earn more and invest in your business's future. Plus, you get a flexible spending limit that changes as your business does, adapting to where you are and where you're headed next. Because when it comes to growing your business, there's truly nothing like Business Platinum.

14:02Not all purchases will be approved. Terms apply. Learn more at go.amex.com. You know, one of the most surprising things about how I built this is just how many pitches we get every year. tens of thousands of businesses reach out hoping to tell their story. Now, while we do try to read most of those pitches, we can't possibly read all, but we choose most of our guests through our own deep research. And that means my team and I spend hundreds of hours digging into companies, what they make, who's behind them, and whether their journey can inspire you. And that's exactly where Claude has become such an incredible thinking partner in my creative process.

14:44Claude is the AI for minds that don't stop at good enough, the thinking partner that works with you to explore the things that fascinate you. Whether you're researching late-night curiosity spirals or working through complex creative challenges, Claude extends your thinking to tackle the problems together. For example, imagine I'm about to sit down with a founder of a snack company. I'll want to know what products they've released, how they've evolved, and which competitors they're up against. In the past, it could take hours to track down reliable details. We'd search articles, databases, reports.

15:19Now, Claude can give me a clear answer in seconds, and then I can decide what to dig into further. It doesn't replace my process, but it sharpens it. And that means I can spend more time doing what I love most, focusing on the human stories, the lessons, the decisions, and the struggles behind the businesses you hear on this show. Experience what collaborative thinking feels like at claude.ai.hibt and see why the world's best problem solvers choose Claude as their thinking partner. One more thing before we get back to the show. Please make sure to click the follow button on your podcast app so you never miss a new episode of the show.

16:03And it's totally free.

16:12Welcome back to How I Built This Lab. I'm Guy Raz, and I'm speaking with Sal Khan, founder of Khan Academy. In 2022, Sal got early access to the large language model GPT-4 before ChatGPT was even released, and he was amazed by its potential. So you sort of get over your initial kind of, oh, my God, and you start to see obviously a huge potential here to kind of transform what you do, which is offering free education to people around the world. So what did you do next? I mean, the chat GPT wasn't going to be the exact right thing, right fit for what you do, but you could build on that foundation, right?

16:56That's right. The first thing we did is we sat in an org-wide NDA and I said, hey, how many folks in the organization can we give access to? And eventually we got about 40 or 50 folks in our organization to get access to GPT-4. And even that, this was the first time that Khan Academy had to do a little bit of cloak and dagger type stuff. Folks at Apple are used to keeping secrets and things like that, but we're not used to keeping secrets and we had to. This is obviously very sensitive stuff. And we immediately, we had a hackathon, which was well-timed because then these 40 or 50 folks were able to, at least for a couple of weeks, say like, okay, let's just be generative here, no pun intended.

17:32Like, what is possible with this? And by the end of that two weeks, we had some really cool demos. And then we started to have the internal debate of like, okay, how aggressive do we get on this? What about bias? What about the math errors? What about hallucinations? What about just the PR implications? People trust us. And if they see us leaning way forward on something that's not perfect, that's something that some people could be wary of, what does that do to our brand? What does that do to the trust that folks have? On the other side, folks were arguing, look, this is a transformational technology.

18:04This is one of these things where it's either going to allow us to magnify our impact by an order of magnitude or it might make us irrelevant. And so we made a decision to say, you know, we have to keep moving forward, but we have to do it in a risk aware way, in a risk mitigating way. All right. I think it's fair to say that you are a true believer in this and your team basically developed an AI that you call Conmigo. And so this is now, I think, in beta, but still integrated into Khan Academy. So tell me how this works. What is it? So first was the guardrails. Like in what context would we even feel comfortable doing this?

18:43And what's interesting is we were starting to do all this in earnest in September and October. Then end of November, chat GPT comes out. And I remember the day that it came out, I slacked Greg Brockman at OpenAI and I said, hey, Greg, we're under NDA and I thought we weren't launching anything until March of 2023. What's this chat GPT thing? And Greg says, it's nothing new. It's based on GPT 3.5, which had been out for many months already. No one really took note when they released GPT 3.5. And OpenAI just decided to publish a bunch of apps that use GPT 3.5. One of them happened to be ChatGPT.

19:22And then being able to interface with the model in a chat interface, I think, made everyone see what the power of it was. And at first, I was a little bummed. I was like, oh, this is going to steal the thunder. I've been telling all my friends at dinner parties something big is coming in 2023. Just, you see. But then I was, I think it was actually a blessing because as we know, ChatGPT goes out there. It amazes people, but it also scares people. And then immediately it creates huge issues for education. Like kids are cheating using this. Right. You can type in an equation and it'll give you the answer.

19:55And you can write your essays. You can write your essays, right? Yeah. And that, I think, gave us more license at Khan Academy to say, look, the genie has out of the bottle with chat GPT. We need to double down on bringing a better version of out there that can mitigate all the risks and maximize the benefits. So we started saying, well, you know, first, whatever we make shouldn't be for cheating. It should be for actually helping students learn. So there's many things we're doing to make it Socratic, not just give you the answer. Two, it has to be accurate. It's not OK if it doesn't know what seven plus four is.

20:26Yeah. It shouldn't hallucinate. And then we also were afraid of students misusing it or using it in some ways that might harm them or other folks and just reassuring teachers and parents that they can keep track of what their students are doing with this very powerful tool. And so that's why we made it so it logs all of the conversations. We have a second artificial intelligence that monitors conversations with the main artificial intelligence. And if any of those conversations go into a gray zone, then it actively notifies the parents and the teachers that, hey, you should take a look at this conversation.

20:59And then we're continuing to add a whole bunch of other things, things where teachers can use it for lesson planning, we're adding memory to it, how teachers can essentially use it as a teaching assistant, get reports on what students are up to. I could go on. So essentially, the idea is, hey, if we can get every single kid a personal tutor, then it will have an impact on their academic performance. And this has actually been studied and measured. I know you gave a TED Talk back in March of 2023, and you cited a study that came out in the 80s, basically, that showed that when kids have one-on-one tutoring, their scores and grades just skyrocket.

21:35Like you can turn an average student into an outstanding student, for example. That's 100 % right. If you want to become a great pianist, you're not sitting in a class of 30 with your piano and the person's lecturing you. If you're a great athlete, you have a coach who's optimizing you, not just giving a lecture to 30 students and saying, hey, you might want to improve your swim stroke in this way, and then who knows if you actually do. So one-on-one tutoring has always been the gold standard. When we did mass public education, which was a very utopian idea 200 or 300 years ago, but we made compromises.

22:03We started batching students together, applying set standards, lectures to them. Some kids get it, some kids don't. Even if you have a gap in your knowledge, too bad, move on to the next concept. And as you mentioned, Benjamin Bloom, 1984, he articulated it well. He called it the two sigma problem. two sigma, sigma the symbol for standard deviation in statistics. And a way to think about it, two sigma improvement, two standard deviation improvement is dramatic. It's going from the 50th percentile to I think the 95th or 96th percentile. But the reason why he called it a problem was, well, how are you going to give everyone a one-to-one tutor?

22:38And he also, in 1984, theorized, well, maybe you could emulate some aspects of that with technology. But I've been citing that study for many years in terms of what Khan Academy is trying to build. essentially old Khan Academy or base Khan Academy trying to get to that first standard deviation. But now with the AI, we can go to that second standard deviation. So essentially the interface, if you're listening and you're familiar with the interface of Khan Academy, let's say you get a math problem and you can essentially click a little sort of eyeball robot icon in the corner and say, hey, I don't understand this problem.

23:12Can you help me? It will not give you the answer. You can't say, hey, can you solve this for me? It's essentially you You ask it to help you. And then what happens? Yeah. And if you say, give me the answer, say, hey, I'm your tutor. I'm here to help you. You need to learn here, as I think a good tutor would do. And if you say, OK, well, give me a hint. It'll say, well, let's take a close word. What is the problem actually asking for? And if you say, well, I think they're asking for this. If I got it right, they say, well, yeah, that's good intuition. OK, so where would you take that? Or let's say I say, I think the next step is x squared minus 5.

23:43It's like, OK, take a double look at that. Are you sure it's minus 5? Pay attention to your signs. And one of the things we've done to make it more robust on the math side is when a student presents their math to the AI saying, hey, I think this is the next step, the AI behind the scenes comes up with what it thinks are reasonable responses from the student. It doesn't share those with the student. Then it compares its reasonable responses to what the student said. And if it gets something different, it'll tell the student, hey, I'm getting something different. Can you explain your reasoning? which is very pedagogically strong.