Klinton Bicknell: Leveraging AI to Power Language Learning

0:04Hey, everyone, and welcome to Generative Now. This is the podcast where we talk to the builders who are creating the world's most exciting AI products and companies. I am Michael McDonough. I am a partner at Lightspeed. And in this episode, we'll be featuring another conversation from one of our generative NYC in-person meetups. This one from December of 2023. At the meetup, I sat down with Clinton Bicknell, head of AI at Duolingo. Now, you might not think of Duolingo as an AI company, but it turns out they've been using machine learning to power a lot of what they do for many years. And in the middle of this ongoing AI revolution, they're only leaning into AI a whole lot more.

0:44And so in this conversation with Clinton, we got into a lot of things like AI and education, what machine learning can teach us about the human brain, and where he sees further AI development heading. Check it out. Without further ado, please help me in welcoming Clinton Bicknell, head of AI at Duolingo. Thanks, Clinton. How's it going? All right. Thank you so much for doing this. Absolutely. So you have been, like I just said a minute ago, you have been thinking about the world of AI long before many of us in this room probably. Give us the story of your background and your journey to generative AI.

1:27Sure. So, yeah, my journey to AI and to Duolingo is, yeah, it's been a long and interesting one. If you go back far enough, I started going in this direction through cognitive science and the cognitive science of language. So understanding how the mind works, understanding how the brain works. I was doing research into that in academia. And it turns out that AI models or machine learning models specifically at the time can provide really good models of how humans do a lot of things and how the brain does a lot of things, thinking about the brain as optimization system, et cetera. And so I was doing research in academia.

2:07I was a professor at Northwestern into, you know, how people learn and process language by building kind of machine learning models that would act as sort of model systems that you could interrogate for how people can do these things. So that was kind of where I was and the AI that I was doing was very kind of interdisciplinary work at the time. And then about five years ago, Duolingo decided to start up kind of a proper AI group and reached out to me to be one of the first people there. So it seemed like the right fit at the time, the right time to make the move from academia into the exciting world of consumer tech.

2:49It's been a really fun ride since then. So tons of questions about Duolingo, which we will get to. and that transition, because that does seem like a big transition going from academia to consumer tech, but maybe focusing in on your experience that you were just talking about your studies. I think so many of us, myself included for sure, have been so blown away by the power of large language models and really what is enabled by transformers. And definitely for me, and I'm sure for many others, it starts to make you wonder, wait, is this how our mind works? Why is the LLM so good and is this how i have learned things maybe talk to us a little bit about the the similarities or differences between the way transformers work and the human mind yeah that's a great question uh so and it's something i i thought a lot about um i think a little bit less about these days i guess but uh i've definitely thought a lot about this question the similarity between the two is at a particular level of analysis so the brain is not literally a transformer It doesn't have that same architecture.

3:49It's not doing exactly that thing of having an attentional pass on a layer and moving that to the next layer. But at a high level, I think what modern transformers are doing, what a lot of different neural network models are doing is essentially just high dimensional function approximation. Can you... Yeah, yes. Sorry. So basically, in most machine learning cases, you're trying to predict something from something else. So there's a function that takes the input and goes to the output, and that's what you're trying to learn. And those functions are often really complicated functions. You can describe functions in simple ways, like a regression model.

4:27That's not a very complicated function. But most real functions are much more wiggly than that, very complex functions that you have to use a lot of different parameters to describe. And neural networks, it turns out, are really good at doing that. They have a lot of parameters that you can wiggle, and if you do gradient descent, that is a process that can make the network learn kind of functions of arbitrary complexity if you have enough data. I think there's a lot of evidence that at least parts of learning in the brain work kind of that same way. They also have a lot of degrees of freedom to approximate functions on the basis of seeing lots of data.

5:04Brains are kind of prediction machines in that same way. So I think that's a lot of the similarity. I think actually back when I was working in this space, the regime was such that as our AI models got better, they also became better approximations to what humans did. That is actually changing now though, because now our models are actually getting, as they kind of get like superhuman in certain domains, like in visual image recognition and things like that, they're actually now starting to look less like humans. It's starting to diverge a bit. So kind of interesting. Why is that? Just because the different types of data you can train them on now?

5:44Like unpack that for us if you don't mind. Yeah. Yeah. It's a great question. So I think, I mean, I think we don't know for sure, but the speculation is I would, I would guess the reason for this is that when models are just to make a very fuzzy kinds of predictions for the world, like they're not that great of predictions. predictions, human predictions are pretty good. And so if you make the predictions better, they're going to look a little more like humans because human predictions are pretty good. But once the predictions get as good as humans or even better than humans, then now making it better might take it further from what humans look like, right?

6:23If that makes sense. Yeah, it does. That's fascinating. Wow. So, okay. So you're doing this type of research for years and then Duolingo comes along and says, hey, stop doing that. Come build consumer products with us. Tell us what that was like. Yeah, yeah. So this was an exciting time. I was a restaurant at Northwestern. I was doing a mix of teaching and research. The research that I was doing was working with different communities of people, you know, people who understood the brain, people who understood, you know, how people think, people who, you know, did AI. And at Duolingo, the opportunity was very different.

6:59It was very different in a lot of ways. So Duolingo had very large amounts of data. We have the, you know, possibly the world's largest data set about learning. This is, you know, I don't actually know, remember what it was at the time. These days, though, Duolingo learners do about 10 billion exercises every week. And we get data about all that that we can use to personalize learning and build models of learning. And so I think the opportunity in terms of research, just by having access to all that data was quite large. That was a big part of what drew me there. I think I was also doing a lot of teaching at the time as a professor and realizing that being a single person teaching a single class, there are obviously some advantages to it, but it is not the most efficient thing in the world as you can scale education much more highly.

7:52At Duolingo, by contrast, you do a single A-B test and it's affecting tens of millions of users every day. It's a way to have a lot more impact in terms of education. And I think that the opportunity of just helping define, you know, what does the future of education look like when you put together the power of a well-designed consumer app with very large amounts of data and AI. So fascinating. You know, I think probably most people, especially people not in this room, just normal average consumers who don't think about AI and AI products probably don't think of Duolingo as an AI company, but clearly they've been thinking about it and building this space for years.

8:29Like what are some of the ways that that Duolingo has leveraged AI and machine learning over the past five years or so since you joined. Yeah, yeah, so that's right. So actually, Duolingo's history with AI goes back even before I joined. It's been really since the beginning of the company that they've been doing some AI. The first, you know, about over 10 years ago, the first AI models that Duolingo was building were in the service of using people's learning data to personalize learning experience. So for example, one of the first models was a model to use to figure out where to place people. If you say, hey, I already know some Spanish, where should you start in the course?

9:06We have to give you an efficient test to figure out where to place you. That's AI behind making that test efficient. So it's not like a test when you're trying to place into Spanish at college, where you go take an hour-long test somewhere. People are not gonna download Duolingo and then go do an hour-long test to start Duolingo. It's not gonna happen. So we need to make that efficient. Or another one was trying to predict what words people know and which ones they need to practice, things like that. So there was a lot of AI around personalization early on, and that is still a big part of the AI that we do.

9:40Over the past, I would say, five years since we had a proper AI group, that has really expanded to lots of different parts of the company now. And I would say over the past year, that's accelerated even further, expanding into even more parts of the company. And so that includes things like we have a Duolingo English test, which is an English certification test, like the TOEFL, where the people might take to go study in an English-speaking country typically. That test uses AI to help generate test questions, to help with grading. It's also a test you take at home, so it's remotely proctored. We use AI for security to prevent different types of cheating, things like that too.

10:16One of the things that Duolingo is known for is being a very engaging product. As we found, that's one of the real secret sauces to getting people to actually learn languages, is that you have to keep coming back to learn it. That's the only way. And so we focused a lot on engagement. AI is useful there in several ways. Some of those things that you would recognize from other consumer apps, like Duolingo's notifications that we send people are optimized with AI. But also building delightful, more interactive experiences. That's another big thing that we're doing with AI. So Duolingo used to be a startup, not a startup anymore.

10:52I think it's, last I checked, something like a$9 or$10 million market cap publicly traded company. But clearly, thinking about innovation and thinking about sort of the forefront of AI, there are a lot of startups out there that are moving insanely fast, right? And just shipping new stuff constantly. How does a big company like Duolingo keep pace and innovate at the level and speed of a startup in this moment of sort of AI explosion yeah yeah it's a good question um i i think a lot of different a lot of different ways i could take the answer one way of taking this is to say that the advantage of being the dominant player in this space right now is the data we have access to to lots and lots of you know 10 billion exercises people do every week and we are as we have been for a while we're investing more and more in trying to leverage all that data to really make the experience better and better and get that into feedback loops with our AI models, right?

11:49So that as we make the experience even better, then we get even more user data, and then we can make the AI models even better. So I think data is one way that we're thinking about this. But the other thing I would say is that I'm not sure how unique Duolingo is in this regard, but it is certainly the case that we are, like, there is a lot of excitement about AI throughout the whole company right now. There are lots and lots of different teams exploring lots and lots of different directions. Multiple teams working on things like more conversational experiences or teams working on generating longer form content, like a little radio show that you watch that's actually just launched, I think, in certain courses.

12:31Yeah, right, where we are using AI in a way to make a feature that just wouldn't have been at all feasible before. So I think it can be a tricky problem, but I think that by investigating AI in lots of different places, we're innovating pretty quickly. Duolingo, as you mentioned earlier, has possibly more data in the space of education than any other company. And you're leveraging it right now to change language learning and learning more broadly right now. But if you were to extrapolate and look 10 years into the future, how do you think education will be transformed by AI? So, okay, the first thing I feel like I have to say here is that 10 years feels like an eternity right now, given how fast things are changing.

13:16I feel like two and a half, I can maybe kind of say something, although even then I have to be very humble and cautious. This stuff is, you know, I've been working in AI for a very long time, and, you know, just the pace of progress in the past year has been quite surprising. So that said, I do think that there are two themes that I see as going to be really transformative in the learning space. One of those, I think, following up on what I've already been talking about a little bit, is personalization. I think just being able to have a product that knows you extremely deeply, knows exactly what you know, knows how you're learning, knows when you're going to learn, all of that.

14:00and uses that to really optimize learning for you in particular. I think that is something that we've been making steady progress on, but I think as AI is advancing quickly, that is something that's going to get much, much better at Duolingo and possibly at other places. The other, I think, theme there is going to be more richly interactive experiences, right? So for language learning, one thing that this might look like is having a lot more like, you know, realistic conversational partners or something like that, right, who are also adapting things, you know, to your level and helping out fill in parts that you're clearly not understanding and things like that in a realistic way that gives you, you know, the confidence to go have a real world conversation.

14:43But I think in other in a lot of other parts of education, this might look more like you're trying to understand some, you know, scientific concept, and you don't quite understand a part of it. And you can, you know, ask useful follow up questions to some agent and critically trust what the agent actually, how the agent answers those. That's, you know, you can kind of do this now, but you can't act, the trust part is not there. So I think those are going to be the two themes that really change what education can look like. And when do you think it leaps from, you know, self-serve consumer tech product that I download on my phone to being in the classroom?

15:17And maybe what types of classrooms do you think will first sort of really see AI enter? Yeah, that's an interesting question too. So, you know, Duolingo is already, okay, so two things. Duolingo, in particular, has been a consumer-focused, you know, education company. One of the few, I would say, that people airs in ed tech that are not focused on, like, selling to schools. But that said, we are, in fact, in the majority of school classrooms already. People use Duolingo as supplementary materials in the majority of foreign language classrooms in the U.S. and in many other countries as well. At all grade levels?

15:59I think so. Wow. I didn't know that. That's awesome. Yeah, it's kind of crazy when I first heard this. But yeah, it's all over the place. You know, often it's not like the primary mode of instruction, right? But it's more teachers will like assign people doing a little bit or it will be like, OK, if you finish this thing early, you go do some Duolingo or different things like that. And so I think that having that kind of like personalized study aid is something that is already having an effect in a lot of classrooms where the people who finish early can actually go learn more on Duolingo, which actually knows they're at a more advanced level and they can get that.

16:33while the teacher then spends their time focusing on the students who actually need a little extra human assistance. That is already happening to some extent. Yeah, and that's probably happening. I mean, definitely heard anecdotally that's happening with ChatGPT, right? Students are using ChatGPT to supplement their studies in the classroom, so that makes sense. I'm curious to imagine what the next phase of that might be, right? And I guess language learning is a great example of where that is already happening. Yeah, yeah. I mean, as a personal tutor that's like in your pocket gets more and more powerful, I think, yeah, the role of what, you know, what is a classroom and what our classrooms for, I think will have to change to some extent, right?

17:13Like, I think the most important part that I think we are unlikely to replace for a very long time about teachers is the humanness, right? The human touch, the human relationship that gives people the motivation to do something or to believe in themselves enough that they can do something. I think that those aspects of teachers are things that maybe someday in a sci-fi novel, like robots can do or like machines can do. But for quite a while, you really need teachers to do these things. And you need the peers, right? Like you need your peer group, you need your classmates. It'll be interesting to see what happens there as well.

17:47If the teachers could be supplemented or a long day even potentially, yeah, an AI tutor, like, but what happens, what happens about your classmates? Like you need that connectivity, you need that community to learn, right? What happens there? Yeah, that's an interesting question. Um, yeah, I asked, you know, Duolingo has, it has like leaderboards where, uh, which many people find extremely motivating. Um, it's not, that's not the same as, uh, as I think like your peers in a, in the classroom, but yeah, I think there's a tension a little bit, right? Between learning as a social thing and, uh, and learning that is deeply personalized to be exactly what you in particular need right now, right?

18:28Because, you know, if everyone is learning on their own personal trajectories, then you might have less in common with the other people in your class. You have been, as we've talked about a couple of times here, you've been thinking about AI long before many of us. What surprised you most about the past year since ChatGPT launched? Like maybe you saw that coming. Many people didn't. But what has surprised you? Yeah. So in full transparency, I did not, well, I saw chat GBT coming. I did not see GBT4 coming. And we actually saw, so Duolingo got early access to GBT4 back in September, before chat GBT was a thing.

19:08And yeah, did not see that coming. That was pretty shocking. I mean, I think the, like the quantitative progress that the models have made in terms of scaling them is actually very predictable, has been very predictable for a while, but the translation of that into real-world impact was very surprising. So that was honestly the biggest surprise, was the first time of seeing GPT-4 and what it could do. So you're saying even the capabilities of GPT-4 surprised you? Yes. The capabilities of GPT-4 surprised me, certainly. I mean, that was, you know, for people who were working in this space for a long time, been using lots of large language models.

19:45it was just a substantial advance and one that no one else has actually quite gotten to get otherwise I think maybe one other thing that comes to mind a slightly less technical point from the past year that surprised me surprised me in a way is that it's really changed who is involved in the AI conversation it used to be the case that a lot of companies where there were AI experts that are like hey we should use AI to do X and you have to go convince other people like, oh yeah, okay, sure, yeah, we can try AI for that. Yeah, let's do it. But now it's, you know, like every executive at any company is like, oh, we should use AI.

20:22We should, let's use AI for this. Let's use it for that. So I think that that is very different in a surprising new way. Another part of what's different or another related piece is that AI is much more democratized, right? Like two years ago, to do any real stuff with AI, you had to know a decent amount about AI and decent amount of technical knowledge about it. Right. Be somewhat of an expert there. And now anyone who can talk can do some useful AI with a large language model. You mean about the usability of the products or you mean training new models? Sorry. Yeah. Good clarification question.

20:57I mean, if you want to build a new AI feature, right. If you're doing it by basically like prompting a large language model, anyone, you know, anyone who can write language can do that. You don't have to know. you don't have to have any idea how the model works. And so that is the kind of democratization I mean, where AI experts have a somewhat different role to play now. What are maybe outside of education, some of the things coming in terms of AI that most excite you over the next year or so? We won't say five years or 10 years, over the next one year. Yeah. So, okay. So yeah, it's always hard to predict what's going to come, but yeah, we've been talking about education for a while.

21:37I will say one other area that I am particularly excited about is I'm curious to see how this is going to play out in the kind of bio and medicine domains. I think there was just the past few days, I think, there's a paper from, I believe, DeepMind, where they used AI to predict a very large number of new crystalline materials for material science stuff. but this is, you know, it seems like a pretty big advance for that field. I'm very excited to see, you know, what a similar result looks like in the biospace, right? I have to imagine a lot of this is coming and I think it's very exciting to see what the possibilities could be.

22:24Well, I could ask you questions all night, but I know people in the audience have questions as well. So we're going to open it up to audience Q &A. If you're in the audience and you have a question, somebody's going to bring a microphone over to you if you just raise your hand. Please wait to ask your question until you have the mic since, as I mentioned, we're recording this. And just say your name and what you're working on and then fire away with your question. Hey, how's it going? My name is Cyrus. Thanks for the talk. I really enjoyed it. I most recently worked in financial technology at Stripe.

22:55I left recently pursuing a number of AI opportunities. I worked in ed tech for a long time, including a company named Quizlet, which is a well-known consumer product, also does AI tutoring. One thing that I found at Quizlet during my several years there is that certain modalities in learning were really tricky to solve via AI. Or, for instance, you were talking about the English proficiency test. Speaking and writing were really hard to sort of get automated answers against versus multiple choice questions, reading comprehension, et cetera. I'm curious how developments in the last year or imminent developments moving forward are allowing you to, for instance, grade someone's writing better.

23:33I know that Duolingo has some speech recognition, but you alluded to this concept of multi-term conversations. Yeah, if you could walk me through what your thoughts are in terms of present day, what it's improving, and maybe in the next year or two, what's going to go on over there as well. That would be super interesting. Thanks. Yeah, that's a great question. Those are, you're right. Those are certainly some of the challenging problems that we face. So what's been happening in the past year there that has made some advances. So one of the reasons that speech is really hard to work with, both for a person's e-test or for just language learning applications, is that speech recognition is not a completely solved problem, especially in the case of people having very strong accents, especially people who are learning a language.

24:23right? Or non-native speakers of a language. And especially if you're doing it in a somewhat noisy environment, right? But I think speech recognition has also advanced quite a lot in the past year. To name one, the latest like whisper model from OpenAI. Sorry to keep talking about OpenAI. I guess it is their anniversary of ChatGPT today. That's right. One year today, right? Also, they've been in the news a bunch lately. So top of mind. It's true. Yeah, Yeah, that was a week. Yes, so the Whisper model, for example, we found to be hugely better at feature recognition than a lot of other models that we had been using.

24:58And that has been really game-changing for a lot of what we can do with those features. I think it doesn't fully solve the problem. And we're also, we have a lot of in-house research on how we can do better on, in particular, like learner language, right? Like English from people who are learning English early on, for example. but in terms of grading writing and things like that I think that is where the current generation of large language models really shine right I mean you you can at a high level I mean you could just like ask a large language model you know here is how we here's how we think about grading essays here's an essay what do you think it should be graded right you're not we're not doing just that but you can you can do that and get quite far actually quite a good start And so I think as large language models get better, as future condition technology gets better, these problems get easier.

25:53Hi, I'm Bhargav. And it's just exciting to hear your story because I'm kind of six years, you, like before six years, because I'm in academia now. And I came actually from industry into academia, but in cognitive science. And basically that's where I could connect to what you were describing. So my question to you is a bit abstract as well. So there is a certain history and philosophy of language and how it played a role in civilization and how basically we evolved, right? Different languages in different places. Now, if you agree that a lot of the recent progress, whether it's LLMs or just GPT itself, is owing to the great computational power that we could get in the past few years.

26:38and then the current pace, acceleration that it's going, are you still wary of some plateau that we might hit in the coming years in this side of things just by the virtue of the structure of language or what we can learn from how it played a role? So do you see any kind of plateau that we might hit even despite this computational power and this innovation? Yeah, that's a great question. I think it's not, to me, it's not a very abstract question. It's a very practical question that is a central one, I think, for the coming couple of years. Like we've, as I mentioned, as you analyze these models, like large language models, to take a single example, you can quantitatively analyze them.