AI's Role in Personalized Math Tutoring with Peter Relan

0:00We're never wanting to automate away the instructor. We want to automate using AI some of the workflows that take a lot of instructor time. There's a difference between wisdom and artificial intelligence. Human wisdom is irreplaceable. Artificial intelligence is also irreplaceable. It's this partnership where we don't automate away the human because the wisdom comes from their expertise. We've got to combine that, the human touch, with the automation that AI brings.

0:32Welcome to the Expert Intelligence Podcast. I'm your host, Paul Estes. Today, we're joined by Peter Reeland, founder of the Uweb Incubator, one of the initial investors in Discord, and a tech industry veteran. His mission is deeply rooted in his personal story of growing up in poverty with two parents who believed that education would transform their children's lives. And it did. Peter is working to pay it forward with one of his latest projects, working to give students around-the-clock coaching for those make-or-break math courses in their career. Peter, welcome to Expert Intelligence. Thank you, Paul.

1:06Really a pleasure to be here, and we've known each other a long time, so I'm looking forward to this. Well, I want to start with where we first met. I was interested in the gig economy. I'll never forget the morning that I woke up and was reading Product Hunt, and I saw it wasn't even AI, it was Got It at the time. It was a unique approach. I hadn't seen it before I'd been in tech. And I reached out and I remember flying down to meet you and the team. And it was a tutoring service where you were connecting people in real time with tutors and education. What did you see then? Because I think it's really important to understand sort of a thread in your thinking.

1:41What did you see then that we're going to talk about later with MathGPT? So, yes, matgp.ai, next topic, but let's go back, rewind, and look at why we got here. So, when we started on the project of what we call conversational microtutoring, the idea was that the iPhone had taken off by the mid-2010s, for sure, well before that. And people were text messaging and SMSing. And we felt it was really important to engage in not just Uber-like on-demand services and DoorDash and so on, which are very much part of life today, but try to give learners an opportunity to have a conversation just the way they would with a friend on demand, just like with a good friend, they would be able to just get them or reach them, but in the learning space.

2:34And the goal was, and it still is, that human wisdom combined with artificial intelligence. Now, at that time, it was human wisdom first. We call them experts, and that's why expert intelligence is what you call what you do, assisted by AI with the goal that we would at one point flip it, where it would be artificial intelligence assisted by human experts. So we've come today to mathgpt.ai, which is exactly what I just said. We started out with humans as the primary sort of resource you could get to with AI in the background helping them to mathgpt.ai where you are interacting with an AI first in learning and education, starting with math.

3:25And there's a human, it's an instructor-led platform, So there's a human in the background who has overseen and set up this whole interaction that you as a learner are going to have with this AI. So to me, it's just the question of when we flipped it over, waiting for this moment, but starting with a kind of a, you know, who's primary and who's the backup. I had the opportunity to hear you talk and really learned about your passion, not only around math, and I want to hear your story about being a math tutor. But why math? Why is math such an important subject? And what problems specifically are you trying to solve with the technology that's acute that you see?

4:09So broadly speaking, and we'll get to how I even discovered math, but broadly speaking, what I feel is a mass scale, and that somebody at Wellesley College has written a nice article, we have a bit of math innumeracy. We call something illiteracy when we can't read or write, and there's this new term innumeracy when we can't figure out lots and lots of things which have some very basic math behind them. So that's the first broad macro trend. If you look at COVID and pandemics, if you look at results of vaccines, if you look at demographic data, if you look at GDP data, if you look at tariffs, which, as we all know, there was this equation that came up that people were like, what is this equation, right?

4:59life, too much of life is governed and we need math literacy or numeracy, a comfort with all these things going on in our world that we may not have because too many people believe they're not good at math, right? So that's a fundamental problem. In particular, there is this notion of pathways and gateways when you get to the first year of college or even dual enrollment, where what I found, and this goes back to my math tutor days at UCLA when I was just a 19-year-old kid, but I was tutoring other 19-year-old kids. And I discovered that a lot of people were giving up on majors that could have made their life and careers a lot better or different anyway, because of a midterm or a quiz that they just didn't feel they could get through.

5:44And so you do find that math is extremely important in majors and careers when you first get to your college math courses. If you want to be a nurse, you want to be a sports journalist. Do you know that sports journalism? You wouldn't think it would be blocked by, well, you got to take statistics to be a sports journalist, right? You want to be an electrical technician? You've got to take some math. So it turns out it's a real gateway blocker. And the correlation between not passing math and I'm not causing cause and effect, but correlation and not getting through college in six years is very, very high.

6:19You're right. Every single career now is computer-based. And it's kind of like the argument of when people are introducing AI and they think it's a computer science course, when actually AI is a general purpose technology. And I think the framing of math is the same. There's probably not a job out there today that some innumeracy is not needed. Not having it is detrimental. When you think of bringing this technology into schools, So you've got this math product. It can help any student at any time that needs coaching. And I think one of the things that when you were talking resonated with me is I was fortunate to have tutors.

6:55And I actually had a math tutor growing up. You had to schedule it. We had to drive there. We had to sit down. We had to do those things. And more and more college kids are studying in the middle of the night or in the morning and might need on-demand expertise, on-demand coaching. When you go to bring this technology into schools and it's AI-based, there's kind of a resistance that we're all seeing that says, hey, I get the AI, but today AI, in a lot of regards, is responsible for systemic cheating inside the university. Yes, yes. What is the framework or the approach that you have that helps address that?

7:30And it helps not only address that, but educate people, administrators, and faculty that, hey, there is an approach that is additive and it's not cheating. Yeah, that's a great numeracy. Higher numeracy because you have real learning. Yeah, that's a great point. So today, madgpt.ai is being used. And I always say madgpt.ai because there's a lot of madgpt sort of websites out there, which are just solvers. And that's the ones that are allowing people to cheat. But what we do is we're instructor-led. So we're using hundreds of classrooms today, instructor-led in community colleges mostly and state colleges.

8:04And why? Okay, the fundamental tension we discovered was between the propensity of a student to just want to get through the class, and therefore if a convenient tool is available to help them, just give them the solution and they can type it in, that's what a student's going to do. It's just life. Let's accept that. On the other hand, the instructors are worried that if they do bring AI into the classroom, they cannot trust it for accuracy. As you know, there's been a lot of discussion of hallucinations and the lack of guardrails and trust and so on. So this tension has existed with one caveat, which is the students continue to increase their use, which if we didn't address by saying, hey, what if instructors, we gave you a guardrailed, trusted, accurate product, not just a raw chat GPT that you can give to your students with some discount, But an actual product built around these language models that added guardrails, that checked everything for accuracy, that was cheat-proof.

9:10So we often show them a demo where in the chatbot, MatGPD is a product. It's got a great UI, and then there's a side pane where you can chat with the chatbot. And you ask it, given a problem, just give me the answer. It says, I'm not here to give you the answer. I'm here to help you understand. Do you know how to do the first step? It's called Socratic tutoring. So once we showed instructors the power of an application built on top of AI that gives them the tool and the product they want versus the raw technology, which is very powerful, we know, but has hallucination issues and guardrail issues, then they were more comfortable bringing it to their students, right?

9:47Because now they feel I can take charge of AI in my classroom versus just kind of trying to figure out who cheated and who didn't. And then the student still gets a tool that is very close to what they would have had if they just went to chat GPT, with one exception. With the chat GPT or other tools that are language models alone, a student is not a good judge of whether the solution or whatever the thing gives you is correct. They just don't know enough to make that judgment call. So they don't know. So that's why they prefer MatGPT.ai, because we guarantee it's accurate. And we've had bug bounties for years on this to all the pilots we did well over a year ago.

10:31So we have guaranteed the accuracy for certain subjects that we support. And second, we don't allow cut and paste into MATGPT, which is where their assignments and practice problems come in. So even if you were using a tool externally, you can't just copy that over. So you'd have to go through a fair amount of effort to use an external tool that gave you the answer, then bring it in. you may as well just work with this chatbot, which kind of helps you understand and then you're finished. So we really brought the student and the instructor together as opposed to this divide that existed with this trust on one issue and the propensity to just want answers on the other side.

11:06One of the big trends you made me think about is personalized learning. We all approach a topic like math or science from a certain starting point. What are instructors able to get from the way I learn. So I'm using the product. Is there any feedback loop back to the instructor that helps them understand students better? Because they're on the front lines. It's not just this product. They've got a bunch of other coursework and projects and in-classroom learning. What are the instructors able to get out of this? Yeah, this is great. So we are not just a student jackpot tutor. It's a complete solution.

11:40It really has three parts. MattGP.ai has a course manager. You bring in your course, your materials, your PDFs, your lecture notes, your slides, your textbook. We allow you to bring all of those in and get your course organized into the sequence in which you want to teach. Now, that's expected because how else are we going to know what to tutor, right? So that would be what we call table stakes. But it is an instructor co-pilot kind of experience, right, where they get to actually set up the course and say, here's the best part then. Once that's set up, and it takes about 30 minutes for them to upload their PDFs and this and that, they hit a button and it will generate the entire set of assignments, quizzes, and practice problems for the course, for the entire course.

12:22You just choose the chapters you want and what sequence and it says here's. And these assignments, quizzes, midterms, practice problems are all auto-gradable. That means all they have to do is set the due dates and grades that they would give to each assignment, et cetera, the number of marks. And suddenly they have a full teaching assistant now who will dish out the assignments, a tutor sitting on the other side of the student who will help them with all the different things. They can control how much tutoring is given in a quiz or a midterm. No tutoring. You have to take it. In homework assignments, maybe some difficult problems, tutorable.

12:58Practice problems, all tutorable. These are some of the operational practices we're seeing. And now all the grading occurs. And what they get is a dashboard with the analytics for the entire classroom in real time as the class makes progress. That analytics, because you're working inside a real-time system, is just invaluable. They can see typically what they see is three levels of a class, above average, average, and below average. And they can quickly group those students and even drill down to a single student to see, did they do a lot of practice problems? Oh, wow, they did, and they still did it right.

13:31Let's give them this particular student five more. Or they can have remediation materials and say, oh, you're doing a calculus course? You don't look like you got the pre-calculus trig stuff, you know, figured out. And then they can add those PDFs and textbooks and recommend certain practice problems. It's incredible if you think about that level of ability to personalize learning. And that's just on the instructor side. I can talk about the student side in a bit. One of the interesting things when you were doing, got it, the tutoring, you were out talking to schools. I mean, you've been at this.

14:06This isn't like you woke up one day and said, hey, I got this thing called AI. Here's my latest AI app. And one of the reasons I'm inspired by you and just love the work is because everything starts with human expertise. And then you figure out how technology can help. But tell me one story from mathgbt.ai on the ground, where there may be a student or a teacher, just some anecdotal story. Yeah. I bring a couple to mind. So the first pilot we did was with a statistics professor, adjunct instructor at De Anza College right here in the Bay Area. It's one of the biggest colleges we have in California.

14:42California has the biggest community college system in the country, 2 million students. A lot of them, two-year degrees. A lot of them go to transfer to the UCs and state schools. So we're talking to the statistics instructor. Here's an incredible story. These people, we hear about, as you know, what's in the news today, large universities, research universities, Harvard, Penn, all the big ones are getting their budgets cut. Why? Federal research budgets. Those are research universities, right? There are researchers there who teach. Here in this community college, it's a teacher. And we said, OK, here, look, we have to pay the costs of this product.

15:17He just loved the product. He said, this is what I've been waiting for. This is something where I have to answer emails at 9 p.m. or 10 p.m. or 11 p.m. Why? I said, but don't they have a tutoring center? They do, but it closes at 6 because the tutoring center is other students and people who are hired to come in and do their job. And what we found is most people are doing their assignments at 8, 9, 10. And so they send emails or they don't do them. And then they come to my office hour two days later because I can't hold office hours every day. So we saw this instructor just like fell in love with this tool.

15:52He's now on to his sixth quarter of using NATGPT.ai nonstop. But here's the interesting part. So when he used it, I said, look, it's$25 per student per course, right? And I said, you just have to pay the bills. We've got these very expensive language models running in the background that we have to pay for. And so compared to most of the other Pearson or McGraw-Hill tools that they usually buy for their students, which are$50 to$100 per student, this was a very good price. And you know what he told me? It's a beautiful story. He said, I care about these students so much, I'm going to pay for it.

16:25So his class of 40 students, he says, I'll pay the$1 ,000. And what we did then is we just waived it. We said, we can't let you pay the$1 ,000. He said, it doesn't matter. These students are going to benefit so much from this. When I do printouts, I pay for the copying and the paper. This is well worth it. So it was a beautiful example of technology meeting human who cares and is the expert on the topic and is doing the work with Matt GPT because it's not like the students could just adopt it, right? He's the human intelligence who took Matt GPT, applied it to his classroom. I want to talk about broader just AI and education.

17:01I was fortunate last week I got to attend a hackathon for an education trust in the UK. And it was eye-opening to me how early in the adoption or AI literacy journey, most of the secondary schools, even primary and secondary schools are, especially in the wake of what China just announced, saying, hey, every student starting today from first grade all the way through college will have at least eight hours a year, which doesn't sound like a lot, but it's tops down all students. Oh, yeah. How do you think about helping educators, faculty and administrators, how do you think of approaching them to start their journey of AI literacy so that they can better have experiences like the statistician that you spoke about?

17:43So there's a whole field in education called PD, professional development, whether it's teacher development or professors or instructors, right? And one of the things that I am sponsoring this year, in fact, more on the nonprofit side, is the notion of a nationwide tour that I will personally be conducting with a couple of colleagues who are in the NGO, where we will be talking about AI literacy and how AI can help instructors. Why? And one is because we want to educate the instructors on how to use these kinds of tools, not just MathGPT, but generally speaking, AI and what it can do for you. But here's the really important part.

18:22What I find is that if we were to call people about AI literacy and the use of AI, et cetera, the students are actually further ahead. They're actually aware of prompts. And by the time they're a little more grown up, but they're not as well, I would say, educated on the more what I would call productive uses of AI. You've heard of character.ai, which people are coming out and they're playing with AI. You heard of all the chat GPT weird messages and where they try to hack through. So there's a lot of pop culture that is building up around AI that I think students are very familiar with. How it actually helps learning is a whole topic.

19:01and the goal is to actually get instructors to, A, understand how it works, and not just math, cross the board, and B, get them comfortable with the idea of how to deploy it and how to co-opt the student. Because if you look at calculators, a lot of people give the example of calculators, we succeeded in both sides getting comfortable with that technology. One of the things we do is we do a lot of hacking testing. The idea is if the student does get their hands on it, and the instructor is configuring it, the student is going to try to hack it. This is just the nature of somebody who's going to try it, right?

19:36I'm not saying every student. So we want that. We actually offer a gift card if you can hack it and try to make it say something that it shouldn't. But it co-ops the student. So I am a big believer in AI literacy for all students, not just college students across the board. And I'm a big believer in a joint understanding, not just the instructor's understanding, and then the students' colloquial understanding or cultural understanding, but a joint understanding. I think YouTube was very good that way, right? Both sides co-opted it. I think Calcular is very good. We've got to make AI that way. And that's going to be a mission for the next, you know, many months, two years of bringing a joint understanding of AI in education.

20:20I think the thing that is interesting to me in your approach is that it comes from a deep experience in the topic. You sort of got to this place from a lot of education, the math focus and tutoring. It feels like a... It's in my DNA. It's in your DNA. It's in my DNA. You're not a Silicon Valley tech executive coming and sharing some shiny object. It comes from a very deep understanding of not only the problem, but the earned wisdom and things that it took to kind of get to this point, which is exciting in not only your story, but in product as it sits today. And especially at this time, I think I was with a bunch of faculty and we were talking.

20:57In fact, even this morning, I talked to the National Institute in the UK, a bunch of teachers on their professional development. It was a professional development session. And we had a conversation and they were asking, help me understand where technology isn't cheating. I mean, they were open to the idea of wanting to learn more and really embracing that curiosity, which was really exciting. Which is right, yeah. And another point I want to make on that, a lot of people do ask me, so everybody's talking about automation away of all the jobs, right? And we won't have any need for anybody, robots and AIs will do everything.

21:30This is where your topic has always intrigued me, expert intelligence, human intelligence. I always say we're never wanting to automate away the instructor. We want to automate using AI some of the workflows that take a lot of instructor time, setting up the assignments, grading the assignments, trying to understand the classroom and collect all the data and do the surveys and what we're going to do all of that through AI. There's a difference between wisdom and artificial intelligence. Human wisdom of a teacher like this person I was describing in the statistics class is irreplaceable. All right.

22:05Artificial intelligence is also irreplaceable for doing some of that stuff. I'm looking into the future. It's this partnership where we don't automate away the human because the wisdom comes from their expertise. As you were describing mine, I was thinking of this gentleman who did our first pilot statistics. He's like my age. Well, 30, 40 years of experience doing this stuff. The ability to read a student's thoughts and mind around when they are in a classroom or when they are struggling with something is very different when a human being can read that and can draw on a vast amount of experience having taught millions of students, right?

22:46And have an intuition. It's in their DNA, like we were saying. We've got to combine that, the human touch, with the automation that AI brings. So the vision is never to automate away the instructor. You may find that the instructor can run now three times as many classes, right? So you'd say, well, they're not really running the class. But the reality is, why do we call it a co-pilot? Well, who's the pilot? The word co-pilot has taken over. Why? Because you were at Microsoft, right? It's like the biggest product out there right now being used by I don't know how many people. Because there's a pilot.

23:19And the idea of a co-pilot is you don't automate away the instructor. And there's a big trend around coaches, AI coaches. And I think that language, I think people are starting to understand how to frame it. I have the meta glasses I just got a couple of weeks ago. And I'm finding myself, I go on a walk and it plays music. And I'm finding myself kind of as a coach, like asking a question. It's about my day. It could be about my relationship. It could be about parenting with my kids. I'm finding it as a good on-demand coach, not to tell me some top secret, really complicated things. It usually just tells me the obvious.

23:53Yeah. But it's useful. It's like a sounding board. But ultimately, you will still use it as a co-pilot because whatever it is, you're going to be, it's not going to exercise for you. It's not going to eat for you. it's not gonna have their relationship for you, hopefully, because of the human nature of certain things. So exactly. We talk about moving from AI tutors to AI coaches now. Why? Because we think they're smart enough to actually go beyond the particular session you're having. You understand your whole session history. We're talking about things like personalized knowledge graphs.

24:32We're MX plus C interesting to somebody who's a swimmer, okay? Or somebody who's a runner and is going up a hill. If you just think back to everything you and I learned, how much of it was made relevant to me personally in my life? And how is it possible for a single instructor to have 40 students in class and know each of their interests and say, oh, okay, let me give you a practice problem where you're running up a hill. The slope is defined by these two coordinates at the bottom and the top of the hill. What is that slope and how do you express it as y equals mx plus c if you start at the base with a certain amount of distance?

25:08That is what AI can now start to do, is really be your coach. There's a difference between saying AI tutor, which I think we all understand as a subject. A coach understands you and the subject. That, I think, is where we're heading to. You say heading as if it's very far into the future. No, it's this year, next year. Yes, exactly. Hey, Peter, thank you so much for taking the time. I, again, continue to be inspired and really learn from how you approach technology, starting with human expertise. And I think that's really missing in the market. I think it's a different view. And I think in the long term, that mindset and that approach along with AI is going to make a difference in education.