In short
Big Technology Podcast Episode Summary
Episode Title
The Professor Who Required His Students Use ChatGPT In Class — With Ethan Mollick
Guest Introduction
- Ethan Mollick: Professor at Wharton School, University of Pennsylvania, expert in generative AI applications in education and the workplace.
- Creator of the newsletter "One Useful Thing” on Substack.
Episode Highlights
AI in Education and Work
- Discussion centered around the integration of AI, particularly ChatGPT, in education and workplace settings.
- The professor emphasizes how generative AI can reshape learning dynamics and work processes.
Current Trends with ChatGPT
- Acknowledgment of a drop in ChatGPT usage, attributed to:
- Seasonal changes (summer break).
- Competition from other AI models (Bard, Claude).
- Users losing interest due to the learning curve associated with effective usage.
Revolutionary Potential of AI in Education
- AI as a general-purpose technology similar to the Internet or steam power.
- ChatGPT perceived as more impactful in professional settings than in casual consumer use.
- Use of AI to foster a universal tutoring system, allowing students to fill gaps in understanding where traditional teaching may fall short.
Cheating and Academic Integrity
- Discussion on the emergence of cheating facilitated by AI tools and the need for educational institutions to adapt.
- Proposal of a "homework apocalypse", where traditional homework methods may become obsolete.
- Suggestions on shifting focus from traditional homework methods to active learning and flipped classrooms.
Redefining Educational Practices
- Emphasis on moving away from traditional lecture-style teaching towards a model that leverages AI for enhanced student engagement.
- Potential for classroom focus to shift towards hands-on activities and applications of knowledge.
AI's Impact on Work Culture
- The need for companies to adapt to AI's capabilities by redefining roles and expectations.
- Discussion on how AI could lead to job displacement but might also create opportunities for more meaningful work.
- The balance between increased efficiency and the potential loss of job satisfaction in a world of automated tasks.
Future of Work and Education
- Expertise Building: Emphasis on the importance of foundational knowledge to effectively collaborate with AI tools.
- The potential of AI to provide personalized tutoring, making education more accessible and meaningful.
Concerns about AI and Jobs
- The episode addresses anxieties surrounding job loss due to AI.
- While technological advancements often lead to job displacement, they also historically create new opportunities and job categories.
Key Takeaways
- Integration of AI: Essential for both education and workplace efficiency.
- Cheating Concerns: AI complicates traditional notions of academic integrity, necessitating a re-evaluation of what constitutes cheating.
- Educational Transformation: The future of education may lean towards active, hands-on learning approaches with AI support.
- Workplace Adaptation: Companies will need to navigate the balance between leveraging AI for efficiency and maintaining employee engagement.
Conclusion Ethan Mollick shares a forward-thinking perspective on how generative AI like ChatGPT can fundamentally alter educational practices and workplace dynamics, while also highlighting the challenges and ethical considerations that arise from such transformations. The episode encourages educators and business leaders alike to embrace change and prepare for an evolving landscape shaped by AI.
---
Contact and Feedback
- For questions or feedback, reach out via email: [bigtechnologypodcast@gmail.com](mailto:bigtechnologypodcast@gmail.com)
- For weekly updates, subscribe to the podcast newsletter on LinkedIn: [Big Technology Podcast Newsletter](https://www.linkedin.com/newsletters/6901970121829801984/)
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Transcript
Automatic transcript. May contain errors.0:00An author and a Wharton professor speaks with us about AI's application in schools and the workplace and talks through the latest in artificial intelligence research. My conversation with Professor Ethan Malek, coming up right after this. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge, and it's simplifying car shopping. Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade and value.
0:38Advanced, intuitive, and deployed. That's how they stack. That's technology at Capital One. The truth is AI security is identity security. An AI agent isn't just a piece of code. It's a first-class citizen in your digital ecosystem, and it needs to be treated like one. That's why Okta is taking the lead to secure these AI agents. the key to unlocking this new layer of protection, and identity security fabric. Organizations need a unified, comprehensive approach that protects every identity, human or machine, with consistent policies and oversight. Don't wait for a security incident to realize your AI agents are a massive blind spot.
1:14Learn how Okta's identity security fabric can help you secure the next generation of identities, including your AI agents. Visit Okta.com. That's O-K-T-A dot com. LinkedIn presents.
1:31Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. Ethan Malek is here with us today. He's a professor at the Wharton School at the University of Pennsylvania. He writes one useful thing on Substack. It's a great newsletter. You should subscribe. I've been looking forward to this conversation for a long time. Ethan, welcome to the show. I'm thrilled to be here. Thanks for having me. Great to have you. I think in terms of the people who have actually gone and used AI and written about AI and its applications in school and at work, I don't think there's anyone as prolific as you, anyone who's built an audience and an expertise like you have.
2:10So I'm thrilled to be able to speak with you about this. One of the things that I've noticed is that interest in ChatGPT seems to be dropping off your professor. Is that because kids are out of school? So, yeah, I mean, there's this sort of stat that everyone's sort of talking about that the numbers are dropped. And it's any of a number of things, right? It's entirely possible that some set of people are losing interest, right? It's not an easy system to use. And if you're using it for fun, you bounce off it pretty quick and move on. It's very possible that it's a big engine for cheating. Schools is out of session.
2:41That might be the cause of the drop. The third is there's now actual competition with chat GPT. So maybe people are using Bing or Claude or Bard. So it's hard to kind of make a big jump at this. but I don't think it matters very much because I think you're going to see some churn. Not everyone's going to find this useful right away. That's not really the point. But I have been thinking about it and it's like, okay, not everybody's going to find it useful, but some people are talking about this as a revolutionary thing that's going to change everything. There's certainly notes of that in your writing in terms of the way that it's going to change education.
3:09So how do you square that with the fact that something that can be so revolutionary can lose some adoption or have the adoption tail off early on? Well, I think, I mean, you have something that is many things at once. AI is a general purpose technology, right? It can do many different things. It can obviously, you know, create content and, you know, cheat on tests. But general purpose technology is something that comes around once in a generation. Think the internet or steam power and has huge impacts on all aspects of productivity and work. So ChatGPT is just the sharpest edge of GPT and general AI use across the many different domains.
3:48So if a lot of people are playing with it as a consumer, finding it boring and moving on, I don't think that's a big deal because I think the consumer use is the least interesting. I think the question is, okay, what about people adopting this stuff for real work? Are people building wrappers around this and using the API instead? Like I don't think it – like the actual use is going to change over time. and I think that how many people are downloading the app or playing with the website is probably less important than we have a sort of technology shock overall. Okay, so let's talk about one of those actual uses, which is schoolwork.
4:19So, I mean, I visited a university. I went back to Cornell where I went to school sometime, you know, over the past semester and just kind of was sitting at the lunchroom asking kids, hey, how are you using this stuff? And it really was unbelievable that every single one of them had heard of it, was using it. they were using it in ways that I didn't really anticipate. They didn't understand concepts that were taught to them in class. So they were going to the LLMs and getting them to teach them in the gaps where the professors weren't able to cover. How have you seen it actually change the way that education is working so far?
4:52And what is the vibe among you and your counterparts about how this is going to change your jobs in terms of educating students? So I think you're right. I mean, everybody goes to cheating first. And that is important and interesting, but in some ways, the least important and interesting. We'll deal with the cheating problem, right? There are ways of solving that. They aren't by detecting AI writing, by the way, which is impossible, but there are ways of dealing with the cheating problem. But the really profound thing is, like you said, using this as a universal tutor, using this as a way of creating new ways to teach new exercises.
5:24I've made AI required in all my classes, my entrepreneurship classes, and like work and entrepreneurial classes have done pretty well. I think out of the intro classes that I know that people teach, people have raised like$2 billion in capital over the last decade or so, you know, plus. Like they do a lot really well. Now they're doing even more. I actually ask people to do impossible things, but there's tons of different use cases. So explaining things, explaining why you got things wrong, people building demos with it, getting feedback. You know, so it's beyond just cheating. But I also think it fundamentally changes work.
5:53People are going to be cheating on essays. People are going to be cheating on problem assignments. All of homework is threatened, right? We're facing what I call the homework apocalypse. But I think we could build a better world afterwards. Yeah, there were other interesting applications that I hear now that I'm thinking about it. There are law students, and I guess this might be in the cheating bucket, but it is pretty resourceful. Law students who are on call in their law school class that are basically chatting with Bing to make sure that they have the answers on specific cases. So how do you deal with that as an educator?
6:22I saw the same thing. You put a Harvard Business School case into it. and if I type into and I just say, tell me what I should say to sound really smart, crack the case, it will definitely do that. So it invalidates whole ranges of how we used to do work, right? I don't think we can do assignments the same way anymore. Now, the plus side is we've known for quite a while that the way we were teaching classes was not the ideal way to do it. The ideal way to teach is actually using what's called active learning and flipped classrooms. So that's where you do your learning outside of class and you do activities and applications inside of class.
6:55So lectures are generally a waste of time for the way most people do them compared to actually doing things. So what I hope to see happen is less emphasis on homework as a way of practice and more of the homework outside of class is the initial learning and the practice happens in class. But that is a radical change from how we used to work, how we used to teach. And people aren't ready for it, right? As you said, like no one's ready for what's about to happen. Essays don't work the same way anymore. Problem sets don't work the same way. In-class conversations don't work the same way. We have to rebuild around these new technologies.
7:30Right. And long-term, maybe we will. But short-term, as you mentioned, this is really going to be an issue. I'm just going to quote from your piece, The Homework Apocalypse. So first of all, it really is astonishing to me, A, how much the internet changed the way that people do homework. So you write, one study of 11 years of college courses found that when students did their homework in 2008, it improved test grades for 86 % of them. but only helped 45 % of students in 2017. And that's because half of the students were looking up homework assignments and answers on the internet in 2017. So if we expand that, doesn't that in the short term, at least before we go to flip classrooms, just totally destroy everything we're trying to do in education?
8:12I mean, people are already cheating. There's 20 ,000 people, at least as of last year, before Chachi V came out, who's in Kenya's full-time job is writing essays, right? Like this has been a longstanding thing. We've just kind of ignored it. So you can't ignore it anymore. And also a different kind of cheating is enabled. And also what cheating is changed. Is it cheating if I ask ChatGPT to help me come with an outline, but I don't use that outline? Is it cheating if I ask for feedback on an assignment? Is it cheating if I ask it for 10 different ways to write a sentence when I'm stuck? We have to redefine how cheating works.
8:43We have to redefine how plagiarism works. Like this is not an easy sort of first step. So it's a shock. It happened all at once. And, you know, as you said, school districts aren't ready. I think in the long term, we emerge much stronger. In the short term, we're going to see people doing all sorts of crazy stuff. I mean, they already are trying to use anti-cheating tools, which don't work. But I expect people to be going back to filling out Blue Book exams as an interim solution. I expect people to be given as homework assignments where people are scrutinized and quizzed about whether or not they answered things.
9:12People will be asked to use Google Docs, and teachers will go over line by line to make sure that they are entering them at the right time because you can look at track changes. like there will be all kinds of crazy stuff that isn't helpful at first as a stopgap measure until we figure out how to do this right so i think we'd both agree that like looking things up on the internet isn't cheating when you have homework right and you're i mean assuming the professor hasn't told them don't look stuff up on the internet for this right i think that's normal and that's already decreased like the recall that's cheating yeah if you look up a problem set if i assign you a problem set i'm grading you okay that is a cheat that's cheating yeah That's cheating.
9:50Doing internet research isn't cheating for writing an essay, obviously, but also, you know, but that homework assignment was about looking stuff up on the internet that are exact answers, right? It was about searching, you know, Chegg or whatever to find out the right answer to a problem. And that's really become the issue. But you couldn't do that with essays, right? Unless you paid someone, which was kind of really cheating. But now you can, right? And the AI will generate essays for you. And by the way, it has vision. So show it your geometry problem and it'll solve it. Like that's a pretty interesting world.
10:19Are your colleagues coming to you and be like, I mean, obviously they're looking at you as a guy that understands this. Are they coming to you and they say, I busted a student using chat GPT in an essay because here's the word as an AI assistant, I cannot or I can. What do you think about that? Has that happened? I mean, look, at Wharton, I'm not seeing much bad cheating. I did have one person at one point during my graded where their assignment switched fonts partway through to the Wikipedia font, and that was a dead giveaway. So there's bad cheating out there. But I actually assign my students to cheat in class.
10:52So it's part of the way of learning what large language models operate. and by the time what I tell them to do is fake an essay about a personal experience, reflecting back on a personal experience and applying a class concept to it, very standard reflection is actually a very powerful tool. Applying class knowledge to a real thing is a very powerful tool. So it's a perfectly good assignment, but I asked them to fake it and they have to use at least five different prompts to tell me what happened. And I will tell you, by the time you prompt the AI four or five times with, and there's multiple techniques, we can talk about that with prompting.
11:21By the time you've prompted the AI four or five times, the results are really good. and no AI detector detects them. And they don't feel like, they don't say as a large language model and they don't end with in conclusion and they don't have the ChatGPT style lists of things that go on in them because you can make it much more human writing. So go a little bit deeper into this idea behind assigning students to use AI in class. Does that, how does that get across some of the learning objectives that you have? And do you think that's going to become more prominent? Well, there's about three different ways of using AI in class, right?
11:52So the surface stuff I gave you is that, The surface level stuff is you do an AI assignment, focused on AI and learning AI. And that's great. I mean, I'm a fairly advanced user of AI. I get to teach my students that. And so that assignment has a reason. Then the second level of assignment is letting people use AI and taking advantage of its strengths and weaknesses. So ask them to do a report and they use AI and they have to critique the AI's answer. And that actually can be very helpful. The third level of AI is integrating AI deeply into everything you do. And that lets you do the impossible.
12:26My syllabus, I used to ask people at the beginning of my class, they had to do a class outline where they would outline the business idea they had. Now, the outline assignment insists that on top of doing an outline, they have to do at least one impossible thing. If they can't code, they have to code. If they can't do HTML, they have to do HTML. If they can't draw, they have to do drawing. And every assignment they turn in has to be critiqued by at least three famous entrepreneurs through history, which they then use GPT to give them perspective. because getting outside perspectives matters a lot.
12:52And I expect them to use AI for feedback. And now they do tons of more work than they did before and get far much further than they did before. So there are different levels of embracing AI for a school. So much of the problem with our education system seems to be, and this is speaking from educators, that it is memorize and spit back. I guess that's why the lecture homework system doesn't work very well because people sit and lecture, they memorize what the professor said, they come to the final and they write that down or they type it down and then spit it back and their ability to retain what they heard is sort of evaluated.
13:25Do you think that these things, these LLMs and the being integrated into classwork can actually help create more of the critical thinking and more of the inventiveness that we need to see in students and in the education system moving forward? So I'm going to actually push back on you a little bit. I actually think, so I think we need to learn more facts and that sounds horrible, but it turns out - I'm shocked. Okay, go ahead. All right, so here's why, okay? Because in order to be able to beat large language models or at least to be able to work with them and who knows how good they're going to get, really, we could talk more about that.
14:00You need to be an expert. Like if you were in the 50th percentile, that's not good enough. You need to be an expert in something to be able to monitor the LLM, use it well, be a cyborg that it helps you out with. And we actually know how to build expertise. And unfortunately, there is no shortcut. Like people want to teach critical thinking as if that's some sort of magical shortcut over what we're doing. It's not. The way you build expertise is unfortunately horribly grinding. It starts with having a deep basis of facts. You need facts. And then you need to start to see the connections between those facts, which let you move the facts from your short-term memory and your working memory to your long-term memory.
14:33Then you need deliberate practice, right? You probably heard the 10 ,000-hour thing. That's not right. It's not 10 ,000 hours. But you do need vast amounts of grinding deliberate practice that push you really hard to apply those facts in different ways. That's how you become a chess master, right? You study a whole bunch of moves and games, and then you play enough against increasingly hard opponents and learn from your mistakes until you get better. So you get good at tennis. That's how you get good at being a carpenter. That's how you get good at being a professor. So we can't skip the facts phase because that's how humans build expertise.
15:03Regurgitating is spitting back facts. So you start to see the connections between, okay, and you're making tons of them here in the room, right? Between all of these different topics, you're an expert in technology. That requires you to know a lot of facts about technology that you can pull together and and throw back out in useful ways. Being an interviewer requires knowing a lot of facts. So we can't skip that phase. We can't outsource that. And so in some ways, the AI requires us to actually double down on the basic knowledge because we can only build the advanced knowledge from the basic knowledge.
15:31Interesting. So then what do you think the future of education is going to look like? I think what makes building expertise hard is that process of deliberate practice is hard. It requires coaching. It requires instruction is mixed in. You have to get lessons. It has to increase the difficulty level on a constant basis. It has to be as engaging as possible. You have to be utterly focused on it. These are things that's really hard to do in a classroom, really hard to do. So we sort of outsource it to a bunch of homework and a lot of grades of doing the same stuff over and over again. And then you start to work as a junior producer in podcasting, and you work your way out, you do crap work, and then you work your way up to mid-level.
16:10that eventually you get good at. Like we depend on all of these systems to do it. AI might be able to let us skip these things and make expertise easier. A tutor, and by the way, if you look at Khan Academy's Comigo, you can start to see the direction here. A tutor that actually knows where you need to go and makes sure the work is always just hard enough without getting frustrating. That meets you at your level, that gives you feedback and helps you through the process of deliberate practice. That's an exciting potential future, right? So I think we can get better at building human knowledge and expertise.
16:36We know how to do it. We just, the most effective way to teach is what's called direct instruction and one-on-one tutoring, right? Like literally the idea of a one-on-one tutor, people who get one-on-one tutoring act in the 98th percentile of a class. Like it's amazing. But one-on-one tutoring is incredibly expensive. AI can do that. So I think we're looking at a world that unlocks that. So do you think that this is going to turn to AIs? Well, I think part of the teacher will turn to AI. I think the tutoring instruction will be AI-based. I think you're still going to use classrooms because we need to work with each other.
17:05We need to apply our knowledge. We need to get feedback from what mistakes other people are making. So that's that flipped classroom idea. In class, you'll be doing exercises, activities. Some will be AI driven, but the teacher will still play a vital role. And they'll be able to provide guidance, support, all the other things you need in a classroom. And outside of class, the instruction that used to be delivered through textbooks and through lectures is going to be delivered through AI. Last question about education. Look, the education system, you know this well, moves extremely slowly. It's not a fast moving system and everybody inside it, well, not everyone, but almost everyone seems to be extremely resistant to change.
17:39So I'm hearing that, you know, these ideas from you and they seem to make sense. But then if you ask me like realistically on what time horizon this stuff is going to happen, 20 years, I mean, what do you think the actual speed of change that is realistic for us to expect? Well, that goes back to that dropout on GPT downloads that you saw before, right? systems are slow right and people overestimate how quickly change happens but systems resist change because it's not just one thing you can't just make one thing different because the rest of the system needs to operate too the same thing will happen with jobs by the way so the education system is slow it's slow for a reason because there's a lot of interlocking pieces like you can't just advance things as one teacher because you need to fit into a class system and you need to be able to fit into getting creative.
18:27The kids need to get able to get to college and you need to know what they're doing and hold them accountable. Like there's reasons for this grinding slowness. It's not just bureaucracy, right? It's that too, but it's all these interlocking systems in place and it has to fit into teacher tenure and, you know, and what parents expect. There's lots of pieces, curriculum. So those are going to take a very long time to change. I think that you'd be surprised at how quickly people can move when it's radical. I mean, schools moved imperfectly online with two days notice. It's insane. I think COVID showed that we can adapt quickly when we need to, but I also think it overestimates the need for systemic change and how much systemic change needs to be just in US school systems.
19:06The exciting thing about something like Bing is it's available in 169 countries around the world. If you are in Botswana right now, you have access to the same GPT-4 that you can get access to as BlackRock. There's no difference. And if I give you the right two paragraphs or prompts, you can learn something new. So I think that we're underestimating the global change on this. I think you're right. Things aren't going to change overnight, but I think there is a capacity to do it. Does that mean all these predictions? And so let's shift to the workplace now. I mean, it does take time to do this.
19:39And workplaces are also slow to change, even though there might be a little bit more nimble than educational institutions or government. Do you think that because it takes this amount of time to adapt and change that all these rumors, all these, you know, this panic, oh, AI is going to take my job might be a little bit overblown right now. So I don't, I think that's part of the reason. So to talk about AI taking my job, we need to get a little bit academic, which is we don't like to think about jobs in academia. I mean, we do, obviously we have jobs, right? We think about the tasks, the bundle of work that you do, and we think about the systems that your work is in, right?
20:14And so we can talk more about tasks a second. And it seems like you want to have that, you know, we'll talk about that. But it does matter, right, that you're part of a system. And that does kind of, but on the other hand, task adoption is very quickly, you know, you can easily change a task. And because individuals are incentivized to do action and because the productivity performance impacts are so potentially huge, like 30 to 80 percent performance improvements in some tasks, that creates a huge incentive for companies to shift quickly. So I think we'll see faster movement. I'm already seeing companies do faster things.
20:48Part of it is that the APIs that you use to access GPT-4 are kind of so easy to do that you can kind of roll your own solution remarkably quickly. Yeah, one of the things that really surprised me in reading your writing is that it seems like this change is actually being driven more by the individuals than the companies themselves. And there are people, you did a poll asking, if you use AI at work, do you tell anybody? And 50 % of people were like, no. So what do you make of this idea that this is being driven by the bottom up versus the top down? In fact, many companies are banning AI, which is quite interesting.
21:22And if you talk to people, the companies that banned AI, they're all bringing their phones to work and doing all the work on their phones and then emailing themselves, right? I mean, let's go back to cheating, right? Like in school, cheating is bad. Shortcuts are bad. At work, if you can figure out a way to do all your work in 10 % of the time, you are going to do that. Humans are exquisitely built to respond to economic incentives. So if I set it up so that you could do more work and less time and less effort and you can outsource boring tasks, you will do that very quickly. And so we're seeing people experiment all the time.
Read the full transcript
21:51Experimenting with companies is expensive. Experimenting with your own jobs is super cheap, right? You try a chat, it doesn't work. Try it again, it doesn't work. Maybe you give up. You try, it starts to be interesting. You keep going. The trial and error is cheap for your own tasks, expensive for other people's. So it's very easy to adopt to the individual level. The benefits are huge individually as long as no one knows you're using it. And I talk to people all the time who outsource 90 % of their work and just don't tell anyone about it. What happens when people find out? Well, so I think that's the real question.
22:25If you're a smart company, you're going to be working really hard to make sure the answer to that is they get richly rewarded, right? Because what I want to do is find out what everybody's been experimenting on. All these companies are used to doing experiments, to doing change from the top down, as you said. They're used to having a boss do things. They have an innovation group. They hire McKinsey. That's not the way that this works. There's no reason McKinsey knows more about how to automate some mid-level manager's job than they do. So I need to get that middle-level manager willing to talk about what they're doing, to share that information with me.
22:54And to do that, I need to make sure that middle manager doesn't feel like they're going to get fired. I need to make sure that mid-level manager knows they'll get rewarded, that it won't fire other people because they've ratted out this capability. I want to make sure that people feel incentivized to be part of the team. So this is where having a hostile work environment where you hate the boss, but are grinding away at work. I'm never going to share anything. If I'm for a company where I'm all in on that company, maybe that's a little bit different. Right. And so it sort of goes to this question of like, all right, so what's going to happen to my job?
23:22It's like, actually, if you can take this, these tools and make yourself more productive, you become more valuable to the company. And I think that like one of the companies really never talk about like, ah, like, you know, we wish we did the same thing with less people. They always talk about, we wish we did more. we just have a labor constraint or a cost constraint. And this could potentially flip that. Is that how you see it? Well, I mean, there's lots of different constraints, right? Like what you might be, lots of companies also fire people, right? Because I mean, let's be realistic. Like if you get a productivity gain, that's what people tend to do, right?
23:54Is like, or they view that there's overhead, they slash it. But if you can figure out ways to turn that extra overhang of people and productivity into positive growth, your company will be much more successful, right? And if you're firing people left and right, when they become more efficient, they're going to stop becoming more efficient because they know what happens. So you have to embrace the idea that there might be some short-term inefficiency to get this long-term gain. But it does tweak the employment contract just a bit, don't you think? Because you get paid to do a job. The contract is basically, you're doing this job.
24:28This is what I think someone on your level is able to do. then you find a way to be you know much more efficient and then all of a sudden it's like you know it it really does you know say basically as the technology has shifted everything there and it's no longer like it's it's almost a i don't want to say dishonest but it's a completely different way of approaching the job if someone can do it like shouldn't shouldn't i mean yeah okay you're shaking your head no i'm agreeing with you actually i i think i think i i i think that you're right that this is completely shattering but every i mean this gpt a general purpose technology breaks things right like this is like this is like old systems are going to get broken everywhere and we have to reconsider them what does it mean to work with an ai like what how much am i a free agent working with an ai to help a company how much does my employment contract matter if i get all my work done in an hour and a half do i get the extra seven and a half hours i'm you know seven hours on my own like or six and a half hours on my own if you're a banker 16 hours on my own.
25:26Like what happens, right? We don't know. What happens when like a lot of our systems at work are built around human limitations. They're built around enforcing limits on human, like keeping everyone operating at the same pace, making sure you're gearing the machine. What happens when that breaks? We don't know. Like this is, you know, it's very funny. People talk about the singularity, right? Is this thing that is like the AI gets smarter than us and murders us all. And I'm definitely think, you know, that seems worth worrying about, But it's also, the original meaning is it's like a mathematical point that we can't predict what happens afterwards.
25:59That's already going to happen at work. I don't know 100 % what the future of work is going to look like. I don't know 100 % what the future of education looks like. Because we're just assuming that everything stays the way it is, right? What happens is work gets more and more easy to automate with AI. What kinds of categories of work change? Like, we're barely scratching the surface. Let's go through a scenario that sort of hints to what you've been discussing. So I am a manager. somebody comes to me and says, listen, I read Ethan's sub stack and I'm confessing I'm using AI. You shouldn't fire me.
26:28Here's the article. This is why, by the way, I'm doing a hundred percent of my work in 10 % of the time. So does the, what is the man, what is the next step for the manager there? Do they say we actually need you to do 10 times the work in a hundred percent of the time, or do they say, yep, that's good. Teach us how to do it. And we're going to have basically give everybody, put everybody on the two day work week. So there's a whole bunch of options, right? I I mean, no, I'm serious. So one option is something we already know works really well, which is let's job craft this. People are more motivated to do better jobs when they go through job crafting, which means working together with managers to figure out what their job is.
27:03So you could say, hey, what outsourced stuff have you done? The 10 % that's left, do you find that engaging? Is that what you really want to do? Because a lot of the early work of AI is about freeing us from drudgery, which is kind of the good, it's ominous in the longer term, but the short term, it looks pretty good, right? Like my job is a bundle of tasks that includes many things like philotic expense reports. If AI does that, yes. Like that's the best thing in the universe. I hate expense reports. Right. And so you could ask this person, what have you outsourced? How do we get you to do more of that 10 %?
27:29Maybe we're giving you a raise because you're doing 10 times the work now. Maybe we're giving you a million dollar bonus because we can spread that idea across the entire company. We're saving that much. Extravagant rewards, rebalancing work towards what people want to do. Right. Thinking about what systems we can use to support that person. It becomes part of a conversation. Exactly. It's interesting. So I know we kind of danced around this question earlier about jobs, but I mean, like, so we don't know exactly what's going to happen there, but there has to be some sort of impact, don't you think?
28:02I mean, there's going to be, right? I mean, and again, general purpose technology, it's going to be a huge, like a huge generalized impact, right? It's going to affect everything in all kinds of different ways. So some jobs are going to change and disappear, right? And like when the telephone system was moved to a digital telephone system, it resulted in a giant wave of change. A lot of people lost their jobs. A lot of things ended up happening very quickly. But most times when technology change happens, people get better jobs and higher paying jobs and the nature of work changes. But we tend to look back on those historically as not in the moment, which is where we're going through now.
28:41So there's gonna be disruption everywhere in the nature of jobs. Yeah, and it's also, there's something that you hinted at that I think we should expand upon, which is that taking away the drudgery is really nice in the short term, but ominous in the long term. I mean, after reading your work, I didn't really fully think about how this could potentially change like the nature of meaning that people get out of their work. And you had this one line that's just amazing. You're talking about, I think, letters of recommendations. And you start to say that we can create documents mostly with AI that get sent to AI-powered inboxes where the recipients respond mostly with AI.
29:18Even worse, we still create the reports by hand, but realize that no human is actually reading them. We can view the destruction of busy work as freeing. and we do not yet have to start our, oh, this is about setting our time on fire as a signal. But like, let's go back to this idea of the AI's community. I mean, this is one of these things where like, you know, it seems like maybe one day instead of us getting on the podcast, my AI will interview your AI and it will be listened to by AI users who will summarize it for people. What is that? How does that change our society? How does that change work?
29:52I mean, we have to reconstruct meaning, right? This is the biggest crisis. I think that everything else is secondary. I am sending a report. As a mail manager, the report that I would write about my employees' work, even if no one reads it, was valuable because it means that I checked the work, right? And then my boss looked at this and saw that I was doing this and is like, okay, the systems of monitoring are in place. Now I hit a button in Google Docs. It creates an automated report that looks plausible. I send it over Gmail with a fake message to my boss, who then hits on a reply and gets a message back.
30:28We've just taken the intellectual meaning, but also the reason for doing this work, and it's disappeared entirely. We have to reconstruct this. This means our work systems were built. The reason we have org charts is because in the 1850s, somebody wanted to make sure we had org charts. early railroad barons, wanted to build a structure that would let them control vast geographic train networks from the top. So they built systems that look like train networks that do that. Henry Ford realized that if he hired low-paid employees, he couldn't monitor and control them and what they were doing. But if they did one task over and over again, he could keep an eye on it.
31:07So thus assembly lines. Agile development is all about the idea that we could track bugs and changes in the internet, but we still need stand-up meetings to coordinate work. All of those things are going to change as a result of AI being able to coordinate, to interact, to change, you know, and that's going to change work. The companies that figure this out first are going to win. It's almost a chicken and egg thing, right? Where it's like, okay, the AI is changing work, but it's also commentary on how much of our work is meaningless, don't you think? I mean, you, in your essay, write about how, and this is what I was getting at with setting the time on fire, which is a concept that you've talked about.
31:41But basically, people are asked to write letters of recommendation, not because they're actually going to read it, but just to kind of show that the professor would put the time in to recommend the student. And it's a signal. And it just seems that if you can automate this and have AIs talking to AI, so much of the system of work today is kind of garbage, don't you think? Yeah. I mean, look, work is broken in lots of ways. Literally, as a management scholar, one of the dominant methods of understanding how organizations work is called the garbage pail method, which is just like stuff happens, right?
32:13Like people meet each other, it all gets tossed together and stuff happens. AI can help a lot, right? It can help free us from drudgery, from bureaucracy. But there's a lot of people who work in bureaucracy. There's a lot of people who, you know, there are reasons part of those systems exist and part of them just grew that way. And, you know, and we have to reconstruct all of how that works. As you were saying, the problem with the letter of recommendation is not just that I can push a button and write it, but that the letter of recommendation that I push the button and write is going to be better than the letter of recommendation I spend an hour writing, because the AI will read all the documentation, all the material, and be able to produce something that's much better.
32:44So the problem is I turn in the old-fashioned school, properly done, morally correct letter of recommendation, and I'm actually hurting my students' chance of getting the job compared to pushing the button. And we don't have things built around that. I mean, Do I just send the prompt I would have sent? Like where I say, I would have told the AI, I like this person, they're a good job, give them a good recommendation. Or what do I do? We don't know. But that's one minor system out of so many that's going to be broken. It does seem, so there's been this question of like whether AI will automate all of our work and I can keep coming back to it.
33:19But it does seem at a certain point, like the further we get into this, we could effectively just like set it and forget it with our economy. And maybe I'm kind of delusional, But it does seem like we'll still make work for ourselves. Like, why don't we just like have the robots kind of take care of our basic needs and just kind of live? Well, I mean, but that, of course, is the bet, right? We do like we're coming back to the billion dollar question, the trillion dollar question, actually, I guess, or what are the companies? What, like seven trillion, 12 trillion? I don't remember off the top of my head.
33:47That's the question, right? The question is, first of all, how good does this get, right? Like, there's too much obsession. not that it's wrong about like whether we build artificial general intelligence and a machine god totally get it i'm glad people are worried about it they absolutely should be government should be worried about it we should all have concerns about you know whether the machine god will save or kill us but like there's a lot of steps between there and now and the real salient question for technology is is this as good as it's going to get i think the answer is probably no will change be linear or exponential in the future that we don't know if change is exponential then maybe the robots take caring of everything is something that happens the next five or ten years If not, if it just keeps getting better, we have a gradual squeeze.
34:28If you're not in the top 10 % of workers in your task, you're probably, AI will replace you. Then top 8%, top 7%. So I think the question about what happens with our leisure time, what happens with work, depends, like right now, if AI were to pause where it is today, I think it would be 10 years of us absorbing the impact, but I think we'd be fine, right? Like it clearly needs oversight. It does not, there are very few categories of workers that completely replaces. It tends to do drudge work, right? I think we'd be fine. Now, what happens if it gets 10 times better? I don't know. Then we start to think about how work gets automated and changed in much more profound ways.
35:07We have abundant resources on this planet and in the modern world that we've developed, but we still fight over them. Like, it just seems so ridiculous in some ways that we're getting this technology and we're still going to fight over them. I mean, it's going to help. I mean, the question is whether it makes us fight worse. you know is is the is a real question right right i mean look it is hard it's hard when new technologies come around everyone gets utopian and uh and cataclysmic at the same time right and so this is the like but this is something quantifiably different the question is just whether it keeps going right where we have it right now it is a complete change in technology and it's very exciting and it will be a huge revolution on par with the internet or maybe steam power.
35:51But if it gets 20 times better, then it starts to get really weird, right? Because then we actually have, you know, and if it starts to be able to outperform humans at every task, that starts to get very strange. And we don't know what happens then. We don't know what the limits are and we don't know the outcomes. And that's where like the entire sort of system modifies. Right. Because if you talk about like, all right, the technology gets good enough that you only need the top 10 % of workers and 90 % you don't need, well, you still need workers. and you're not at like maximum production and that drives inequality in society and destabilizes it.
36:23That's what you mean by the system shifting. Potentially, right? I mean, at the very least, you need government intervention to make things happen. But, you know, again, if we look at the history of work, change will be slower than we think, generally. And people generally tend to get better jobs. We just don't know if this is the end of those rules, right? The historical rule has been technological change every other time it's happened has resulted in generally an increase in GDP, an increase in quality of work. There's been some exceptions. Will that keep happening is the big question. Ethan Malik is here with us.
36:58He is a Wharton professor and he writes the One Useful Thing newsletter on Substack. On the back end of this break, we're going to talk about a few new models and a bit of a lightning round and take some questions that popped up on Twitter and threads over the past couple of days. All right, back right after this. These days, it feels like every dollar should be working a little harder, but figuring out where to put your cash can be confusing. That's where Wealthfront comes in. Wealthfront is a tech-driven financial platform built to help you grow your savings into long-term wealth. Their high-yield cash account through program banks offers a 3.5 % APY on your uninvested cash as of November 7, 2025.
37:35And there are no monthly fees, no minimum or maximum balance to earn that rate. and you can even make free instant withdrawals to eligible accounts in just minutes, 24-7. So your money can always be within reach. Right now, Wealthfront is offering new clients an extra 0.65 % APY over the base rate for three months on up to a$150 ,000 balance. That's a total of 4.15 % variable APY when you open your first cash account. Go to wealthfront.com slash big tech to sign up today. This is a paid testimonial for Wealthfront. It may not reflect the experience of others and there's no guarantee of future performance or success.
38:08Wealthfront brokerage is not a bank. Rate is subject to change. Promo terms and conditions apply. For more information, see the episode description. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge, and it's simplifying car shopping. Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade in value. Advanced, intuitive, and deployed. That's how they stack. That's technology at Capital One.
38:51And we're back here with Ethan Malik. He's a professor at the Wharton School of University of Pennsylvania. He writes the great one useful thing on Substack. It's a great newsletter. We've been talking a lot about his most recent pieces here. Let's do a quick lightning round. So, Ethan, you've been experimenting with some of the bots that have been coming out from the non-OpenAI, non-Microsoft groups. So I'm going to ask you about your experience with two of them, first of all. So what is your experience with Claude from Anthropic and how do you find it different from OpenAI? So Claude was developed by people at OpenAI and they said they left because they were worried about OpenAI's risks it was taking with its AI.
39:30So it's supposed to be a friendlier and less harmful AI. They just released Claude version two, which is somewhere between GPT 3.5 and GPT 4. So it's a really good model, probably the second best model out there right now. And one of its really great capabilities that it has in the short term over anyone else is it has 100 ,000 token context window, which basically means it can hold a book, a short book in memory. And that means you can upload PDFs to it and it's very good at working with documents. So it's very good at summarizing and annotating at combining documents. I find it very useful for those purposes.
40:04Because it's safer, it also is a little bit preachier, so it can be a little bit more annoying to work with. But it's a very powerful model. And if you're working with documents, I'd strongly recommend trying it. Yeah, you have this chart of breaking down all the different models. And I think for Claude, it's like, good, but a bit too preachy. Exactly. They can be that way. Well, and I don't really mind some of the others because without guardrails, AI gets really bad really quickly. But Claude could get a little over the top. You mentioned that anthropic researchers are dreading what they're building, which is really strange, right?
40:36It's like they're going ahead. This is covered in the New York Times article, but they're going ahead and they're really freaking out about the nature of AI. And this is a question I ask sometimes, but it's just so unbelievable to me that some of the people with the biggest concerns about AI are like, man, this stuff can really destroy society. Let's build the next new model. What's happening there? There are three theories, right? Theory number one is it's all cynicism. I don't think that's the case. I mean, some people are being cynical and marketing it, right? Or paying lips, but that's one option.
41:06The second is that they are genuinely worried, but not that worried, right? So they have to say that they believe in possible doom, but they really think it's going to be okay, which a lot of researchers do. And a lot of researchers are also very worried. The third option is to take what they're saying at face value, which is that they actually think that if they don't build a safe version, people will build the dangerous version. That's always a little bit weird to start an arms race that like, if we don't burn down the village first, someone else will burn it down is sort of a disturbing viewpoint.
41:37But that may be a serious kind of approach, right? But they may think that they could shape the future that way. So it's some combination of those things, right? I do believe talking to people at AI labs, many of them are true believers. They really believe they're building super intelligence. Whether or not they're right or wrong, I don't think anyone really knows. What do you think about this Pi bot? It's a Reid Hoffman's bot, right? It's supposed to be a very friendly bot. I started talking to it a little bit. It can even like play like, it gives you an option to be like a therapy role, which was interesting.
42:08I was like, I'm about to confess my deep, dark secrets. And it's like, your secrets are safe with me. What's up with Pi? Pi has gone from driving me absolutely insane to intriguing me, right? So like it is, it vomits emojis. It tries to be a friend really hard. It will not do work to save your life. It's to save its life, which is something I push really hard on. But on the other hand, it is pretty good at chat. It's pretty good at adapting to you in the way AIs are. Pretty good at keeping chat going. If you have not tried the app, I strongly recommend it because there's the ability to get on a call with it, which is near real-time conversation back and forth, which is pretty nuts.
42:44So I admire the attempt to create an AI that doesn't do the thing the other chatbots do. Is there a market for it? Is it an open question? Is it like, is this good or bad for the world? Like high engagement bots are kind of a high risk also in their own weird way. So like it can be friendly, but it takes away time. But I've been increasingly impressed by its ability to be interesting to talk to at the very minimum. Yeah, it's definitely fun to speak with. Do you think we need all these? I mean, there's so many. There's ChatGPT, there's Claude, BingBot, Pi, Character AI. Like this stuff is going to consolidate eventually.
43:19it has to, right? I don't know. I don't know. I mean, I think the models could get specialized too, right? I mean, if we're talking about electricity providers or railway systems, having more than, I don't know how much consolidation there is. And I actually don't think for large language models, there aren't that many contenders, right? And the question, of course, is, is there a flywheel? Once you've built a large language model, right? Does that LLM give you an advantage? Does that help you code the next model? And so there's only a few companies that have significantly big large language models so far.
43:52And so OpenAI, which has Microsoft stuff built on top of it, there is Claude and Anthropic. There is the company behind Pi and there's Google with its bard. Those are the models. Elon Musk's announced a new model he's building, but it's going to be years of training to get that thing up and running. So the question is, are there really going to be that many? And will they specialize the way Pi is the conversation bot, AI is the document, you know, Claude is the document bot, and GPT is your general purpose tool. Right. Yeah, that's interesting. That's an interesting possibility. Do you think that prompt engineering is going to be a new job?
44:33I think that's kind of a ridiculous idea. I think everybody's going to learn how to do this. What's your perspective? I agree with you. And not only is everyone going to learn to do it, it's just going to be do it for you. If you already use something like, I mean, go back to Pi, you never prompt it once. It keeps a conversation going. If you ask GPT what I should ask, it will tell you what to ask. If you use an AI art system like MidJourney, it went from like when I did MidJourney prompts a year ago, those prompts would be these elaborate invocation spells, code of Chrome, art station seven. We just didn't know what we were doing.
45:07We'd throw everything into it to get a good picture. And now you just say, show me the thing. And it shows you the thing. I think that prompting is going to get easier. I don't know anyone who works in an AI lab who doesn't think that prompting is just going to get easier to the point where prompt engineering is kind of silly. Yeah, I agree. What are the US AI efforts look like in comparison to China? As an academic who studies this, I'm curious what you think. I'm not an expert on Chinese AI policy. The US is typical US stuff, which is laissez-faire while a thousand different court cases resolve themselves.
45:38And there are Senate hearings. It is going to be a slow process to figure out what all this means. Japan, for example, has said pre-training. The training data is, can it be protected by copyright, which is interesting. The U.S. hasn't taken a position on this. China, the early evidence seems to be a desire to regulate these systems. But on the other hand, national militaries are probably spinning up these. So I think regulatory systems are going to be really interesting. I think Europe is really being very careful and we'll see what happens with that. I mean, I think there's a lot of change happening all at once.
46:09Code Interpreter is just something that you've picked up on and different plugins that can help you make sense of data. What's your perspective on those? I think that that's some of the most exciting stuff out there, right? So yeah, say more about it, yeah. So Code Interpreter lets an AI, lets GPT, both take in data, give you data to download and also run its own Python. It gives it a Jupyter notebook, essentially, to do its own work in. And it turns out, increasing evidence suggests that when you give AI tools, it becomes much more capable. So given the ability to use tools, it first of all solves all the problems that have made AI really annoying to work with, right?
46:46So if you've ever tried to use AI to work with language, that is a hard thing to do, right? It doesn't understand sentences and paragraphs the way we do, right? If you ask it to count the number of words in a sentence, it can't do that. but code interpreter will write a little program to do that it'll solve math problems that it couldn't solve i write a little program and it starts to do really complex stuff like it will you throw data at it and it will apply theory and it will i give it a paper i give it a you know i did a little experiment where i um gave it the nba playoff data and had it find interesting stuff and it found all these cool hypotheses and then graphed it and then i threw in some work by um by tufty who's a famous graphic uh design infographic design person and said apply these skills to make the graph better.
47:29And it did. Like there's a really powerful democratization of coding and analytics that is already starting to happen. Do you have to know how to code to use them? I don't. I can't code in Python to save my life, but I do know stats. So I can look at the statistical outcomes, but no, I don't need to know code to do it. We got some questions from social media. So one person asked, how is accuracy necessary in terms of like, let me just read the actual question. Taka, ask him about the necessity of accuracy, whether it's a core need for enterprise use cases for LLMs. So should tools like Code Interpreter, here we go again, give companies greater confidence that LLMs won't hallucinate as often and provide reliably accurate outputs?
48:14So this idea of hallucination is a real problem, right? Yeah. I would argue that it's probably not as big a problem as people think. So first of all, hallucination rates are dropping. So we don't have a lot of comparative data, but there was a really cool study where they gave the variety of AIs the neuroscience board qualifying exam, neurosurgery board qualifying exam. And unsurprisingly, GPT passed the flying colors, but they tracked the hallucination rates. Bard hallucinated 44 % of the answers, GPT 3.5, chat GPT 22%, and GPT 4 only 2%. So hallucination rates do seem to be dropping. And then when you attach to code interpreter or data or PDF, it drops further.
48:52But hallucination is a genuine problem, which is why right now there's a huge advantage in that expertise thing we talked about. If I'm an expert, I can check over the results and fix problems. And I do need to do that with code interpreter. So I think that it's less of an issue. It favors experts. And also, there's a lot of jobs where accuracy isn't the number one thing. If you're doing marketing writing, it's not that big a deal. Even if you're doing customer service, a lot of the interaction, if there's small errors, they don't make as much of a difference as getting the big things right. So the other question is how accurate you need to be.
49:21For some applications, AI is totally out because it hallucinates, but for a lot more than you think, it works. There's some concern that people are, we're going to develop brain chips with AI inside them and attach them to our brains. And we could like have our data sucked out and stuff like that. What do you think about that? That feels that in a world where we're living in science fiction, that still feels too science fictional right now. So I don't think involuntary brain chips are in the cards and physical world and biology are much harder than software. So I think let's not get ahead of ourselves too much, I think.
49:55Would you get a brain chip if you could? No. I mean, no one's getting generation one brain chip. I mean, you know, and also like, you know, I think that, and I think when you look at what the actual brain development, you know, the chip stuff is, it's not telepathically communicating with AI. I think we need to be really careful about, you know, even a world where a lot of hype is coming true, there's still hucksterism going on. So there is no telepathic brain chip in the near future. We don't know how the, you know, we don't know how LLMs work. We don't know how humans brains work either. Like it's, we know how they work, LLMs work technically, but we don't know all the details of, you know, why a particular decision is made.
50:25I think we're going to be a long ways off from connecting AI directly to our brains. Yeah. Are there models outside of LLMs that you're looking at or interested in? I mean, so LLMs underlie a bunch of other things, right? So large language model and the transformer technology that powers it is what is powering art-based AI dialogue and, you know, podcast-creating AIs, you know, all of these things. So like the large language model technology is the transformer technology and attention mechanism that's at the heart of this technology are what's driving all of these sets of tech changes all at once.
51:00Okay. Last question for you. What are you looking forward to in the fall when students come back and how are things going to be different in terms of the way that you teach? Because you basically had to deal with this mid-stream this year. What does next year look like? I'm excited. I mean, I think like it's great to take the burden. It has allowed me to reimagine how teaching works in really exciting ways that I think will benefit my students and make things interesting. But that's for me. I think a lot of other people are terrified and or putting their heads in the sand or they try AI a bit or they've heard of AI.
51:30I mean, I talk to teachers all the time, we've heard of it, but haven't really tried it. Like, I think that we're in for sort of a mess. Yeah. Okay. I can't wait to watch. I hope we can keep in touch. Ethan Malik, thanks so much for joining. Thanks for having me. Thanks for being here. Thank you, everybody, for listening. Thank you, Nate Guatney, for handling the audio. LinkedIn for having me as part of your podcast network. And all you listeners, if you enjoyed, please hit five stars on Spotify or podcast. First time here, if you hit subscribe, that would be awesome. Thanks again to Ethan. I've been looking forward to this conversation for a long time and it delivered all the way through.
51:58So really appreciate you being here. All right, we'll see you next time on Big Technology Podcast.
From the publisher
Ethan Mollick is a professor at the University of Pennsylvania’s Wharton business school and writes One Useful Thing on Substack. He is one of the world’s foremost researchers on practical applications of generative AI and is an immensely engaging speaker. Professor Mollick joins Big Technology Podcast for a vibrant discussion of how AI changes schoolwork and office work, covering his decision to make his students use ChatGPT in class. Tune in for a insight-packed interview that will illuminate where the field is heading.
---
Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice.
For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/
Questions? Feedback? Write to: bigtechnologypodcast@gmail.com
Learn more about your ad choices. Visit megaphone.fm/adchoices


