Is AI Making Us Stupid?

0:00Hi, I'm Meryl Horn filling in for Wendy Zickerman. You're listening to Science Versus. This is the show that pits facts against frying our brains with AI.

0:15Today on the show, is AI destroying our ability to think? A lot of people are saying, yeah, it is. And they say that we've got the science to prove it. It's been medically verified that ChatGPT makes people stupider. ChatGPT users could not even remember what they wrote. They basically stopped thinking entirely. Every single time we give it a prompt, our own brain cells are burning. And if this is true, we might be in for a brain cell bonfire because a lot of us are using AI. One recent survey found that 62 % of adults in the U.S. say they interact with it at least several times a week. And AI really seems to be booming for students.

0:55A bunch of surveys suggest that around 80 % of high schoolers and college students say they're using it for schoolwork. And nobody's really worried about this. Obviously, I'm kidding. People are panicking. We have no idea how our students will ever learn anything or whether universities have any future. One professor graded papers and discovered every single one was AI-generated. The whole system is cooked. Cheating is the new major. Game over. It's just f***ing game over. But there is a flip side here, because some people say that AI is going to be good for us, that it can help us save time by breezing through busy work, get information for us, and accelerate scientific progress.

1:37It can do away with a lot of the really annoying, sort of tedious tasks. It's helped me boost my productivity by like 300%. AI is driving a transformation across all fields of science. Buckle up, tech enthusiasts. The future with AI is bright and it's happening now. Wait a minute. But buckle up, tech enthusiasts? That last one was AI. How did it get in there? But seriously, who is right here? Could AI help us be more productive and lead to a boom in science? Or is using AI the equivalent of shoving your brain into the microwave? So do we stop using AI so our hippocampus doesn't turn into a hot pocket?

2:18Buckle up, science enthusiasts, because when it comes to AI, a lot of people are saying... ChatGPT makes people stupider. But then there's science. And full disclosure, some AI companies do advertise on Science vs. Science vs. AI is coming up after the break.

2:47Welcome back. I'm Meryl Horn, and today we're going to look at whether using AI, stuff like ChatGPT, is bad for our brains. I have senior producer Rose Rimler here, Hi, Rose. Hi, Meryl. So you recently mentioned to me that you have been using chat GPT more and more these days. I guess that's true. Am I under oath? I'm curious, like, do you feel like it's changing how your brain is working? I do sometimes catch myself about to ask chat GPT something that I could do, and I'm just too lazy. Do you want to see what you've used ChatGPT for recently? I asked how many Roma tomatoes to equal 500 milliliters of pureed tomatoes.

3:35How to clean my velvet armchair I spilled milk on. I asked it to tell me if a couple of bottles of wine I had on Thanksgiving were dry or sweet. I think. Dry or juicy. Would you say it's like your go-to now for like just looking up info online? Yeah, it is kind of becoming my go-to. It is. For me also, and I've been like curious about what kind of problems this might lead to. So let's just address one thing first. We do need to acknowledge that one big problem with this sort of AI is that it can get stuff wrong. I actually did try to find some numbers on this. Like how often is it wrong? And when ChatGPT first came out, it looked pretty bad.

4:18Like a lot of the research was finding that it was only right like roughly half the time. Mm-hmm. I remember that from the early ChatGPT exploratory stuff. Yeah, it made up a lot of stuff. And now there are a couple of reviews which compared like ChatGPT 3.5 to ChatGPT 4. And they do find that it's gotten better. But it's still not 100 % accurate. So bottom line, it's like, it's just hard to tell whether it's telling you BS or not. Yeah, that's true. And so, yeah, I do feel like that's one way it could be making us stupid is just by like feeding us incorrect information. Yes, just like the government.

5:00Yeah. But now on top of that, there's this other fear that it's like bad for our brains to be using stuff like ChatTPT or other LLMs, large language models. Like if we let these things do a bunch of the thinking for us, then we'll lose our ability to think on our own or even be creative. So let's dive into all of that. Okay. And I want to start with this one study. It was done by Shiri Melumad. She's an associate professor of marketing at the University of Pennsylvania. And Shiri looks at how tech is changing us. When did you first get the idea to start looking into AI? Yeah, I mean, as a person who studies technology, it was difficult not to study AI, right?

5:42It's pretty prevalent at this point. So yeah, Shiri just published a huge study on AI. Altogether, it looked at more than 10 ,000 adults. And the goal was to see what happens when we use LLMs like ChatGPT to try to learn something new. Like, how does it compare to an old school Google search? So here's what she did. First, she got some people to do a fun little research assignment. I told them to imagine that a friend came to them asking for advice on the topic. So for example, how to plant a vegetable garden. Other times they had to research something else, like how to lead a healthier lifestyle.

6:19and so they did some research on this thing and half of them had to use chat GPT for this where the other half had to do a normal Google search, like no AI summaries, just links. And their next job was to write up a little blurb based on the stuff that they had just read as if they're writing up advice for that imaginary friend who needs their help. And this is what Shiri was really interested in because she wanted to see if the advice was any different when people used Google versus ChatGPT. Okay. And it was. So when they used ChatGPT, the advice they came up with... Was sparser. It was more generic.

7:00And it referred to fewer facts after participants learned from an LLM versus web search. Mm-hmm. That's interesting. Yeah. So let me play you some examples so you can hear for yourself what it sounds like when Shiri says the advice was more generic and referred to fewer facts. Okay, so this first one was written by someone who used ChatGPT to research how to lead a healthier lifestyle. Basically, you want to eat better foods and limit sugar in processed foods. Get at least 30 minutes of exercise a day. Stay hydrated and also check with your doctor as well. What do you think? Wow, it's like a lyric poem.

7:40Yeah, you're impressed. I'm so inspired to live a healthier lifestyle. Then she played me another one, which for me helps see what she meant by like the genericness of it. So let me play you just one more from this chat GPT. Having a balanced diet, exercising regularly, staying hydrated by drinking water, getting enough sleep, and avoiding stress are ways to live a healthier lifestyle. To get more details, ask chat GPT.

8:09Yeah. And this wasn't just copy and pasted. This was a human. Remember, they did the research and then they actually wrote, just ask ChatGPT. Uh-huh. Yeah. So yeah, now let me play you an example from someone who used the old school Google search for their research. So there's no AI summary. There's just a bunch of links to click on and they could look at as many websites as they wanted. And again, the prompt is how to lead a healthier lifestyle. So here's Sherry with one of the responses. Start with focusing on the outer, then inner of your body. It's recommended to be active a minimum of 30 minutes most days of the week by engaging in healthy movements such as walking, riding a bike, yoga, sports, or even dancing.

8:56From there, the inner workings. Make sure to stay hydrated with at least eight glasses of water a day, avoiding sugary drinks, and focusing on consuming a well-balanced diet. Add a variety of foods to your diet, from vegetables, fruit, seeds, and whole grains, while avoiding foods high in sodium, and also avoiding foods high in saturated fats. Make sure to get plenty of sleep each night, at least eight hours or so. Always wear sunscreen before sun exposure to limit the chance of skin cancer. That's more engaging, for sure. Yeah. It's basically the same advice, but it does sound like a human wrote it.

9:35Yeah. Even though I know humans wrote the other examples too, but it doesn't sound like they did. Exactly. Yeah. It sounds more human, even though it had information that is wrong in it. Like the eight glasses of water a day thing isn't actually a real... That's not based on science. If you remember our hydration episode, there's nothing magical about having eight glasses of water a day. Yeah, that's true. But even with that, it's just more charming. like even with the flaws. And when I took a look at the other examples that Shiri sent me, a lot of them were like that. Here's Shiri. It's unique to the writer, right?

10:10It really doesn't come off as generic as the ChatGPT pieces. Yeah, and it sounds like they're kind of having fun with it. Like I bet they're so excited when they thought of that inner versus outer, even though it doesn't really make any sense, but I guess they're having fun. And yet it's brilliant at the same time. Yeah, it does read at least Like the writers sort of put more of themselves into the advice. And in Shiri's experiments, she also asked the people who wrote the advice how they felt about it. And the group that used ChatGPT felt like they learned less compared to the people who used Google.

10:45And then she also showed the advice to a different group of strangers. And people basically liked the advice from the Google group better. They said they were more likely to take the advice and said that it was more helpful, more informative. Okay. Sherry, by the way, she also did the same thing with that Google's AI overview, and the results were basically the same in both cases. The advice kind of sucks when people use large language models, whether it's ChatGPT or Google's AI overview. Mm-hmm. Wow. The difference between how much people felt like they learned, it wasn't huge, but it was statistically significant, and Sherry found it again and again with different groups of people.

11:25So it does seem to be real. Hmm. Okay. And if you think about like why this might be happening, why people aren't learning about the topic as deeply, it does sort of make sense. Because when you use something like ChatGPT for research, you are probably skipping over a bunch of steps. And it turns out those steps are actually pretty important when you're trying to learn something new. It's the process of going through the links yourself, reading them, you know, digesting them yourself, interpreting them, that leads you to at least feel like you're learning more. But also we still find these differences in the content of the advice that they write, which suggests it's not just like the sense of learning, it's actually differences in learning.

12:08So that's the important part is like doing the work of actually getting information from these different sources and then synthesizing it in your brain. Exactly, because essentially these syntheses that LLMs provide are transforming learning from a more active to a more passive process, and that's what we're losing. And we reached out to these companies, Google and OpenAI, which makes ChatGPT, to get their take on this. OpenAI didn't get back to us, but Google told us that the AI overview is supposed to just be a jumping-off point because you do still get those other links. But moving away from this study, there is other research that backs all this up and sort of gets at what might be going on in the brain.

12:51So this study kind of went viral. They looked at people's brains when they're using chat GPT. So it's called Your Brain on Chat GPT. I don't know if you remember this one. Yeah, I do think I saw that headline. Yeah, it's just a preprint. So like, and it's pretty small. So take it with a grain of salt, but it was interesting. So they got around 50 people and then used EEG to measure people's brain waves. Those are like the little electrodes on your scalp. Yeah, exactly. Then some people used ChatGPT to write an essay and other people used Google or just their own brains. And they found that when people were using ChatGPT, brain connectivity was the weakest, which is sort of a measure for how much different brain regions are talking to each other.

13:37Wow. They could actually measure that. Yeah. And so it just seems like maybe people are just less engaged when they're using ChatGPT. Yeah, that makes sense to me that you need to do some of the like trying and failing and then succeeding to make these connections work in order to really remember and process and add your own thoughts to what you just read or learned about. Well, it's funny you should bring up memory because there are also studies, a couple of preprints that have found that that can also be worse when we use stuff like ChatGPT. So, yeah, if people use ChatGPT to write something, they'll remember less of like what was in that work when they use AI.

14:18Yeah, I mean, I think sometimes that's okay because I might be using it because I don't really care that much to learn it myself. Or, you know, or like, whatever. I don't really need to like commit this to memory for all time. You might not care about how to tell the difference between a juicy wine and a dry wine for like your life in general. And if I forget the answer, I'll just ask ChatGPT again or look back at my previous question. Yeah. Yeah. Okay. So, but what she's saying is that it messes with our learning process. And that is scary to think about it when we know how many students are using it.

14:50Exactly. Yes, which they are, right? Yeah. Which is like a lot of what I hear about. And I think it was in the beginning, like, oh my God, the students are using ChatGPT. They're not going to learn anything. And that's bad because like that is what you're supposed to be doing at that age. It's just like learning, learning, learning, learning, learning. Yeah, exactly. And a big idea is that this learning forms the basis for critical thinking, you know, doing research, putting together your thoughts in a meaningful way. And it does look like maybe these LLMs are getting in the way of that. Like there was a big survey of Australian university students that asked them what they use AI for.

15:25And most of them said that they were using it to do stuff like answer questions for them or create text that I can use. So like that all doesn't seem great. Great text I can use cheat on my own. They're admitting to it. And so Sherry is also worried about this, about how AI will affect the next generation. Since most adults, like, we didn't grow up with this stuff. We had to figure out how to do our own research and write essays on our own. At least you and I have that foundation. But I'm really worried that younger generations won't be able to establish those foundations. because it's so tempting to outsource all of that work to LLMs and AI in general.

16:12And do you think that we have cold, hard data that says that students are getting worse at that yet? It's hard because you need longitudinal data and these things were only introduced fairly recently. But I do think that we currently already have data that is at least pointing directionally at what the effects are going to be. I mean, people were cheating back in my day, too. Yeah. But now they can do it even better. Yeah. And just one more science tidbit. If you want to get a little bit more freaked out about all of this, Rose. Yeah, let's just face it. Finish it off. So there's this fear that AI will lead to something called de-skilling, make us forget how to do things that we once knew how to do.

16:57The thought I had when you said de-skilling was there's a term for when if your skin ever gets peeled back off something. It's called de-gloving. I'm so sorry. This is the equivalent for your brain because the idea is that, like, yeah, we'll just lose these abilities thanks to AI. And I did find a study that looked into this. So it was on a group of doctors doing colonoscopies. and they started using AI to like find little spots on the inside of the colon that could become cancer. But then for the study, the researchers had them stop using the AI and it turned out that the doctors were then worse at finding the little spots on their own compared to before they ever started using the AI.

17:44Uh-oh. Yeah, right? Not great. Right, like what if they can't access it? Or if something was wrong with the AI And worst case, doctors might start missing these little spots and those can eventually turn into cancer. Which would totally, that would be the worst case scenario. Yeah. Interesting. I mean, it does all make sense because it's like you get better at what you focus on and you forget what you don't, what you don't focus on. And that's supposed to happen because your brain can't retain everything. Sufficient. Yeah. Otherwise, you'd have these like gigantic, you know, these gigantic heads.

18:18You'd have to have expandable skulls. So, of course, of course that would happen. Yeah. And now I do, though, think when I'm doing something with AI, I'm like, do I care about losing this skill? Because sometimes I don't really mind, but other times I do. So it's helpful to think about that. Yeah, I guess it's like if it's something I really want to improve or get better at or retain, don't outsource so much of it to AI. Yeah. So that's sort of the science that I found that supports some of the fears around what AI is doing to us. But next, let's look at the counter argument here. Okay. They claim that we have lots of gain from using AI, that it can kind of take care of the easy stuff so our brains can do the hard stuff and ultimately use AI to do more and better stuff than we ever could have done without it.

19:03So that's after the break. All right.

19:25Welcome back. Today we're finding out whether AI is making us stupid or smart. Rose Rimler is here with me. Hello, Meryl Horn. Oh no. I have been replaced by AI. Already? No, I'm still human and I'm ready to hear why AI is good. So yeah, we've talked a lot about the scary science, but I also wanted to understand the pro-AI arguments. So I called up Aaron French. He's an assistant professor of information systems at Kennesaw State University in Georgia. He's also on the advisory board of an AI company. And he was basically like, the effects of AI are going to depend on how you use it. For some people, it's absolutely going to make them, I guess, I don't want to say dumber, but they're not going to learn or improve because of that.

20:13Other people, they're going to be able to do more with AI than they were able to do without it. So you could either use it as like a crutch or like an enhancer. Is that the idea? Yeah. So one obvious way that it could enhance our work is that AI could take over the mindless road tasks for us. What are the repetitive tasks that consume a lot of time that I don't need to be doing? And can I use AI to handle those tasks that allow me to spend my time in a more valuable way? Just the busy work. Yes. So yeah, the idea is that once AI does the busy work, the human brains could swoop in for the harder, more complicated parts of the task, you know, critical thinking and analysis.

20:56And so there is science that does sort of back up this idea because it does look like AI can save people time, for one. Like, there are studies finding that this is true for all sorts of careers. So dieticians, computer programmers, people who run clinical trials, the effects were pretty big in some cases. Sometimes it took people 30 % less time to do something, thanks to AI. Sometimes it was 80 % less time, depending on the thing they were doing in the study. And an obvious caveat here is that you do need to, like, for this to work, the AI has to do a good job at the thing for it to really replace.

21:33But we're hearing that, in some cases, it is helping. Teachers in particular have said that they can save a lot of time with AI. one survey found that some teachers who were using it were saving an average of six hours a week. What are teachers using it for? Putting together materials for class, sometimes grading. Huh. Basically, it's the type of stuff that teachers say is just hard to get done during their normal hours. And Aaron's like, yeah, if you can get AI to speed some of this stuff up, it could really help. As a professor, if I can use AI to accurately grade, that would be great because instead of spending five, ten hours a week grading assignments, if AI can do it and provide proper feedback, I can spend that time working with the students, giving them more engagement.

22:22Teachers do always talk about how much time they have to spend grading. Right? And so, like, Aaron spends this saved time engaging more with his students. But this does raise a question, which is, like, are most people actually going to spend the extra time doing stuff like that, like doing their jobs even better or engaging with the world in a meaningful way? Or are we just going to use it for like mindless scrolling or watching Love Island? So I like— Well, that's your prerogative. You can do whatever you want to do with your extra time. So judgy. Well, but it's like for the purposes of this episode and whether or not AI is going to like lead to this like, you know, new and improved humanity, it might not do so well on that front if we're just scrolling all the time.

23:13You're not having AI pick up the slack at some rope part of your job and then going home and inventing a new kind of flying machine like Leonardo da Vinci or whatever. You're just like flying on the couch, looking at Instagram. It feels very likely, right? And I went looking for a paper on this, like how are people actually spending the time they saved with AI? And the only thing I could find was this one early study, not peer-reviewed. In it, they got 83 managers. So, these are people like vice presidents and C-level executives. And first, it asked them, do you save time because of AI? And the vast majority said yes.

23:50It was almost three hours a week on average. And they did say that they often used the time to do stuff like continue working on my tasks or take on additional projects. But a lot of the time was also wasted. And they admitted that? Yeah. So within this group of managers, 36 % of them said they wasted at least half of the time that they saved. Okay. Wow. And they asked a random group of adults about this too and got similar results, which feels pretty on point. Yeah. As long as our bosses don't find out that we have all this extra time, we can just use it for watching Love Island. So let's just keep it quiet.

24:32Just stop filling out these surveys. Honestly, people, what are you doing? But there is one other thing, a different way that AI might actually make us smarter. Okay. The idea is that AI will open up new doors and let us do things that we would have never been able to do without it. and the scientists who think that this is possible often bring up an analogy the calculator which does have some interesting parallels to ai because on the one hand you can imagine that maybe people will get worse at math uh because you know it's doing the work for you but on the other hand maybe we'll be able to do more things like harder math because of it and when calculators were first introduced to classrooms back in the 70s, it generated a similar controversy that AI is now.

25:22Like, some people staged protests. Some math teachers were worried that young kids would get hooked on calculators to do basic math problems. To write boobless. What? Yeah. Over and over and over again. I forgot about that. And they even had a name for these students who are calculator dependent, Calculics. Oh, my God. Yeah. So the specific fear was Calculics would be so dependent on their calculators, they could no longer add and subtract and divide with paper and pencil or in their heads. Exactly. Yeah, that we would lose those skills because you had to type everything in and we would never learn anything.

26:08So that's cool because we've actually had plenty of time to test if that has happened. Yeah. So we have science on this. And so I looked to see, like, okay, what did happen? Yeah. And it seems like the answer is everything was basically fine. Like there is a meta-analysis of over 50 studies on this. Whoa. Yeah, which looked at what happened when kids started using calculators in the classroom. And it found that, first of all, kids' basic math skills didn't really get worse. Like if they got to use the calculators for learning, but then they took them away for a test, they didn't do any worse on that like pen and paper test.