AI's Academic Milestone: GPT-4 Successfully Completes Freshman Year at Harvard

0:00Chat GPT has just passed freshman year at Harvard and its GPA is probably going to surprise you today on the podcast we're talking about an experiment that was recently done by freshman at Harvard where she essentially got GPTGPTGPT has just passed freshman year at HarvardGP is a freshman year at HarvardGP is a freshman yearGP is a freshmanGP is a freshmanGP is is totally passable. And I think it really highlights what is going on in the education system and the fact that, you know, you really could use ChatGPT to write your essays in college and what the impact of that is. So let's dive into it on the podcast today.

0:44I think this experiment was done by a freshman named Maya Bodnik. And I think her experiment really illustrates how AI like ChatGPT is going to disrupt the conventional learning and kind of evaluation mechanisms at universities. And then more specifically, I think this is going to potentially influence, you know, social sciences and humanities education, which traditionally rely really heavily on take-home writing assignments for student assessments. You know, I can remember back to my days at college, I had classes where, you know, they would say, you have to go and write the entire essay in the testing center.

1:20And, you know, sometimes it'd give you a few hours to do that. And, you know, so that's kind of an obvious solution if this was really a super integral part but I do think more broadly we may start seeing that there is going to be an evolution in what assignments are in college and how we measure success and understand in education in general so in any case with this specific test may you know she had seven Harvard professors and teaching assistants who were asked to grade essays written by chat gpt for and of course they were all involved and knew this was a study but um and all of the all of the essays obviously were in response to prompts from their actual courses so this was you know doing actual classes and each essay was presented without telling if it was the actual author or if it was chat gpt so she said you know in the study uh you're going to be getting either an essay written by chat gpt or by me you won't know you got to grade it um and that was kind of her attempt to get rid of the bias that they're just going to grade them all bad.

2:22But in reality, every single one was written by GPT-4. So the topics range from economics and politics to conflict resolution, Spanish and literature. And they were really, she was really trying to kind of test the AI's ability to handle a really broad range of subjects. The AI generated responses were submitted verbatim. So exactly what GPT-4 spit out is what she, you know, submitted for the reports. And the only thing that she altered was a little bit of resequencing to meet the word count. You know, ChatGPT, GPT-4 only really gives you a 750 word response. So sometimes she would have multiple questions to get a few paragraphs longer or whatnot, and then she would just stick those together.

3:07But whatever it wrote, she didn't edit. She just gave it, you know, straight as it was. ChatGPT-4 really surprised everyone, I think, by not only passing the courses, but it actually achieved some pretty impressive grades and some professors and TAs I think this is actually kind of funny they praised the AI as quote beautifully written or well articulated and I'm not sure if they knew this was GPT-4 if they were like they thought oh this is probably by the actual person so they were just you know giving her good grades but in any case it was written by GPT-4 so some though did offer some criticism for it you know overly embellishing some things so I think while the AI seemed to do pretty well with style.

3:46The content of the essays received some more nuanced feedback. I think it received an A in American presidency and microeconomics based on its attention to detail and adherence to requirements. However, I think its paper on Latin American politics and Spanish lacked what, you know, quote unquote, sufficient analysis. So it got a B minus and a B on those. But I will say that none of its grades actually dipped below a B minus. I think on all of its papers, the grades were something like A, A, A-, B, B-, and then I think it got one pass on one of its papers, one of the seven. So honestly, fairly, fairly good, all things considered.

4:26One thing that I think these results really do indicate is that AI like GPT-4 is likely going to achieve passing grades in most liberal arts courses at most universities. And I think given the model's really powerful performance, even if it might not, you know, get the top grades at universities with stricter grading systems, right? Like Princeton or UC Berkeley. You know, someone could argue that, you know, it's not going to work at an Ivy League. Well, okay, it worked at Harvard. So that's pretty substantial. That's kind of, you know, the name brand university. So Princeton and UC Berkeley, yeah, maybe they'll be a little bit more strict, but with a little bit of editing and work, like I think these essays can be an A plus essay at, you know, these are A essays on at most all universities.

5:12And so I think that this is going to, no matter what, you're definitely going to pass. You're definitely going to graduate regardless of, you know, if your GPA was the most important for your specific degree. So I will say, though, I think these results have, you know, some deeper implications. I think, number one, they suggest a really forthcoming paradigm shift in the way that humanities and social sciences are taught. Up until now, while Google has obviously been a help, the internet has not really been a super effective tool for high-level plagiarism. It obviously, you know, fails to give good answers into complex creative or personal prompts.

5:47And I think that this, you know, up until now really required students to invest some sort of effort in finding the material online and often, you know, mixed with their own writing and creating their own citations and whatnot. but I think in the era of chat GPT this kind of changes the equation so I think with its improved accuracy that we're now seeing with GPT-4 I think with its ability to answer any prompt specifically I think this really generates a it will it really is able to generate a full answer and it requires minimal editing from the student right that's it's not like you have to go paragraph by paragraph you really can get it to write a big huge chunk it pulls and ties everything together and if you want it to be longer you can prompt it for something longer but you don't really have to adapt and be too creative because it is able to do that it's able to have personal prompts you know the model has yeah some would argue the model has made cheating easier than ever however i think current ai detectors are also deeply flawed so i think it really kind of brings us to a point where obviously chai chibi t can write a really impressive essay it's really hard to get an essay run through an AI detector to the point where OpenAI launched their own AI detector for the education system pretty much.

7:03And they just shut it down because it wasn't actually able to work and it gave a high percentage of false positives. And so in addition to that, a lot of AI detectors haven't really been widely adopted by educational institutions. And I think a big part of that is when you have a company like OpenAI, who's obviously the one who invented chat chp and if they can't even do it it's hard to trust other companies so i think all of these really make it much more challenging to catch students who might use ai to complete their assignments and i think this is a really big problem that educators are grappling with at the moment i think in light of some of these results one of the really pressing questions is how to combat this new form of um you know potential academic dishonesty or do we just give up on that and decide this is the equivalent of a calculator for text and you just are allowed to use ChatGPT now to generate things and it's now it's a question of like when you can and when you can't use it and in my personal opinion I think that that is the direction this is going to have to go whether people like it or not or agree with it or not that's the direction it's going to go because if it's impossible to detect and everyone uses it there is no way you could actually really like there's nothing you can really do about it so I think at that point you just have to decide it's like a calculator and you know maybe some situations you have to go to the testing center and write a whole thing yourself to prove that you're capable of writing something well thought out but in other cases I think they're just going to have to start evaluating how everything is done you know I even saw a funny meme recently where it was like because of chat gpt my teacher said we all have to like write our essays from now on and then he had he like created a little robot with a pen in its hand that would copy his handwriting and write out his essay for him.

8:48So it's really, I mean, unless someone's watching you live, which I mean, perhaps a testing center or in class is possible. But especially with like online universities and stuff like that, it's very, very challenging, I think, to get around ChatGPT entirely. So anyways, it's going to be really interesting to see what happens there. Of course, it's incredibly useful to have the experience of writing things out yourself and practicing using your brain. And I totally get that. But I think we are in different times. And it's going to be interesting to see how people adapt to that. I think because of this, and because, you know, of course, AI detectors aren't really working, I think educators are going to have to consider shifting from take-home essays to in-person assignments.

9:32That's going to be a big thing. I think this approach, while it's obviously not without its limitations and trade-offs, I think it's going to help maintain some academic integrity if that's really important to your degree. That being said though, right, like I remember in college having to write English papers that were super, super long, had a ton of citations. You know, you're reading a book and you're doing all this research on it to write this really in-depth paper. And if the only way that you can be tested on that is to do it in like a testing center over, like you do have like a time limit, right?

10:03Like I worked on that paper for like a week right i had a lot of classes i had to fit stuff in um and so it's going to be interesting maybe we just maybe it just moves to shorter papers or something i'm not sure so that's it's going to be interesting definitely something people are grappling with i think the implications also extend beyond kind of the classroom i think if ai can replicate the academic work performed by students it's not far-fetched to imagine you know a future where ai might take over jobs that involve similar kinds of tasks in the real world right like obviously people are saying oh no how are we going to stop like students from using AI to write their papers like that's in my mind less relevant to what is the what does that mean if AI really can write really good papers does that mean that journalists get replaced does that mean authors get replaced does that mean you know what what does that kind of look like and so I think this kind of underlines the pressing need for educators and policymakers to grapple with the implications of AI not just for academic integrity but really for the future of work and society as a whole and you know some people say awesome ban AI, get rid of it, it's going to take jobs.

11:04But then it's like, well, if we develop something that is able to do it better than humans, and, you know, there's always, of course, the debate of, well, what, why do we work? What is the importance of work? Should a human do it? And then, you know, there's a lot of people's counter argument, which is like, if it brings you joy, and that's what you love to do, even if a robot could do it better, maybe you want to keep doing that, right? And I think we see this in a lot of cases today. And maybe this isn't something a lot of people have thought about but think of the fact that like if you go to a if you go to like a farmer's market or something um very popular where i currently live there's a ton of people who have hand witted or hand knitted wool sweaters and handmade soap and handmade you know beeswax candles and all sorts of like you know like um handmade glass crafted um ornaments and sculptures and all sorts of things like that and I think it's really interesting because of course like those things have a very high value people absolutely love them but all of them could be mass manufactured by machines right like soap candles knitting sweaters glassworks like a lot of that can just be manufactured all of that can be manufactured by machines and so it's like why do we buy it from a human it's like well it has this like artisan and craftsman touch and it makes me wonder if at some point we'll get there where it's like we like journals that are or like newspapers that are exclusively written by humans but they're almost like this more artisan kind of touch like indie like vibe you get from them where it's like if you just want like interesting information and it's super fast and it's super cheap and it's super relevant and it's got some good you know basic overviews of whatever like ai could 100 take over a lot of newspapers that just are writing high level news they're all aggregating from routers anyways and writing about whatever you know routers wrote about that morning so i think there's there's a lot of and maybe they could even have some special models built in that they can go and grab extra insights from places that no one else has access to data and they you know tie in some new insights and stuff so it's not just like regurgitated content there's a lot of things that could happen and so i think it's going to be really interesting to see where that goes um you know obviously this is interesting to me from a journalistic perspective as I cover the news every day.

13:23But I think ultimately this experiment that happened really paints a picture of the future of education in the AI era. I think it's a wake-up call for educators, students, and institutions if they already didn't have that after testing out ChatGBT. But I think it really kind of reassesses their practices and values in a very rapidly evolving digital landscape. I think the rise of this kind of technology is going to redefine the dynamics of higher education. And I think it's going to really require us to kind of reimagine how we impart knowledge and assess understanding in a world where an AI model can pass college.

13:58So this is an area we'll definitely be following into the future. Very, very interesting and a lot of incredible advancements happening live.