Decoding AI Detectors: Understanding How They Work and Why They're Important

0:00Are ChatGPT detectors accurate? it. That's a question a lot of people are asking right now as ChatGPT has come out specifically for teachers that are looking at if their students used an AI model like ChatGPT to write an essay or not. So this has a lot of actually implications beyond just education as well. Google specifically said that they have an AI detection software that is able to detect if content online is written by AI and they demote that content in their search algorithm. So theoretically speaking, if you wrote articles using ChatGPT, posted them on your blog in hopes of, you know, having your pages on your website show up on Google.

0:39Google is supposed to demote those and devalue those because they were written by AI. Now, people have different opinions on that, but today what we're going to talk about is how to detect AI writing, how accurate those tools are, and things that people are doing to get around them. Because listen, there's obvious incentives to have your content look human generated and when there's incentives it's always going to be a cat and mouse game of people trying to go back and forth catching them and then people trying to evade their safeguards so what's good to know or what's important to know on this is that the advancement of technology is really accelerating right now we know and so what's first important to understand is how ai writing tools work or ai writing detectors work.

1:25So these tools typically use machine learning algorithms to analyze text for features that are indicative of AI-generated content. So for example, some AI writing detectors look for patterns in the word use and also sentence structure that are typical of AI-generated text, and others use deep learning techniques to analyze the text and compare it to a large database of known AI-generated text to make a determination. So what this actually looks like in real life is, for example, this is what OpenAI did with their recent tool they just launched. They went and had it trained on a massive set of AI written content, right?

2:01They obviously can just pull that from their own data set of ChatGPT. And they compare that to content written online. Now, that being said, there is a loophole or I guess an issue with this. They say they don't know if all the content they trained on online, if people have already used AI content and published it there and they trained it on that, right? So, for example, a big data set they used was Wikipedia. They told the AI and they fed all these Wikipedia pages through a neural network and said, this is what a human writes like. Then they took all of their AI content, fed it through a neural network and said, this is what an AI looks like when it writes.

2:37And so it had their AI detect differences between the two. But the problem is a lot of people could have edited those Wikipedia pages using this AI technology. Well, ChatGPT is relatively new. the technology that is presented in there, which is on OpenAI's DaVinci model, has been available for over two years. So a lot of content that on Wikipedia or other places that they trained their models from can technically be, I guess you could say, compromised by AI content. And that would actually make these tools a lot less effective. In fact, I think ChatGPT says that their AI detection tool is only 40 to 60 percent accurate.

3:15So it's really not that great. I've seen other ones online that seem to nail ChatGPT content pretty well. But that being said, what a lot of people are doing is they're using tools like Quillbot to get around this. If you take a ChatGPT written article or essay, throw it into Quillbot, it will rewrite it in a more human way, and it will evade AI detectors. So teachers don't like that because, you know, when they're trying to get students to write essays, they have plagiarism checkers, but if they're unable to detect if a student just used AI to write it, then that's difficult for them. And it kind of brings up this whole conversation of, you know, like, what is the role of chat GPT?

3:55What is the role of these AI writing models in education today, right? Because I'm sure back in the day, math classes were very mad or upset when, you know, calculators were invented and students could bring a calculator to class to calculate something. All of a sudden, you know, the teacher's upset because that used to previously just been mental math or whatever else they did back then. So, you know, eventually in math classes, they just decided, they just made rules for what types of calculators you could use, how advanced they could be, and when you could use them, what tests and whatnot. And so I personally think that for chat GPT and these other language models, it's going to get incorporated into education.

4:35I mean, it's a very powerful tool for business and for people's jobs. And that's kind of the purpose of education in a way. So, yeah, so I honestly think that it's just going to be, it's going to come down to people deciding when it's appropriate to use ChatGPT, when it's appropriate to use these AI writing models, and how much they can use them. And it's not going to be a thing where they can just completely ban it, like a lot of them are, you know, New York, I think, has made some rules trying to outright ban it. So one other thing I will say on OpenAI's training of these, you know, AI detection models is another interesting thing they did is they did the whole Wikipedia page thing, but they also went and they would take half a Wikipedia page.

5:19So they take, you know, or like, so they take like a thousand words from a Wikipedia page. Then they would go to chat GPT and say, continue writing this Wikipedia page. So it was really powerful because they would have this article that was half human written, half AI written, but it was all on the same topic so not only are they able to say you know this is what an ai writes like this is what a human writes like but it's on a specific topic a very specific similar topic this is how each of them write and they have a neural network that can be fed this data back and forth and identify patterns and little with using algorithms identify patterns in the writing so um ai is typically very perfect in its grammar very perfect in its punctuation very perfect in spelling humans make a little mistakes here and there that's one easy giveaway which is kind of interesting um and so yeah there's a lot of different things like that that go into this but to date there are some powerful ai detection uh softwares