OpenAI's AI Detector Demise: The Short-lived Debut

0:00Something that's pretty common we see with a lot of big tech companies is they will come out with a new product and if it doesn't get a lot of adoption they'll kill it. Google's probably one of the most famous examples. They've killed a ton of products. There's Google Buzz. There's Google Plus. They recently are killing Google domains. So many ones. Today, we want to talk about OpenAI's foray into this because OpenAI, up until this point, doesn't really have a massive product line. They don't have tons of stuff. And so when they typically make things, we see it stick for a while. They're a startup.

0:30They're trying to be really innovative and fast. Even products that I think are absolute garbage like Dolly 2. and when I say absolute garbage I mean I guess I'm being spoiled I think it's impressive if you know when it came out but really at the end of the day compared to some of their competitors like mid-journey I feel like they're just getting absolutely crushed so I think that there definitely is space well they have a lot of funding so I think they actually just spend more money and make their their products better but that aside from that today we want to talk about on the podcast opening eye shutting down one of their most recent products they launched not only not more than just a few months ago why they're doing that and what the implications are and this is actually the text classifier product they made that essentially was able to classify if text was AI generated so let's jump into the podcast today.

1:17I think the idea that AI generated text can exhibit certain characteristics that would allow for reliable detection while initially this was very plausible it's actually proved to be a lot more complex than OpenAI originally anticipated. So I think the rapid advancements and nuances in a lot of these LLMs have made any telltale signs very elusive to the point of unreliability. And this is something I have predicted for a very long time. So originally, last summer, around a full year ago, I began using different AI article generating tools to generate articles for some products or for some SaaS companies for some of my websites that I have.

1:58And I was using that just largely for SEO to drive more Google traffic. Now, the problem was at the time, Google was a lot more anti-AI. They said, you know, if we detect its AI, it's going to be bad for your search results, blah, blah, blah, or at least there was a rumor of that happening. So at the time, I would use tools and software like Quillbot that would essentially take your AI generated text. It would kind of rewrite the article or swap words out of the article or rearrange things a little bit using AI. But the idea was it did it in kind of a more random, sporadic way that made the text undetectable.

2:29And at the time, there was a number of AI detection software and tools that I was using. If you got something by OpenAI generated, at the time it wasn't ChatGPT, it was DaVinci. But if you got something generated by DaVinci, which is more of a developer API tool anyways, but you threw it into one of these platforms, it would tell you this is AI generated. And then if you threw it through QuillBot, it would say this is not AI generated. So the idea was you throw everything through that and it would you know it would make it so that it wouldn't be detectable then of course the concern was well eventually google is going to figure that out and they're going to figure out a way to see um what is generated uh through quillbot whatever and it's going to be this cat and mouse game forever that was my assumption and then we started to see some of the big shoes drop and the first one was google saying look we know that this ai tech is so big it's going to proliferate into everything we're not going to demote uh content that is ai generated we're actually not even going to look at it anymore we don't care all we're going to do is if it's high quality we'll keep it if not it dies just how it's always been so if you create a high quality piece of ai content um it's just going to stick i think this is really interesting because this concept kind of proliferates into everything with images and video and everything else it's going to be the same right like eventually we're going to have ai generated youtube channels and people will be like oh man should ai should google ban them nope youtube i think they're going to follow precedent with their google search and they're going to say if it's a high quality video it's going to get shown if not it's going to just you know die in the algorithm um the same death everyone else faces so and i think images and google search i mean google is trying to do ai image classification in their google images so that will be interesting to see how they roll that out um but it doesn't appear that they're really being able to do that in text i think text is particularly difficult because with image it is one file you can identify as ai but with text what if you used you know you generate the intro of your article with ai and maybe the outro or maybe some frequently asked questions in the middle, but the rest was all hand generated.

4:23How is Google supposed to like determine, okay, this, they used any AI in this thing, like kill, you know, kill the whole article, make it not rank. Or, you know, for example, maybe you wrote an article, but then you go use an AI that comes up with really good titles and descriptions for your article or generates good SEO metadata. And all of a sudden, you know, Google would have to determine if it was going to get killed. And so it really just became this big mess that Google didn't want to play with and they said we are out but before that there has been this whole controversy of you know teachers worried that their students are using ai and they really wanted to detect it and so in this whole controversy mostly i think from an academic perspective everyone worried that ai was going to essentially corrupt the young minds of the children today so they're never actually going to learn anything i'm not saying if that is or isn't the case or isn't isn't is or isn't reasonable that was a concern and it was a big talk with chat gpt and i think open ai wanting to be the responsible player in the room and show how their, you know, AI is responsible and good and not wanting bad press was like, okay, well, guess what?

5:20We are going to make a tool that detects AI. It's an AI, you know, tech classifier. We got this thing. We got you teachers or anyone else worried about, I don't know. I don't know really who else is too worried about it because in business, if it sells a product, it sells a product and no one really cares that much, but in school, it is a problem, right? So they came up with this tool, which is the AI classifier, and it immediately got sort of roasted. Honestly, I never found it very good. I had a lot of other AI tools that worked much better. But the problem, the biggest problem with this AI text classifier is that it would give you false positives.

5:54Now, there's a bunch of big stories where teachers essentially fired the, or they tried to fail the entire classroom because they said, all of you guys are using AI to generate your thing. And the teacher actually didn't really know how AI worked. I think he just copied and pasted essays into ChatGPT and said, did you write this and chad chippe said yep i wrote that to like everyone's essay and then he tried to give everyone an f so it's kind of silly but also at the end of the day i think just it gives if it gives a false positive can you imagine being a student working really hard on a paper submitting it and then them being like this is ai generated because this software that isn't perfect said it was and it wasn't like oh my gosh that'd be horrible so anyways um they're killing it i think it wasn't effective enough it didn't know enough i think the fact that google kind of let go of the reins on that and said we're not even going to try it makes it a lot harder for anyone else to have any sort of accountability in this space because um there's so many different ai tools i even know there's like tools that are you know there's for example 11 labs has like an ai detector where they're like yeah we can detect if if someone generated some audio content on 11 labs but it doesn't detect it if they generated on a different audio platform and if they generate on 11 labs and maybe change the pitch and tempo a little bit um which people have been you doing for, you know, music copyright stuff on YouTube for like many, many years, then it's not detectable either.

7:12So it's essentially not that useful if there's any sort of altercations or it's on a different platform. And I think that's the same problem with OpenAI. Maybe they'll be able to detect eventually make their thing good enough that it could detect AI generated content from ChatGPT. But what about Claude? And what about all the other models, Anthropic and Inflection and everyone else is going to make, I think eventually this is a perhaps a losing battle. and so because of that they have made the decision to actually kill this so in a series of tests that were actually conducted by chat gpt they actually highlighted the nature of these ai text detection tools one or out of seven ai crafted texts open ai classifier correctly detected only one compared to chat or compared to gpt zero's five right so there's two different tools that TechCrunch was testing there.

8:01GPT-Zero was able to get five of them. OpenAI's classifier only actually got one of them. I think this kind of mediocre performance is a massive, you know, downside for OpenAI. It was pretty embarrassing, especially when, like, they're releasing this product because it's definitely not state-of-the-art and other startups are doing this more reliably and better. So I think they had to kill it. I think really interestingly is despite the clear limitations outlined by OpenAI when they first released their classifier tool many people continue to use it at face value or even beyond so users you know tried to identify ai written content in the work of students job applicants freelancers and they really put their faith in this classifier even though i think they said like hey this is just uh you know this is just a testing tool and obviously this thing was not very good so i think amid all of the constant improvements and spread of language models someone at OpenAI seemingly decided that they needed to pull the plug on this tool.

8:56It was not reliable, so they decided to kill it. And this is what they actually said. They said, we are working to incorporate feedback and are currently researching more effective providence techniques for text. And this was first reported on Decrypt. But in any case, inquiries regarding the exact reason and timing for discontinuing this are yet to be addressed. I don't think they really brought that up. But I think that the decision really coincides with OpenAI's involvement in kind of this whole White House led voluntary commitment to developing AI ethically or ethical transparency that's going on right now.

9:33And I think one of the commitments that these companies have made includes the development of some really robust watermarking and detection methods for AI generated content. So it's kind of interesting because right now, while they're making all these commitments that they're going to start detecting more AI generated text and they're going to the White House and whatnot, they're also pulling the plug on this tool. I think it's because they know this tool does not meet the standards that they actually need for their tool. So I think for the last half of the year or so, last six months, I think the AI community has really witnessed a lot of talk, but not a lot of tangible progress on watermarking.

10:08Right. We have some commitments from Google that they're working on that for their images. We haven't seen that a lot. And so I think it's going to be, I honestly think it's going to be a massive problem. It's going to be very difficult to figure out. I think Google, Microsoft, OpenAI, I think these companies are the only ones that even have much of a chance. Maybe there'll be better tech, but here's the deal. So Google has indexed the entire web and they have records of all their indexing images, videos from YouTube and, of course, articles. And at some point they can say, you know, OpenAI launched their first version of AI generated text at this point.

10:42Therefore, we can assume everything on the Internet from before that was probably human generated. And I think having that data set is the solution that is the final key to being able to determine. and I think it's interesting because not a lot of people have this data. I think there is a handful. So I'd be really curious to see how this plays out, who is the winner in this AI detection space. It's going to be very lucrative, right? These people are all making big promises, big companies making big promises to the government. They better deliver. And so I think whether it's a startup or internally, there's going to be a lot of money put into this space and this is going to be a very interesting place and space to continue following in the future.