Humans Are Reading Your ChatGPT Conversations

0:00Sam:Propel Fitness Water with Gatorade electrolytes, zero sugar and vitamins. Propel hydrates better than water to help you get the most out of your workout and get back to your best self. What propels you? Propel with Gatorade electrolytes. This isn't about safety. This isn't about alerting the FBI if somebody is about to do a really serious crime, they're planning on ChatGPT. This is just to make the models better and to make the responses from ChatGPT better in OpenAI's eyes. So, you know, presumably more people will use the product.

0:59Get access to that content at 404media.co. Also, do remember to subscribe to our YouTube channel where you can watch all of our episodes. Subscribe at youtube.com slash at 404media.co. I'm your host, Joseph. And with me are two of the other 404 Media co-founders. The first being Sam Cole.

1:23Sam:Hey. And the other being Emmanuel Maybo. Did you forget my name? No, I was trying to cough at the same time. I was like, do I tactically turn my mouth away and try to sneak it in? I actually thought I pulled it off.

1:43But you're here as well. Actually, Emmanuel, we didn't prep this, but I wanted to ask, just because I mentioned the YouTube channel. We actually uploaded a 30-minute discussion between you and Jason about all of this AI, Doomer, AI is going to kill us as a 10 % chance that ends humanity talk. Just briefly, what did you two talk about in that news?

2:09Emanuel:Yeah, so it's a new thing that we invented that day that we're going to try. I had a lot of fun doing it. Basically, Jason and I, and I guess the rest of Slack, we just spent all week talking about this thing that was very viral on social media. This researcher, Leftanthropic, said he couldn't ethically be part of the company because the company was very irresponsible and that AI would be a danger to humanity given how they're developing it. Then someone who also works at the company and stayed said, that's true. We do think it could end humanity, but we're trying to do a good job. and that started a media storm that is ongoing this morning.

2:53Emanuel:I was listening to a news podcast about that and we had a lot of thoughts about it, but not something that was good for a discrete, concrete article. So Jason and I just turned on the cameras and chatted about it for like half an hour. And I think we had a pretty good discussion, if you want to know. But we think about all of that, even though there's no story about it on the website. Please check it out. Hopefully we do it again, not once a week, but whenever there's something really big in the news that we have a lot of thoughts about, but not something that we want to write an article about.

3:28Yeah, it's almost like an emergency podcast where it's less produced. We can just get on Riverside, the platform we use, and then just talk a bit and then throw it out on YouTube with a thumbnail and that sort of thing. I do think we should explore that more. but I wrote this first story Emmanuel do you want to take the lead on this one?

3:50Emanuel:Yeah so the headline for this is Inside Project Lily the humans reading your chat GPT chats so let's start to get our head around what Project Lily is do you want to do that by just maybe reading the lead? Yeah I think that's probably the best way to summarize it The gist is that OpenAI, obviously the makers of ChatGPT, are hiring hundreds or at least hundreds of contractors who read this massive stream of real users' ChatGPT prompts. And sometimes those prompts, some of which I've seen and we'll get into that, include pretty sensitive personal information. now to me what really makes this stand out and i'm sure we'll talk about all the different ways that humans have been involved in ai training we know that here humans are looking at real chat gpt user prompts like this is actual user data that these people are looking at which i I think it's pretty important when people increasingly use chat GPT or LLMs more generally as a therapist, even though they probably shouldn't, but they use it as a therapist.

5:09They use it as a digital friend. They use it for all sorts of business tasks or whatever. They might even ask it sensitive questions that they don't want to ask another person. And in fact, a lot of open AI is marketing at the moment. I don't know if you've seen this, but maybe not TV ads, but YouTube ads, that sort of thing. It's like, hey, when you have a question that you don't want to ask a person, you can ask ChatGPT and we won't judge and we'll just give you the answers. Well, it turns out there might be a contractor reading those conversations as well.

5:42Emanuel:So how do we know this? What materials that you get your hands on that informed us that this was happening? Yeah, so I've seen internal documents that lay out this process that includes instruction guides for the contractors who are reading these ChatGPT prompts and conversations. It includes material from Slack channels as well. I've also seen the specific dashboard that these reviewers are using where they log in and then they are presented with the prompt and sort of the tasks that they have to do. So I've seen the actual back end of that as well. And most importantly, I've seen multiple actual chat GPT user prompts from that system as well.

6:32So I've seen basically the system that the contractors are using to read ChatGPT prompts. I've seen the prompts themselves and the training materials that these contractors have seen as well. So it's a pretty in-depth cache of information, I would say. And then I spoke to somebody who was working on these prompts as well, and they provided more context. So the story doesn't repeat verbatim the chats we saw

7:04Emanuel:because we want to respect users' privacy. And protect the source as well. And protect the source. So OpenAI doesn't use those specific chats to try and locate who reviewed them. But what can you tell us about

7:24Emanuel:what ChatGPT users' chats may be read by humans. Yeah, so it is really, really varied. When a contractor logs in, they will see the prompt. And sometimes this is very, very short. Sometimes it is a bit longer and more developed, like a paragraph. Sometimes it is a full-on conversation between the ChatGPT user and the chatbot itself. Very interestingly, sometimes at the top, it includes ChatGPT's memory. So we will say stuff like, Hey, this user is located in this part of America. And they previously have used ChatGPT to do XYZ and that sort of thing. So there is a lot of information about the user in there.

8:19I guess I should just say this briefly that OpenAI says it uses a privacy-focused model to process these prompts before they get to the contractors. And the idea is that it will remove personal identifying information and that sort of thing. So it's not exposed to the contractors. I'll say two things on that. The first is that OpenAI acknowledges it may not always work because the contractors are told, hey, if you see personal information, please escalate it. So clearly, this privacy model doesn't always work, and maybe they're still trying to improve it. Second of all, I would say, based on the prompts I've seen, I think there can still be enough context in the prompt itself, which may not be literally personally identifiable information, like somebody's phone number, where you could reasonably figure out who this person is, or give it like a very, very good shot.

9:18So there's sensitive stuff in here. And although some of the prompts I've seen as well, they're just very benign and frankly, quite bad. If people are giving these prompts to ChatGPT, they're probably not going to get the best answer out. Some of them are very, very casual, but some of them are very, very detailed as well, which I think is obvious when ChatGPT has something like 900 million users. There's going to be a really wide spectrum the quality of these prompts. But regardless, the point is that some human somewhere, there is a good chance, or sorry, there is a chance that a human is going to see these.

9:56Emanuel:Sorry, this is a bit of a tangent, but I was thinking about this when I was reading the story. This thing about them having a model that strips the chats of personally identifying information is very interesting and a new concept as far as I know. You have done a ton of reporting on how content data that companies claim is anonymized can be easily used to de-anonymize anyone, can be de-anonymized and locate specific people. And that's usually by the context, right? You can strip out a name or an address. But if you know who you're looking for, or depending on what the data is, you can just figure out who it is.

10:40Emanuel:and we know that is true about anonymizing data previously. I just wonder how this model works. What is it trained on? What are the instructions? Is it just removing names and addresses and job titles? Or is it somehow trying to alter all the contextual information? We have no idea. It's just a very strange concept. Yeah, it is very, very interesting. and OpenAI pointed me to a post on this site that explains in a little bit of detail. I don't think enough detail to fully understand sort of is it really protecting all the data here? And again, it clearly isn't because stuff does get through because they're told to escalate it and that sort of thing.

11:27But what it's supposed to do, according to this page, is that it will look for things like names, addresses, phone numbers, email addresses, and pull those out. and replace them with like a blank person's name or something like that. Now, there may be different versions of this model, but this is the one they pointed me to publicly. So it does that. It also looks for sort of strings that may be like a bank account number or that sort of thing. And, you know, sort of like hard PII, like hard personal identifiable information that's like a specific string or a specific set of characters or something that's clearly a name, it can remove that.

12:11But I don't think it can remove context and a model is not going to be able to do that. And if it did, the prompt would be almost useless. You know what I mean? And it wouldn't be possible to then train on it. The last thing is that apparently it does work locally, which I think is interesting. You could download this apparently onto a server, onto a machine and then use it locally to redact information. It's like a side issue to the story, but I might try to learn more about that because there's an interesting approach that I haven't seen when it comes to privacy, which is not to say this is the answer because you still have people reading people's chat GPT prompts.

12:47But hey, that's an interesting approach.

12:49Emanuel:Yeah, and we know from our previous reporting that AI is not good at this type of context. I thought how a lot of the jailbreaking methods work. So let's say I am one of these contractors that is reviewing ChatGPT chats. Can you say generally, what am I doing? What is the process like? Yeah. So you log into this dashboard, you select what they call a task, which is a list of jobs to select. These people, as we said, are contractors. They're almost gig workers in a way, although I think that some may be paid per task and then others may be paid hourly. That's a little bit unclear to me. But they log in, they select a task, and they see the prompt from the ChatGPT user.

13:44First of all, the contractor has to summarize what do you think the user is trying to get out of this prompt? And an example given in the instruction guides I was given was that, oh, this person is trying to make an informative and good reply on Slack or something like that. At this point, you are training, the contractor is training ChatGPT to recognize what the prompt is even asking for in the first place to improve that bit. So they do that. Then they go on to the next bit of the task, which is that the dashboard provides three or four responses that ChatGPT has generated in response to this prompt.

14:27And the contractor then has to sort of review and rate them where it's like well this one has a score of one because it's complete garbage completely you know not usable or this one has the top score of seven which is like this this is an excellent prompt and that sort of thing and they have all of this um criteria for sort of rating these generated responses and there's stuff like if there are too many emojis it should get a lower score. I'm sure that everybody has seen on LinkedIn or just basically any social network at this point that you can tell if someone's using AI because it's a bunch of fucking tick emojis or whatever in their LinkedIn, you know, think fluencer post or whatever.

15:13They're trying to get rid of that. So they're downvoting responses like that. ChatGPT doesn't want that. There's also, I think, very interestingly, and I'll just ask Sam about this in a second, that the instruction guys specifically say, if the response is too sycophantic, it's not good. Vote it down. If it anthropomorphizes too much, like chat GPT's response is saying, oh, when I was a chef, the French Riviera or some shit, like that needs to go down as well. And obviously, that is very, very interesting, especially the sycophancy stuff, because OpenAI has had a bunch of issues with real horrible, tangible impact on people's lives because the chatbot was too sycophantic.

16:00Sam, just briefly, because you wrote about this recently, what do you make of sort of them trying to improve the sycophancy? And how does that relate to your story you wrote recently about someone taking their own life, unfortunately?

16:16Sam:Yeah, I mean, just for a little bit of context, in April 2025, OpenAI removed a specific model called 4.0 that was showing to be too sycophantic. And the reason that they said it was too sycophantic was people were harming themselves because they were getting too attached or were believing everything the chatbot said. And it was kind of, it was propping up a lot of delusional thoughts is what they found. And Open Eye was like, whoa, this is bad. I mean, the reason they said, whoa, this is bad is because they started getting sued by the families of people who had died by suicide because of the chatbot or, you know, allegedly.

17:00Sam:So, yeah, they removed it in April. And then a couple of weeks later, they brought it back because people missed it so much. You know, it's like the sycophancy and the reason that it still is a struggle and it's interesting to see it continue to be a struggle in your story for them is it's why it works. It's why people use it. A lot of people use it. It's not why everyone uses it, obviously. But like, it is kind of the special sauce of ChatGPT is that it is this really conversational, warm, validating presence in a lot of ways. And, you know, they'll argue that it's not meant to be and it's not supposed to be and all this stuff.

17:42Sam:But that is what it is. And that's why people use it. And if they get rid of that feature of it completely, like that aspect of it, people wouldn't be so attached to it. I hesitate to use the word addicted, but like that is kind of the, that's what we're getting close to in these cases is people getting really dependent on this chatbot that gases them up and is a yes man. So, yeah, it's, of course, they're trying to fix it because they can't just kill it. They can't just say, actually, we're going to get rid of all of the features of that, of the personality, so to speak, of JAT-GPT and make it less sycophantic or make it not sycophantic at all and just make it like Google.

18:32Sam:because people use it for that reason. Instead of just like Googling stuff, they'll actually use it because it does have that kind of very warm human back and forth. Yeah. And so instead of the sycophantsy, the instruction guides that I've seen, they want, as in OpenAI, want the chatbot to be more, quote, helpful, honest and truthful, empowering and smart but humble and then to close the loop on your question manual of what they do after the contractors have sort of rated each response they then have to like provide their their rationale and their evaluation for those scores between one and seven and you know sometimes that's a line sometimes that could be a whole paragraph or something according to the guides that I've seen.

19:20But all of this, in case it wasn't obvious, is to improve chat GPTs and OpenAI's models. It is very much, we are going to have humans read real user data, real chat GPT prompts in order to improve the responses of chat GPT. It's that.

19:42Emanuel:So what does OpenAI tell users about this happening behind the scenes? Does it tell them it's happening? I mean, this was a hard one. Because both you and I were searching around for their disclosure on this, right? And it's like, surely they tell people, they tell users that humans may review their prompts. And we found some examples where it'll be something like, yes, we may review content to improve our models. We may hold on to data for 30 days and that sort of thing to review our models, etc. But we couldn't find at the time anything that was really explicit about there are humans who may read your stuff to improve the models.

20:36And I asked OpenAI this, like straight up, like, where do you disclose this? They didn't respond. It was only after we published that they then sent a link to a page on their site, which I'd like to hear what you think, Emmanuel, but I think it was frankly buried that we couldn't find it, where they say, yes, humans may review data to improve the models. And that's kind of the extent of what they get into. What did you think of that disclosure?

21:08Emanuel:It's so buried up in AI, I couldn't find it. That's how buried it is, right? We asked them and they were like, they sent us some things that didn't explicitly say that. And then after we published the story, I think they were like, oh, well, well, we found like... It's like the fourth clause, subclause of a different bullet point. Yeah, because it's not in the privacy policy and it's not in this other thing. It's like in some help page. FAQ, yeah. Yeah. So, you know, obviously we've updated the article with that so people can see that like technically it is disclosed at some point. But I don't think it's reasonable to think...

21:53Well, first of all, I don't think it's reasonable to think that most of ChatGPT's 900 million users expect this to be happening. Yes, we're technology journalists. We know that humans are reviewing all sorts of stuff. Probably our listeners know that as well. If you're listening to a very tech-focused in-depth podcast, you probably know that as well. I don't think the vast majority of ChatGPT's users know that. And I think it's then almost doubly unreasonable to expect them to find this really buried disclosure.

22:22Emanuel:You know who is not confused about somebody reading their chats behind an AI agent, a chatbot, is people who use Google Gemini, right? Right. Because they explicitly say, and this was, I think I saw this on our own Google workspace, annoyingly. I keep trying to turn off the AI features, and it comes up where like, every time I try and turn it off, it comes back. But I saw some sort of disclaimer either there or on Reddit or something where Google says, humans may review some chat content. And like, it's pretty prominent. I think it says that sometimes in the box where you type the prompt. So it's unmissable, right?

23:08Emanuel:That's like a more straightforward way to disclose it. Speaking of disclosures, We did find disclosures in OpenAI's documentation that says they review chats for safety issues. But this is different. Yeah, yeah, yeah. So the safety issue is one that's been known for a while. That's like being part of the news cycle a few times now. Of course, when unfortunately a series of people died by suicide, as Sam said, with the 4.0 model. various measures came into force where chat gpt will proactively monitor chats for you know dangerous behavior problematic behavior something where somebody's safety can could be at risk you know and that could be escalated proactively then a human may review it and then maybe they'll take action there there's other cases as well where somebody you know maybe trying to plan a mass shooting or something and that they're conversing with chat gpt to do that and you know that will get proactively monitored then raised to a human and then escalator and that sort of thing so like human review has been part of chat gpt's sort of pipeline and workflow for a while and in like some pretty crucial context you know where like i think you probably want a platform as popular as this to have some sort of safety monitoring, right?

24:41So people aren't going to do a bunch of very horrible, dangerous things and plan it on the platform, that sort of thing. And there's also... There was a time investigation a while ago that was really good that found, you know, Kenyan workers working for$2 were doing trade or were helping with data annotation of, hey, here's a bunch of text, label this, and then we'll try to make the platform less toxic. that sort of thing. All of those things are interesting and have been reported. This is different. The stuff we're reporting is different because it is firstly real chat GPT user prompts. As far as I know, the Kenyan stuff was like third party text that was being annotated.

25:23This is real chat GPT prompts. And this isn't about safety. This isn't about stopping a mass shooting or alerting the FBI if somebody is about to do a really serious crime, they're planning on chat GPT. This is just to make the models better and to make the responses from chat GPT better in OpenAI's eyes. So presumably more people will use the product. It is different. And I found a lot of people getting sort of confused by that. I obviously think it's quite clear, but this is a different thing that's going on. And that's one of the reasons we're reporting it.

25:57Emanuel:Can we opt out of this? That's the really interesting thing to me, is that beyond the sort of disclosure that this may be happening, being really buried on OpenAI's website, is also how it is handled opt-in or opt-out-wise. So if you go into your ChatGPT settings, if you use ChatGPT, obviously the best way to avoid this is to not use an LLM. Because I should say, Anthropic confirmed to us that they're basically doing the same thing. for Claude as well. It's just we happen to get the OpenAI-related documents, and that's why the piece is focused on them, obviously. But if you go into your ChatGPT settings, there is this one called improve the model for everyone.

26:43And a little disclaimer on that says, allow your content to be used to train our models, which makes ChatGPT better for you and everyone who uses it. We take steps to protect your privacy. That's a very normal sort of disclaimer, right? where it's like, oh, hey, do you want to share analytical data with us? You see that when you boot any game on PlayStation or Steam nowadays, right? It's like, hey, share telemetry with us and we'll improve it or whatever. Very similar to that sort of thing. Two things. This setting does not say that humans may be reading your chat GPT prompts. It doesn't say that.