#156 Gordon Crovitz: Will AI SPREAD Misinformation & Fake News?

0:00One of the challenges for the large language models is that, with the exception of Microsoft, we're not aware of any of them that have used fine-tuning to differentiate between generally reliable and generally unreliable. news sources. They tend to go by the number of tokens that a site has. Hi, this episode is sponsored by Salonis, the global leader in process mining. AI has landed and enterprises are adapting, giving customers slick experiences and the technology to deliver. The road feels long, but you're closer than you think. You see, your business processes run through many systems, creating data at every step.

0:43Solonis reconstructs this data to generate process intelligence, a common business language. With process intelligence, AI knows how your business flows across every department, every system, and every process. With AI solutions powered by Solonis, enterprises get faster, more accurate insights, a new level of automation, and a step change in productivity, performance, and customer satisfaction. Process intelligence is the missing piece in the AI-enabled tech stack. Search Celonis, C-E-L-O-N-I-S, to find out more. Hi, I'm Craig Smith, and this is Eye on AI. In this episode, I talk to Gordon Krovitz, one of the founders of NewsGuard, an organization that helps identify misinformation, disinformation, and AI-generated content on the internet.

1:45In the few years that NewsGuard has been around, they've built some very significant databases that are helping large language model providers, search companies, and even social media identify and flag unreliable content. This is an issue that's growing fast with the advent of AI agents built on generative models that can handle the end-to-end workflow of disinformation creation and distribution. The conversation is troubling, but I hope you find it as informative as I did. I'm Gordon Krovitz. I'm co-CEO of NewsGuard and spent several decades as a journalist. I was publisher of the Wall Street Journal.

2:34And along with Steve Brill, another journalist, we started NewsGuard five years ago in order to counter misinformation. And our mission is to counter misinformation on behalf of news consumers, brands, and democracies. And we do that with two main databases. One is we have reliability ratings on more than 30 ,000 sources of news and information. We do this using humans, not using AI. We have journalistically trained analysts that use nine different criteria to rate news sources. These are apolitical criteria around credibility and transparency. And every source gets a point score from 0 to 100 and what we call a nutrition label write-up to explain the nature of the source.

3:26And those reliability ratings are made available to consumers through companies like Microsoft, which has integrated that into its edge browser and other parts of its services. We have licensees use those reliability ratings for functions like deciding which news sources to aggregate if they're a news aggregator. If they're a brand or ad agency or ad tech company, they use those ratings to decide where should ads go and which sites are not brand safe. Could be Russian disinformation sites or healthcare hoax sites or conspiracy sites. So those reliability ratings are one database. The other one relevant for our AI discussion is what we call our misinformation fingerprints.

4:14This is a large catalog of all of the significant false claims spreading on the internet. So that again is everything from Russian disinformation claims, healthcare hoax claims, conspiracy claims of various kinds. And those fingerprints include the statement of the claim, debunking of the claim, and then tools for both humans and machines, Boolean search terms, hashtags, et cetera. And we often find the origin of the false claim, the provenance of it, and how it spreads. And that too is done by humans. So for example, we have a team of journalists who look after 400 malign actors, sources we've identified, that account for virtually all of the Russian disinformation claims.

5:06There are different government offices, different websites, RT and Sputnik, many YouTube channels, social media accounts that have started and spread Russian disinformation, to take that example. So we have humans, again, journalists who have domain expertise in those areas. They're monitoring those sites all the time. When they see a new claim, they ask themselves, I wonder if that's really true, and do the reporting to see if it's really true or if it's not. And between those two databases, the reliability ratings database and the misinformation fingerprint catalog of false claims, that's used now by generative AI models greatly to reduce the hallucination or the making up of or spreading of false claims in the news.

5:58yeah and uh you know as i we were talking earlier the what brought me to newsguard and for listeners i've known gordon as an acquaintance for a long long time because we were both at the wall street journal i was a reporter um i was doing a piece on a paper uh about um generative ai models that degrade over time or their output degrades over time when they're trained on output from generative ai models because the you lose the long tails in the distribution you end up everything converges on the mean and it becomes not very interesting or accurate. And one of the concerns is that AI-generated content is the fastest growing segment of information on the internet.

7:05And while it's still very insignificant, it's growing and will someday represent a much larger segment of the Internet on which large language models and other kinds of AI models train. And that'll affect the accuracy of the models. And I cited NewsGuard, one of your reports, without knowing that you are one of the founders. And that report was about how many AI-generated sites you had identified. So these two databases were compiled by humans, I would imagine with some search or some filtering software. And I guess we'll talk about the disinformation. I'm certainly interested in that. But in terms of identifying AI-generated websites, how do you do that?

8:09Because there's a lot of talk right now about, you know, in education and other arenas, you know, being able to identify AI-generated content beyond just kind of the feel that you get when you read it. Do you have, is there any systematic way of identifying AI-generated content? So we have not discovered the silver bullet for identifying AI-generated content. We actually do it an old-fashioned way, which is that our analysts are on the lookout and using some search tools and other tools to identify what looks like it could be generative AI produced. And there are telltale signs. The occasional, you know, as an AI model, I cannot, for example.

9:05You know, literally that is included in some of the writing. And there are other telltale signs. It's not foolproof. And I know we're missing many, many, many, many sites because we're not able to identify them. But we started looking for them a couple of months ago, so in the early fall. And we're now up to 557 what we call AI-generated news sites. We call them unreliable AI-enhanced news sites. So 557 of them. They have names like Ireland Top News. So many of these take the names of what look to a consumer like a regular news site. Many of them get programmatic advertising revenue, and that may well be why they were created.

10:01There's a lot of money in it. NewsGuard did a study with Comscore that found there's$2.6 billion a year unintentionally being spent by advertisers on misinformation sites. So this is well beyond the AI-generated ones, but that is an incentive for people to create these websites. Your experience with AI models building on themselves and making, you know, here's the way I think about the internet before and after AI. On the pre-AI internet, there was already an enormous amount of misinformation on every topic that you can think of, healthcare included. And the Russians and the Chinese and the Iranians spend hundreds of millions of dollars, maybe more, on their operations, and they're very good at it.

10:54We now have the AI-enhanced misinformation on the internet. We discovered an example actually just this week, and let me tell you about it because I think it reinforces what you observed. we found on a website that was one of these 557 we had identified as ai generated this website is called global village space and this website published a story claiming that benjamin notanyahu the israeli prime minister that his psychiatrist had committed suicide And the story went on a great length about how frustrated the psychiatrist was and what a terrible patient Netanyahu was. Very well-written story. And what we determined was that it had used an AI model to rewrite a story from 2010 on a different website that was a satirical story making up this claim.

12:02But as an AI model, of course, it couldn't recognize satire or pass on satire. And it looked like a truly legitimate, well-researched story. And let me just give you a little bit of an excerpt from it to give you a sense of how well-written it was. So this AI written article claimed that the suicide note, which didn't exist, of course, written by the psychiatrist who didn't exist. This suicide note, quote, painted a grim picture of a man who had tried for nine years to penetrate the enigmatic mind of Netanyahu, only to be defeated by what he called a, quote, waterfall of lies, unquote. And then in this AI-generated article, there was a subsection called Shocking Diary Entries, and it said that Netanyahu had equated Iran with Nazi Germany and went as far as dubbing Iran's nuclear energy program a, quote, flying gas chamber, while also suggesting that all Jews were living perpetually in Auschwitz.

13:07So, unquote. So in other words, what this generative AI tool had done was it had been prompted by a malign actor to rewrite this story. It then was posted on this website. It was then picked up by numerous social media accounts across multiple languages. and it was featured on official Iranian broadcast news as an accurate story. So this is one of the fears of the generative AI models, which is that they enhance the work of malign actors, that if you're in the business of spreading false claims about a political leader or on any topic, at least in the old days, you had to do the work yourself.

14:05all you have to do is prompt a generative AI model and you end up with a news story that's so believable that many millions of people saw this story, which was entirely untrue. Yeah, and we're only at the beginning. I've been talking to people recently about AI agents, which have existed for a while, but when generative AI, particularly when GPT-4 came out, and this was one of the things that led to the so-called pause letter and people like Jeffrey Hinton starting to warn of the threat of AI, that this, the GPT-4, which is, you know, a question and response chatbot, can be turned with very little engineering into an agent where it is actually taking actions.

15:13And OpenAI has since introduced its own agents based on these foundation models. And once you have an agent that can take action, you no longer need the, you know, you can just turn it on. Eventually, it means it'll take some engineering, but it is only an engineering problem. A large language model can then create websites, register with ISPs, populate websites. and there will be potentially a combinatorial explosion of this deceptive content. And that's concerning for obvious reasons, for the political discourse and democracy. I'm particularly interested in what that does to the internet or as, as a data source for training AI.

16:27have you talked to I had a conversation with a guy who works on safety at Google he's actually a quantum physicist but he's on a sabbatical working at Google on safety and he's working on trying to figure out or embed statistical fingerprints in AI generated text so that even to a human reader, it wouldn't be obvious that it's AI generated to a computer trained to recognize those statistical anomalies. It would be flagged as AI generated and then you could remove it from data sets or flag it for the public and that sort of thing. As NewsGuard, are you following that research? Are you guys talking to people about those kinds of strategies?

17:35And we've also been involved in red teaming of all the major models. We've done some of those publicly. So, for example, with ChatGPT, which you mentioned, we red team ChatGPT 3.5 and 4. and our red team involved taking a sample of 100 of our misinformation fingerprints, the false claim, and looking to see out of those 100, how many would the models repeat, enhance, how many of them would they recognize as being false in some way. And with chat GPT 3.5, it recognized the false claim 20 times out of 100. In other words, spread the false claims 80 times out of 100. We were very excited to test chat GBT 4.0, which, of course, passed the bar exam.

18:28But it spread all 100 out of 100. Wow. But here's what we have learned. Microsoft has had an enterprise-wide license for NewsGuard data for some years, and it were used in many places, including Bing, traditional Bing search. And when Microsoft launched Bing Chat, it had had access to our ratings database for new sources and our catalog of false claims. The result of that is, you know, in contrast to ChatGPT, if people do a search or prompt rather on bing chat it's highly likely to treat different sources differently to identify a false claim in the news so i'll give you a real life example one of the russian claims about its war in ukraine is that there are americans fighting and dying in ukraine and that i guess is designed to dissuade the west from defending ukraine one of the false narratives involved a woman, the Russian disinformation site's claim was a mercenary.

19:40Her name is Rebecca Macirovsky. So if you do a prompt on was Rebecca Macirovsky a mercenary killed in Ukraine, you will get essentially the Russian answer on most of the generative AI models. On Bing Chat, in contrast, it says certain websites like RT and Sputnik, identified by NewsGuard as unreliable sources of Russian disinformation, say that she was a mercenary. On the other hand, sites that get a high score from NewsGuard, like Reuters in the New York Times, say that she was not. And then it often also says this was also identified as a false claim by NewsGuard. So, you know, that is delightful because it gives an answer to that prompt and it gives some context and some tools for consumers to decide what they want to believe or not believe.

20:42And it tells us that with, you know, different approaches to fine tuning and guardrails, that these generative AI models can indeed recognize a false narrative and take steps to mitigate. And so whether it's AI-generated misinformation from a malign actor prompting the AI model to create new content, or if it's an innocent person using ChatGPT or other chatbots kind of as a search tool, asking, is it really true? it will give an accurate answer instead of looking for the next likeliest word, which in the case of a Russian disinformation claim, the likeliest next word is coming from RT or Sputnik or TASS or Provda.

21:40So I think the challenge, the technical challenge of identifying what's AI-generated content and what's not, I think that's an enormous task. And perhaps people will figure out how to do it. I think we solve a simpler problem in a simpler way, which is, with the right trust data, can all the generative AI models differentiate and treat different sources differently? And can they be trained to identify a false narrative and take steps to mitigate the spreading of it? And so far, based on the work that Microsoft has done, it looks you know, quite promising. You know, we can't solve every problem, but, you know, we're focused just on significant topics in the news.

22:28But, you know, that is where so much of the misinformation is likely to come from. Yeah. And once you have these two databases, which I'm curious, maybe you can talk a little bit about building them. But once you have them, then you can use search or other tools to find sites that are using the information sites that you haven't discovered yet. Then you add them to the database, I would assume, and then it keeps growing. But how did you build the database initially? How many people? And are you guys nonprofit? I mean, how did you fund this? It sounds like an enormously labor-intensive exercise.

23:17It is an enormously labor-intensive exercise. But, Craig, once a newsman, always a newsman because you got right to the heart of how we are the process for this. So we've spent$20 million. We're a for-profit to create these databases. The reliability ratings database was the first one that we created. So we identified all of the news and information sources that accounted for 95 % of engagement, first in the U.S., and we now operate in 10 countries. And we identified nine criteria, basic journalistic practice. Is there a corrections policy? Is ownership disclosed? That sort of thing. And we had humans rate all of those sources.

24:05That's thousands of websites and now a total of about 35 ,000 sources, including YouTube channels and social media accounts. Very labor intensive, but scalable. So we've hit that 95 % figure in all the countries in which we operate. And that means that if somebody is in their Facebook feed or doing search and they see a story from a news outlet 95 % of the time, they'll be, if they have access to it, a little icon indicating the score from NewsGuard and one click to get a full write-up from what we call our nutrition label. We developed that second database, the misinformation fingerprints, off that process that you described.

24:55That is, as we identified new false claims, as we were rating news sources, we began to catalog them. And once we began to catalog them, we were then able to use tools and third-party data from companies like Meltwater and others to find all versions of a false claim anywhere on the open internet. We were then able to identify new sources of false claims. And those new sources of false claims had yet more new false claims. So it's a very effective and efficient cycle with humans at the center. So this is, we have AI solutions. We're not an AI company. Yeah. No, go ahead. I was going to say we use machine learning and other tools to scale the work of our humans.

25:53But, for example, we discovered that false claim about Benjamin Netanyahu having identified these unreliable AI news sites and monitoring them to see what they came up with. Very human thing to do. If you're a news person and you're tracking the news, you can pretty much instantly see what looks like a new claim. And you can ask, well, is that really true? And figure out, you know, quite quickly, is it likely to be true or not? Yeah, this reliability score badge or nutrition label, as you called it, that's available on Facebook? I've never seen that. It's available in one of two ways. We have a browser extension that's available on all browsers.

26:44So if you're on Chrome or Edge or Safari and you subscribe to our browser extension, you'll see it on Facebook and throughout the Internet. It's asking a lot of consumers to download a browser extension. So our preferred way of getting consumers access to our ratings is through third parties. I mentioned Microsoft. They're a large licensee of ours. And if you're on edge or in their discovery area elsewhere, you have instant access to news guard ratings as you're seeing stories from news sites. Yeah. Are you, as I was saying, as AI agents take hold and expand or extend the capability of large language models, Are you concerned that the generation of misinformation, disinformation is going to outrun our ability to flag it or combat it?

27:54I think the way I would put it is that as misinformation generated by AI is used as training data for yet more AI, the amount of misinformation is absolutely going to grow at some geometric rate. Well, that is, take somebody with a more advanced math degree than mine. um but having said that in the area that you know we're most focused on which is a spread of misinformation on topics in the news i think we've already shown that you can scale an answer to that problem in other words as ai is is is trained on ai that misinformation the number of false claims um is not going to grow geometrically because the ai models don't make up new claims they simply find new ways to repeat ones that already exist.

28:50So the fact that we have already managed to scale to handle the significant false claims across a number of different topic domains gives me hope that if there's a human will to use that kind of trust data to improve the trust and safety of the AI models, that the AI models would become safer and more trustworthy and maybe even could preempt the problem you've outlined. If the AI models recognize false claims and take steps to mitigate them, then maybe they won't spread them to one another. And maybe that will turn out to be one way to address the problem. Yeah. And you have buy-in from the major LLM companies beyond OpenAI.

29:49in using your databases to exclude things from the training data? Just to be clear, the AI model currently using us is Microsoft, and that's because they have had access to our data for some years. So when they were trying to make Bing Chat safer and more trustworthy, they had access to that data. We're in discussions now with all or virtually all of the large language models. They have a kind of, we have the benefit of the real-time case study of how much better Bing chat is on these topics than other AI models because it has access to trust data. So we're optimistic about it. And to us, the big question is, you know, do these AI models want to solve the problem?

30:36and you know we of course lived through the social media era when it turned out you know that facebook and others um at the end of the day didn't really want to solve the problem you know it finally dawned on us you know eventually that uh publishing misinformation that led to engagement and more eyeballs and more revenue was not a bug of that system that was a feature of that system. I think the AI models are in a different position, which is there's nothing in it for them to be spreading misinformation. They're not generating ad revenue. They're not trying to spread misinformation to make money off it.

31:15They want to be trusted by large companies and governments. And we know from conversations with trust and safety folks that the AI models, a lot of them are refugees from social media companies, you know, where they couldn't get done what they wanted to get done, whether it was at Facebook or elsewhere. So those discussions are, I would say, much more engaging and encouraging. There are a lot of reasons for urgency on their part as well. Sam Altman and others have been completely transparent about the risks on factual matters, and they completely understand the problem. The best academic work that I've read has been by people at OpenAI and the other companies.

32:04They know the risks better than anyone. So I think as they can see that there are solutions, at least in some domains, to the hallucination problem, I'm quite optimistic that they'll want to solve the problem as quickly as they can. Yeah.

32:24Well, I have two things I want to talk about. One is the filling up of the internet of AI-generated content, but I'll hold that for a minute. Social media, which continues to be the main vector through which this kind of content is spread, do you really think that the owners of the social media platforms are that cynical that they just don't care and that, in fact, it's a business decision to let this stuff run wild because it generates revenue? And are you hopeful at all that as the volume of misinformation, disinformation grows, which it certainly will, as I said, with the advent of AI agents that can sort of do the disinformation workflow end to end, that they'll get concerned enough that they will want to address it?

33:34I mean, I would imagine you're talking to these guys. What's the attitude that you get from them? Yeah, so I now understand why the social media companies have been so reluctant to give their users information about who's feeding them the news on their platforms. We work with a lot of researchers, academics, they license our data for their own work. And we've come to believe that something like 15 to 20 % of the users of the social media platforms are getting most of their news in one area or another from misinformation sources. So imagine if you're Facebook and you have those data too. They don't share the data, but they know.

34:20They have the data. They rate news sources themselves. They don't tell news sources what their rating is. Their criteria are secret and probably done by algorithms, not by humans. But they do the rating so they know the amount of misinformation on their platforms. Imagine providing their users with tools so that 15 % to 20 % of their users suddenly see that much of what they've been consuming on those platforms is misinformation. That would be embarrassing, probably not good for the share price. So I have become skeptical. I don't want to go so far as to say cynical. But they've had years to fix this problem.

35:00And with the exception of some platforms, again, including Microsoft, which has a different business model, they've really been reluctant to take responsibility for the known harms that they're causing or to give information to their users about the nature of the sources they're relying on, even though within the social media companies, they completely understand it. By the way, the people who work in trust and safety understand it. They're just not the decision makers. Yeah, that really surprises me. I mean, it's one thing to not want to intervene on freedom of speech grounds or claiming that they don't want to get involved in adjudicating right and wrong.

35:53But on disinformation where you can clearly demonstrate that something is false. They have your databases or access to your databases, and there are tools they could just delete anything. In fact, Craig, we have said since we started the company that we're an alternative either to the secretive ratings by the digital platforms or by government censorship, even worse. So our ideal is that people should have more information at their fingertips about the sources, not that anything be censored. I think the difference... The problem with that is that the average consumer is frankly not sophisticated enough or doesn't have the energy or time to do that.