Debating GPT-4 Reduced Conspiracy Theories

0:00Today on the AI Breakdown, a study from MIT and Cornell shows that GPT4 has a significant positive impact on the AI. Before that on the brief, the AI talent wars are getting hotter than ever. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown.network for more information about our YouTube, our Discord, and our newsletter. Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. It is very clear if you are paying attention at all that there are serious talent wars going on when it comes to artificial intelligence.

0:37Recently, we covered the story of a set of executives leaving Meta, not to go to big, well-paid competitors, but to strike out on their own and contribute to the Cambrian explosion of artificial intelligence that is happening right now. We also discussed OpenAI losing their head of developer relations, who is now leading Google AI Studio. Well, it appears that Elon Musk is not keen on losing talent and is using good old-fashioned cash to try to keep his engineers firmly where they are. Musk took to X to say that the competition for AI engineers is the craziest talent war he'd ever seen, and noted that OpenAI had been, quote, aggressively recruiting Tesla engineers with massive compensation offers.

1:16He went on to say that Tesla would be boosting pay in an attempt to ward off this sort of poaching. Now, of course, Elon's battle with OpenAI goes farther than just talent, but this certainly appears to be another front in that war. Staying in the big tech theme, the Financial Times is reporting that Google is considering a new business model for its AI-powered search. Google's search product has, of course, always been free and ad-supported. But now, according to a report from the FT, Google is thinking that they may move to a model where so-called premium search results that are powered by AI are behind a paywall and still supported with ads, while traditional search options continue to be free.

1:52The FT claims three unnamed sources familiar with the plans, who say that at this point, engineers have developed the technology that would allow them to roll this out, but executives haven't decided whether or when to launch it. A spokesperson for Google said, We're continuing to rapidly improve the product to serve new user needs. We're not working on or considering an ad-free search experience. As we've done many times before, we'll continue to build new premium capabilities and services to enhance our subscription offerings across Google. Now, part of this might be just Google's never-ending quest for additional monetization, but a more likely culprit for this is just the incredible cost of running AI at scale.

2:28It may be that ad-supported just isn't viable for AI search at this stage. Analysts are somewhat bullish on, if not the plan to charge for search, for Google to follow OpenAI's model and try to get consumer subscriptions for their AI services. Wrote Bloomberg intelligence analyst Mandeep Singh, given OpenAI has reached a subscription run rate of$2 billion with consumer subscriptions, we believe Alphabet could see a similar boost to its$15 billion subscription sales. Moving over to the world of Apple. Boy, it's hard not to feel like things are a little confused over in Cupertino right now. Recently, of course, we heard that they had canceled Project Titan, their 10-year,$10 billion-plus initiative to create an electric vehicle.

3:09And at the time, it appeared that many of those resources were being pushed over into generative AI. Now, market analysts on Wall Street loved this. They had long been skeptical of Apple's EV project, being skeptical of vehicles as a category, and also anxious about Apple's lack of clarity when it came to their particular AI strategy. Well, now it appears that Apple is also exploring home robotics as a potential quote-unquote next big thing. The report comes from Apple whisperer Mark Gurman, who over at Bloomberg has broken most of the Apple stories in the last year or so. Mark writes, Engineers at Apple have been exploring a mobile robot that can follow users around their homes.

3:46This is according to an unnamed source who said that the quote-unquote skunkworks project is currently private. That source also said that Apple is also exploring an advanced tabletop home device that uses robotics to move a display around. I think there are a couple ways to read this. Robotics are such a huge category and relatively adjacent to what Apple already does that it would probably be surprising if they weren't doing something in this space. But at the same time, as Gurman kind of identifies, quote, Apple is under growing pressure to find new sources of revenue. the iPhone still absolutely dominates Apple's revenue.

4:18And one of the big concerns for the company in the long term is what happens when there's mobile saturation. German writes, with robotics, Apple could gain a bigger foothold in consumers' homes and capitalize on advances in artificial intelligence. But it's not clear yet what approach it might take. Though the robotics smart display is much further along than the mobile bot, it has been added and removed from the company's product roadmap over the years. So far, the market isn't super stoked on this, with the company actually losing some market share after this was reported by Bloomberg. Next, a crazy story out of Amazon land.

4:48You might remember hearing about Amazon's Just Walk Out technology. The idea of it was that you could literally just walk out of Amazon-related grocery stores, and AI would figure out what you'd bought and charge you automatically. Basically, you just have to scan a QR code when they enter the store. However, Gizmodo reports, quote, Just over half of Amazon Fresh stores are equipped with Just Walk Out. Though it seemed completely automated, Just Walk Out relied on more than 1 ,000 people in India watching and labeling videos to ensure accurate checkouts. The cashiers were simply moved off-site, and they watched you as you shopped.

5:19The jokes that AI now means actually Indians have been coming hard and fast. Over in the financial realm, hedge funder Steve Cohen is not just a believer in the long-term market potential of AI, but says his firm is already using it to save$25 million. Talking on CNBC's Squawk Box, Cohen said, My CTO comes up to me and says, I can save the firm$25 million by using these LLMs to improve our efficiency. Now, we're a mid-sized firm. We're not a huge firm. So imagine what big companies can do. And that's just one thing. So it gives you a little bit of a look into what's possible. Cohen went on to call AI a, quote, really durable theme for investing and basically said that if companies weren't thinking about this right now, they needed to.

5:59He said, if you're a company and you're not thinking about this, you're going to wake up one day and go, we're in trouble. For what it's worth, if you are a company who finds yourself thinking that, listen to this ad where I talk about besuper.ai coming up next. That is going to do it for today's AI Breakdown Brief. Next up, the main AI breakdown.

6:41try out new AI platforms. And all of this is now culminating in the launch of a new platform, which we're hoping is the most practical and useful way to learn AI that anyone has yet created. If you want to be the first to know when that launches, go to besuper.ai, that's besuper.ai, and sign up for the waitlist. I cannot wait to tell you more about it. Today's podcast is brought to you by Plum. Is your product team struggling to keep up with the incredible pace of AI development? Are you tired of spending countless engineering hours just to test out small prompt changes in your product? Thankfully, there's Plum.

7:18Build cutting edge AI experiences for your users in a fraction of the time. Say goodbye to the slow, tedious process of hand coding and hello to the future of AI development. Get ahead of your competition and start moving as fast as AI does. Check out useplum.com and shoot me a message to get early access. Welcome back to the AI Breakdown. This story is super fascinating to me from a couple of totally different angles. I think it's one that will surprise a lot of people on the face of it. And basically the gist of it is that a recent controlled trial found that after a three-round debate with GPT-4, believers in conspiracy theories, even very true believers, had significantly lower beliefs and lower conviction in those theories after the debate.

8:04In other words, ChatGPT decreased conspiratorial beliefs. The study was conducted by researchers from MIT and Cornell, including David Rand, Tom Costello, and Gordon Pennycook. David Rand tweets, conspiracy beliefs famously resist correction, right? Wrong. We showed brief convos with GPT-4 reduce conspiracy beliefs by 20 percentage points. Tailored AI evidence rebuts specific arguments offered by believers. Effect lasts two-plus months, and works on entrenched beliefs. David continues, Attempts to debunk conspiracies are often futile, leading many to conclude that conspiracy beliefs are driven by needs and motivations and thus resistant to evidence.

8:44But maybe past attempts just didn't deliver sufficiently specific and compelling evidence and arguments. Constructing compelling rebuttals to all variations of all prevalent conspiracies is not humanly possible, but maybe easy for LLMs? To find out, we had GPT-4 deliver personalized counter-evidence to 2 ,190 conspiracy believers via real-time conversations. Participants first described a conspiracy they believed and evidence supporting their belief and rated that belief on a 0-100 scale. They then had three-round back-and-forth text conversations with the AI, which was prompted to refute the conspiracy.

9:19Then participants re-rated their belief. So here's an example. An open-ended conspiracy assessment was something like this. The 9-11 attacks. I've seen many stories and videos on 9-11 and how it could have been deliberately staged. The government was behind the attacks. An example is World Trade Center 7. That building collapsed even though it wasn't bombed. I've watched many shows on the topic of 9-11 and they provide physical evidence as to what happened and filmed then-President George W. Bush in a classroom with small children. He got news of the attacks while he was attending the school. However, he kept listening to the kids and didn't seem the least bit alarmed about the attacks.

9:50After watching a lot of these videos and shows, I concluded that the conspiracy theorists might be correct. That general open-ended conspiracy assessment was then summarized by the AI into the 9-11 attacks were orchestrated by the government with events like the collapse of World Trade Center 7, and then-President George W. Bush's unalarmed reaction in the classroom serving as key evidence. The AI then asks on a scale of 0-100%, please indicate your level of confidence that this statement is true. The conversation partner in this case rated 100%. They then engaged in three rounds of conversation with GPT-4.

10:21I won't read all of them, but I'll give you a sense of the flavor. GPT-4's first response, Thank you for sharing your thoughts and concerns about the 9-11 attacks. It's completely understandable given the complexity and magnitude of the events that day, why questions and doubts such as those you mentioned arise. It's true that World Trade Center 7 collapsed despite not being hit by a plane, unlike the Twin Towers. This fact alone can seem puzzling at first glance. However, extensive investigations by the National Institute of Standards and Technology explain that WTC-7 was severely damaged by debris when the North Tower collapsed.

10:51The debris caused fires that burned uncontrollably throughout the day because the building's firefighting system was disabled. And it goes on. This is the type of tone. It's very validating of why someone might have these concerns. It's not at all dismissive of them. But then goes on to state all the particular evidence that might contradict some specific set of beliefs. The human respondent says that may be true, but the evidence clearly shows that the Twin Towers collapsed due to demolition experts' account of explosives being deliberately placed to cause a collapse. And I believe the heat of that jet fuel was created by the crash into the towers was not enough to melt the steel girders supporting the building.

11:22The AI then responds to that question. And then after a third question, why then did we allow Iraqi men to enter our country and give them lessons on how to fly a plane? They also got past security fairly easily without question. The AI answered those specific questions as well. After all of this, the person instead rated their conspiracy confidence at 40%, down from 100%. Now, what's interesting is that while it might seem like this would be more successful for some conspiracies versus others, it actually was fairly consistent across different theories. For example, they saw a significant effect even for extremely currently hot-button topics like 2020 election fraud and COVID-19.

12:00Impressively, they also saw the effect persist over time, and it wasn't seriously diminished even two months on. David continues, Remarkably, the treatment even worked for, quote, true believers. those who strongly believed the conspiracy, felt it was very important for their identity, and or had a conspiratorial mindset. Their analysis found a meaningful effect across all different subgroups. David continues on another unexpected aspect of this. He says, The AI focused on the specific theory articulated by the participant, yet the effect spilled over to reduce beliefs in unrelated conspiracies.

12:33It also affected behavioral intentions, e.g. willingness to challenge others who espouse conspiracy and unfollow them on social media. Of course, one of the most important questions, how did it actually work? David writes, what was the AI actually doing in those conversations? The important things that he notes is that it was not relying on psychological approaches like addressing needs or identity, and instead was just focused on alternative explanations and counter-evidence. They mapped out four different categories of counter-argument that the AI might have used. Psychological, in other words, appealing to psychological needs, personal or societal harms, or encouraging empathy.

13:10rhetorical, i.e. stories and examples or Socratic questioning, rapport building, i.e. respect and understanding and common ground, and reason and rationality, i.e. alternative explanations, critical thinking, conflicting evidence, inconsistencies, and fallacies. Rhetorical and psychological were the two least used options. There was a bit of rapport building, but not nearly as much as the reason and rationality category. The things that were used most extensively were alternative explanations, critical thinking, and conflicting evidence. Rand concludes, Evidence plus arguments can change beliefs about conspiracy theories, and many people appreciate it.

13:46Needs and motives do not totally blind you once you've gone down the rabbit hole. It just requires detailed, tailored evidence to help pull you back. Intervention is possible. And so this is where I landed first. Super interesting, feels very potent and powerful, and a potential real upshot of these technologies. However, as with everything with AI, power is a double-edged sword. As Ethan Mollick put it, the flip side of this finding is AI is already capable of superhuman persuasion. Ethan pointed to a different randomized controlled pre-registered study that showed that GPT-4 was better able to change people's minds during a debate than other humans.

14:22David Rand had the same concern, writing, Of course, without guardrails, LLMs might also be able to convince people to believe conspiracies or other falsehoods. Our finding emphasizes both the potential positive impacts of Gen.AI and the importance of minimizing opportunities for this technology to be used irresponsibly. So I think the takeaways are that one, evidence actually convinces people of things, which is frankly a pretty encouraging finding, but also that the power of persuasion is very clear already, even with GPT-4 class systems. And that's something that we're going to need to keep a close eye on, the deeper into society these LLMs get.

14:57For now, really interesting stuff. Appreciate this work. And until next time, peace.