Study: Reinforcement Learning from AI Feedback Performs As Well As Human Feedback

0:01Today on the AI Breakdown, we're looking at a new research paper that suggests that reinforcement learning using AI could perform as well as reinforcement learning using human feedback. Before that on the brief, the first AI COVID drug heads to phase one trials. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our newsletter, our YouTube, and our Discord. Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. We kick off today with a story that I think reflects something that we're going to hear a lot more about in the future, which is AI-generated pharmaceuticals.

0:41Fox News reports that a new AI-generated COVID drug has entered phase one clinical trials. So in terms of the drug itself, this is an alternative to Paxlovid. In other words, it is an oral medication that you take after you've gotten COVID, not a vaccine that prevents it in the first place. The company behind the treatment is called Incilico Medicine, which is based both in Hong Kong and in New York City. The drug is referred to as ISM-3312, and in their early trials, the company says it has some advantages over Paxlovid. That includes being effective against variants that are resistant to Paxlovid, being more stable and working for a longer period of time, and not having the Paxlovid rebound, which as Fox describes, is when patients recover from COVID and test negative, but then test positive again a short time later.

1:23Here's how the article sums up the process of creation for Insilico. Quote, to create its new drug, Insilico's research team first used its target discovery platform, Pandaomics, to identify the target protein within the coronavirus. Next, it used its in-house generative chemistry platform, Chemistry42, to generate new molecules that would attack the protein as a means of treating COVID and other coronaviruses. A phase one study means that ISM332 is now being evaluated in healthy volunteers, and the expectation is that the results of the trials will be released at the end of this year. Said the company's CEO, Generative AI offers us a powerful tool for accelerating the drug discovery process and allows us to quickly identify new solutions that we hope can provide more potent defenses against mutating COVID strains and prevent another pandemic.

2:08Now, this is definitely a theme we're seeing more and more. A couple weeks ago, Vox wrote a piece, AI discovered drugs will be for sale sooner than you think. Among other things, the piece discusses just how much faster AI models can be than traditional drug discovery processes, which typically can take up to 12 years or more. Next up, we move to a follow-up from last week. In the wake of approvals from the Chinese government, the companies that have received those approvals are mass-releasing a huge number of AI models and applications. Baidu is among the companies that have released their LLMs following last week's announcement, and Reuters reports that 360 Security Technology and iFlytech have just launched their own AIs, including voice recognition technology.

2:50In our next story, unions in Las Vegas are gearing up for a battle around AI. NPR's All Things Considered covered how quickly AI is infiltrating Sin City, noting that recent studies have suggested that between 38 and 65 % of jobs in Las Vegas could be automated by 2035. Many commentators believe that these shifts are fairly inevitable. John Restrepo, principal at RCG Economics, said, Wherever the resort industry can replace their workers and not affect productivity, profits, or the customer experience, wherever they can do that with artificial intelligence, they will. The question is, how do you factor in and how do you adapt your economic development strategy, your community strategy, your resiliency strategy to accommodate a world where certain jobs no longer exist?

3:33Now, one of the answers is, of course, to fight back. As NPR writes, unions in Las Vegas are closely watching the changes. The largest union in Nevada, the Culinary Union, represents 60 ,000 service and hospitality workers in Las Vegas and Reno. Later this year, it hopes to have a new negotiated contract that includes protections against AI replacing jobs. Ted Papageorge, who's the Secretary Treasurer of the Union, told NPR, quote, We had a huge fight about tech in our previous contract. We're going to have the same fight this time around. Promising bellicose action, he said, We'd like to say we're going to be able to get to an agreement, but if we have to, we're going to have a big fight and do whatever it takes, including a strike on technology.

4:10Now, obviously, we've been following the writer's strike and the Screen Actors Guild strike in Hollywood, and part of the arguments of the unions involved in those strikes is that this fight would be coming for other workers as well. The fact that the culinary union in Las Vegas is gearing up for just such a battle suggests that that may be an accurate assessment. Moving back to the world of AI and the consumer internet, a new plugin for ChatGPT is getting people excited in a way that most haven't. That plugin is Canva, the popular image creation and design tool and in some ways gives ChatGPT a new light multimodal feel.

4:43Decrypt writes, tapping into the broad and rapidly evolving social marketing space, OpenAI has unveiled a Canva plugin for its popular chatbot, ChatGPT. The strategic move aims to make the process of creating visuals such as logos, banners, and more even more simple for businesses and entrepreneurs. Now, speaking of AI-generated content, a new report suggests that 90 % of online content will be AI generated by just 2026. The report comes from Europol and is called Facing Reality, Law Enforcement, and the Challenge of Deepfakes. Now, the report is actually pretty interesting, even if you're not in law enforcement.

5:18And I think even if you see that 90 % of online content being synthetically generated by 2026 as perhaps a bit hyperbolic, it shows at least the scale and magnitude of the problem in many people's estimation. At the same time, it doesn't seem totally insane if only because content that is created by AI is so much cheaper and faster to produce than content that's created by humans. The question is whether this type of shift is completely inevitable and also what it will do for the value of content that isn't created by AI. Lastly, on that theme of content created by AI, Nunori is a completely AI-created influencer who has 400 ,000 followers on Instagram and has previously done fashion and lifestyle brand deals, and is now officially signed with Warner Music, which Nunori's team claims makes her the first AI pop singer.

6:04Forbes sums up the appeal for a company like Warner. Nunori won't get worn out from touring and promoting her music, and she can be restyled in seconds to keep in step with changing teen trends. And if she makes it to superstar status, she won't start making diva-like demands or demand an enormous pay raise. Given that ABBA's virtual tour is now making$2 million a week, which is a show produced entirely with virtual avatars and not the musicians themselves, It feels like this is the beginning, not the end of a trend. Anyways, friends, that is going to do it for today's AI Breakdown Brief. I'll be back soon with the main AI breakdown.

6:37Before we get into the main AI breakdown, I want to tell you about today's sponsor, Supermanage. If you work in a professional setting, you probably have some version of a one-on-one meeting, either with the people that work for you or the people that you work with. Unfortunately, all too often, those one-on-one meetings become glorified catch-up calls. Don't you wish you could jump right to the stuff that really matters? That's where Supermanage comes in. Supermanage AI magically distills your team's public Slack channels into a real-time brief on any employee, any time. Catch up on contributions, work in progress, challenges they're facing, sentiment, everything you need to show up ready for a truly meaningful conversation.

7:15And it's completely free. Visit supermanage.ai forward slash breakdown today to start making the most of your one-on-ones. And thanks again to Supermanage for sponsoring the AI Breakdown. Welcome back to the AI Breakdown. Today, we are digging into some new research that I think has some fairly interesting implications for the development of foundational models. Now, we're getting a little bit wonky in this one, but I actually think it's fairly significant and touches on not just model design and training processes, but AI ethics as well. Over the weekend, I started to notice people posting about research from Google called R-L-A-I-F, Scaling Reinforcement Learning from Human Feedback with AI Feedback.

7:59Now, at core here is a concept called Reinforcement Learning from Human Feedback. RLHF is one of the key techniques by which we align large language models with human preferences. Now, Hugging Face did a great blog last December explaining this concept. The entire blog post is worth a read, but here's how they saw ChatGPT explain RLHF. And keep in mind, this was an Explain It Like I'm 5 example. ChatGPT says, Imagine you have a robot named Rufus who wants to learn how to talk like a human. Rufus has a language model that helps him understand words and sentences. First, Rufus will say something using his language model.

8:34For example, he might say, I am a robot. Then a human will listen to what Rufus said and give him feedback on whether it sounded like a natural sentence a human would say. The human might say, that's not quite right, Rufus. Humans don't usually say, I am a robot. They might say, I'm a robot, or I am a machine. Rufus will take this feedback and use it to update his language model. He will try to say the sentence again using the new information he received from the human. This time he might say, I'm a robot. The human will listen again and give Rufus more feedback. This process will continue until Rufus can say sentences that sound natural to a human.

9:03Now, as this paper from Google points out, reinforcement learning from human feedback is one of the, quote, key drivers of success in modern conversational language models like ChatGPT and BARD. By training with reinforced learning, language models can be optimized on complex sequence-level objectives that are not easily differentiatable with traditional supervised fine-tuning, end quote. What the goal of this study was, was to compare reinforcement learning from human feedback versus reinforcement learning from artificial intelligence feedback. They write, in this work, we directly compare RLAIF against RLHF on the task of summarization.

9:39Given a text and two candidate responses, we assign a preference label using an off-the-shelf LLM. We then train a reward model on the LLM preferences with a contrastive loss. Finally, we fine-tune a policy model with reinforcement learning, using the reinforcement model to provide rewards. Now, fascinatingly, what they found is that RLAIF achieved comparable performance to RLHF measured in two different ways. First, compared to a supervised fine-tuning baseline that didn't have a reinforcement learning process, the results of both RLAIF and RLHF were preferred by humans a significant portion of the time.

10:14For RL-AIF, it was preferred 71 % of the time. For RL-HF, it was preferred 73 % of the time, which the researchers say is basically statistically equal. Now, when compared head-to-head, RL-HF and RL-AIF were both preferred 50 % of the time. In other words, they were preferred an equivalent amount of times by human judges. Now, there are a couple reasons why this is significant. As the researchers put it, quote, the need for high-quality human labels is an obstacle for scaling up RLHF. And one natural question is whether artificially generated labels can achieve comparable results. So first, there is just simply a question of scale.

10:49Any process that involves humans is inherently going to be less scalable, more time-consuming, more challenging, and more costly than a process where the humans can be replaced by artificial intelligence. Now, a second piece, though, which isn't really discussed in this paper as much, is the human cost and AI ethics questions of the actual reinforcement learning process. There have been a number of articles about this challenge that have come out over the last month or two. One in The Guardian was called, It's Destroyed Me Completely, Kenyan Moderators Decry Toll of Training of AI Models. The Guardian piece tells the story of Moffat Okinyi, who, as part of a previous job with ChatGPT, would view up to 700 text passages a day, many he said that depicted graphic sexual violence.

11:30From The Guardian, quote, he recalls he started avoiding people after having read texts about rapists, and found himself projecting paranoid narratives onto people around him. Then last year, his wife told him he was a changed man and left. Slate's Alex Kantrowitz wrote a piece about this in May. It was called The Horrific Content a Kenyan Worker Had to See While Training ChatGPT, and reads, This type of work has been crucial for bots like ChatGPT and Google's part to function and to feel so magical, but the human cost of the effort has been widely overlooked. In a process called Reinforcement Learning from Human Feedback, or RLHF, bots become smarter as humans label content, teaching them how to optimize based on feedback.

12:04AI leaders, including OpenAI Sam Altman, have praised the practice's technical effectiveness, yet they rarely talk about the cost some humans pay to align the AI systems with our values. Now, a big question in both of these pieces is what OpenAI and the contracting company SAMA that they worked with did or didn't provide these Kenyan workers, which is a really important set of questions to ask. But the broader point is just to acknowledge that reinforcement learning in the context of LLMs is not without cost. Now, other companies have tried other approaches for exactly these concerns, both around scalability as well as around the harm to individuals who are involved in the RLHF process.

12:37One of the more notable of these is Anthropik's constitutional approach, which they explain as, quote, giving language models explicit values determined by a constitution rather than values determined implicitly via large-scale human feedback. Of RLHF, they write, this process has several shortcomings. First, it may require people to interact with disturbing outputs. Second, it does not scale efficiently. As the number of responses increases or the models produce more complex responses, crowd workers will find it difficult to keep up with or fully understand them. Third, reviewing even a subset of outputs requires substantial time and resources, making this process inaccessible for many researchers.

13:11So given all of this, shouldn't we just switch now to reinforcement learning from artificial intelligence entirely? Well, even the Google researchers who produced this paper caution against running that far that fast. They write, While this work highlights the potential of RLAIF, we note some limitations on these findings. First, this work only explores the task of summarization, leaving an open question about generalizability to other tasks. Second, we did not estimate whether LLM inference is advantageous versus human labeling in terms of monetary costs. Additionally, there remain many interesting open questions, such as whether RLHF combined with RLAAF can outperform a single approach alone, how well using an LLM to directly assign rewards performs, whether improving AI labeler alignment translates to improved final policies, and whether using an LLM labeler the same size as the policy model can further improve the policy, i.e.

13:58whether a model can self-improve. We leave these questions for future work, and we hope that this paper motivates further research in the area of RLAIF. Now, even beyond just having comparable results, the paper also pointed out that, quote, RLAIF appears less likely to hallucinate than RLHF. The hallucinations in RLHF are often plausible but are inconsistent with the original text, meaning that it's not just potentially scalability and replacing the human cost of the reinforcement learning, but that there actually could be benefits as well in terms of model performance. Still, to the extent that Google's goal was to provoke more interest in further exploring this area, I think it's certainly going to be successful in that.

14:34In many ways, it's also part of a larger conversation around what parts of the AI model training process can be done by AI versus need human input. One big topic of conversation throughout the year has been the concern around model collapse, which is something that might theoretically happen as AI trains on more AI-generated content. However, recently, when Llama 2 was released, one of the things that Meta noted was that the model that was trained on synthetic data actually outperformed models that were trained on non-synthetic data, bringing up a lot of questions about how much our assumptions about model collapse actually hold true.

15:05Now, if you want to take it from the biggest implications perspective, it's really just how little we know about these models and about how to improve them. And for as advanced as we are, it feels very much like we are at the most infantile stages of figuring these things out. In that context, I find it worthwhile to approach everything with a certain base level of humility and an appreciation that the next research paper that comes out might fundamentally change our understanding about how the entire space works. For now though, some really interesting stuff to chew on with some really big potential benefits.

15:36I will of course include a link to the research in the news in the show notes, and I hope you take some time to go check it out. Thanks as always for listening or watching, and until next time, peace.