OpenAI Forms Catastrophic Risk Preparedness Team

0:01Today on the AI Breakdown, we're looking at OpenAI's new Catastrophic risk preparedness team. Before that on the brief, an incredibly cool model that can watch videos and actually understand them. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube channel, our Discord, and our newsletter.

0:24Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. And man, we have some really cool stuff today. We kick off with a new foundation model. it is a video language model from the company 12 Labs. Alongside the announcement of this model, they are also announcing an investment from NVIDIA, Intel, and Samsung, which I think probably tells you all you need to know about just how exciting this is given those huge companies participating in this investment. So like I said right at the beginning, this is a model where you can have it watch a video and then ask it to summarize, to give highlights, to generally interact with the video in an intelligent way.

1:03Now, you may have seen other tools that do this, but 12labs says that their solution is somewhat different. They write, contrary to existing solutions that either utilizes speech-to-text conversions or rely solely on visual frame data, Pegasus 1 integrates visual, audio, and speech information to generate more holistic text from videos. Basically, what they're saying is that we are not taking this, translating it into a script from which we can interact with in the same way that a normal LLM would, but instead that they have built a model from the ground up that integrates all of this different type of information.

1:37Pegasus 1, they write, has approximately 80 billion parameters, with three model components jointly trained together. Their video encoder, their video language alignment model, and their language decoder. Now, yes, this looks cool, but does it perform? According to their research, the answer is yes. They write, Pegasus 1 exhibits massive performance improvement over previous state-of-the-art video language models, and other approaches to video summarization. Now the company has released a technical report on this that you can go read, where you can get more information around how they compare to other types of models, both in terms of their strategy and in terms of their performance.

2:11And you can also learn more about the dataset underlying it as well. What you can't do unfortunately is actually use it yet, but you can join a waitlist. This is a product category that has tons of opportunity, and I think that will really break down the barriers between the different categories of content that we all consume. We're moving quickly to a world where regardless of what format the creator creates content in, you can consume it in a format of your choice, be it audio, video, or otherwise, with the help of course of AI. Next up in the world of AI image generation, a slightly different approach from Shutterstock.

2:44Obviously in this space, one of the big questions is where regulation, policy, and the courts are going to come down on questions of copyright, and we've seen a number of different strategies to try to get out ahead of that. Companies like Adobe have created models in their Firefly 2 that are trained entirely on images that they have the rights to, trying to sidestep those issues. And of course, they've also offered to pay for legal fees should that not be sufficient. Shutterstock is walking even farther down that path, basically by allowing users to transform photos that are already in the Shutterstock library and paying artists if those images are licensed after they're edited with AI.

3:21So really, in some ways, this is more of an AI extension of existing stock photo libraries, but one that solves some of the questions of artist compensation. Some of the ways that users will be able to modify those images include expanding the image or background, resizing the image to match the required dimensions, removing the background, and doing something akin to Adobe's generative fill, where you can select an area of the photo and describe in natural language what you want to add, replace, or erase. Now, obviously, ultimately, this is still more creativity constrained than the pure play image generators that many people are using now, but in a way that does answer a lot of questions that are still dogging that space.

3:58Moving back to the world of business, we had another earnings call that was totally focused on AI, and that comes from Amazon. Earlier in the week, we had earnings calls from Microsoft and Google. Microsoft was the big winner out of that, seeing its Azure business grow even more than analysts' expectations. Google's cloud business had a little bit of a tougher time, but in both cases, AI was clearly the feature. So too was it with Amazon's earnings call on Thursday. Yahoo Finance kicks off their article, Amazon Entered the AI Hype Zone. Now, there were two things going on that analysts liked here.

4:29One was that Amazon beat earnings estimates, which is the most important thing. But two, CEO Andy Jassy said that their AI business was an opportunity that was worth, quote, tens of billions of dollars to their Amazon Web Services business. Earlier this year, of course, Amazon launched Bedrock, which is a way for enterprise clients to help customize, develop, and implement AI models. And Jassy said, quote, our generative AI business is growing very, very quickly. In fact, he said it was growing so quickly they were even surprised at the growth of that business. Now, last weekend, we talked about how Amazon fits into the larger big tech competition around AI and how they're trying to operate at the level of applications and enterprises and actually getting this technology into companies that need it, which is, of course, their Bedrock approach, but also on the level of actual foundation models, specifically with their recent tie-up with Anthropic, and then down even a little farther into their approach to developing AI chips.

5:21Seeing that it is actually already having an impact on the bottom line means that markets are likely to be very eager that they continue down this path. Meanwhile, over in the land of Google, things were a little quiet this week, but the company did announce a new bug bounty program specific to generative AI. Cyber threats, they write, evolve quickly, and some of the biggest vulnerabilities aren't discovered by companies or product managers, but by outside security researchers. Today, they write, we're expanding our vulnerability rewards program to reward for attack scenarios specific to generative AI.

5:50Now, this just makes total sense to me and I think is a thing that you're going to see from lots of companies, especially because the very existence of AI increases the variety of threats that companies are going to face. Now, lastly today, I want to talk about a really interesting survey from The Information. The Information is a Silicon Valley publication. You hear about it a lot here because they have a really good beat on particularly the big tech companies. They have been behind a lot of the scoops when it comes to new forthcoming features in the AI space. And their readership also sort of reflects, I think, their sources, right?

6:23In other words, they are a very tech-native publication through and through from their reporters to what they cover to the audience that reads them. They recently did a survey with their readers around how they're using AI and whether or not they're paying for it. The way that I would treat this is as a look at perhaps how an early adopter cadre is actually using these technologies. And in that, I think there's some pretty interesting data. When they asked the question, do you ever use any AI services for personal or professional use? The overwhelming majority did. Only 12 % in fact said no. 11 % said yes for professional use, 19 % said yes for personal use, and 58 % said yes for both.

6:58But what about what they spend on AI? Well, apparently one in five respondents who use AI, which again represents about 90 % of the people surveyed, said that they spend at least$100 monthly for AI services. That's specifically for AI services at work. A similar portion of the group said they spend at least$30 monthly on AI services for personal use as well. What's more, it seems like people are pretty happy with these tools. When asked if they plan on continuing to pay for these same AI services next year, more than half, 54%, said yes, and another 30 % said probably. In other words, the vast, vast majority are not churning and are finding enough value to continue paying.

7:35Now what about which services they use? Totally unexpectedly, ChatGPT is right at the top of the heap, with only 14 % of people not using ChatGPT, and a full 51 % using a paid version of it. Google Bard was next with 45 % using a free version. After that were Dali and MidJourney. Dali had 15 % using a paid version with 22 % using a free version. Not surprising given that it's now integrated into ChatGPT. and Midjourney had 15 % using a paid version with 14 % using an unpaid version. GitHub Copilot had around 13 % using a paid version and another 6 % using a free version. Anthropics Cloud had 15 % using a free version but only 6 % using a paid version.

8:12One of the takeaways for the information authors was that people were more willing to pay for AI services that have very specific applications like generating code or generating images. Now when it comes to why people are using these tools professionally, Some are because it saves them money, and some are because it saves them time. One respondent runs a company called Element Human that uses machine learning to analyze body language and says that he spends around£500 a month on AI services, but that that saves the company£2 ,000 per month and likely more in the future. Another respondent uses AI to prepare for interviews on his podcast.

8:45He said, Normally it'd take a couple hours to put together eight questions for an interview, and using AI, I can cut that down to under 10 minutes. Now, overall sentiment was the best that they've had since they began their monthly surveys back in February. 58 % of respondents said they think conditions will improve, and only 14 % said they think conditions will get worse. Like I said, this is a very early adopter kind of group. A full 56 % of the people who participated in this survey actually work in tech, but it still gives, I think, a great little insight into what early adopters are doing and how much these tools are changing people's workflows right now.

9:17However, that will do it for today's AI Breakdown Brief. Next up, the main AI breakdown. Welcome back to the breakdown. There is a lot, and I mean a lot of discourse right now around questions of AI safety and AI risk in general, but specifically catastrophic risk in AI, real human extinction level risk. The reason for that is a couple parts. One, it's just the culmination of a narrative which has been coming up all year. You've seen a steady beat of scientists and academics and some people from industry saying that these are issues that we really need to take seriously, that we really need to think about.

10:00Another part of that is that it's coinciding with a growing policy conversation. Governments around the world are figuring out how they are going to handle the full spectrum of risks from AI, as well as take advantage of it and leave their populations in the best possible place. And most sees its advantages to help their populations. Now along those efforts, next week we are getting the UK's much ballyhooed and much discussed AI safety summit. This has been in the news for a variety of reasons. One, because the UK Prime Minister's office has been really putting emphasis on this and making it a big political issue.

10:34Two, because there is intrigue and controversy around it. One of the decisions that Rishi Sunak's government made was to invite and include China in the proceedings. that apparently angered allies, especially the United States, who has taken a very it's-us-against-them approach when it comes to China and AI. That has obviously been embodied in the extension of an increase in export restrictions around AI chips, including cutting off loopholes that China was using to get around the export restrictions that were put into place last year at this time. So, this event is coming up next week. It's happening right at the beginning of November.

11:08And around that, it seems like both governments and labs are positioning themselves vis-a-vis this type of issue. We talked yesterday about how the White House is planning to release their executive order on AI on Monday, and OpenAI has also just announced a new initiative around this, which is directly focused on these catastrophic misuses of AI. So, in a blog post called Frontier Risk and Preparedness, OpenAI announced their preparedness team and the preparedness challenge. The company writes, As part of our mission of building safe AGI, we take seriously the full spectrum of safety risks related to AI, from the systems we have today to the furthest reaches of superintelligence.

11:45In July, we joined other leading AI labs in making a set of voluntary commitments to promote safety, security, and trust in AI. These commitments encompassed a range of risk areas, centrally including the frontier risks that are the focus of the UK AI Safety Summit. As part of our contributions to the summit, we have detailed our progress on frontier safety, including work within the scope of our voluntary commitments. Now, we will come back to exactly what they said and what they reported in that in just a moment. But OpenAI continues that when it comes to catastrophic risks from Frontier AI, there are a set of questions that society needs to answer.

12:17One, how dangerous are Frontier AI systems when put to misuse? Two, how can we build a robust framework for monitoring, evaluation, prediction, and protection against these problems? Three, if our Frontier AI model weights were stolen, how might malicious actors choose to leverage them? Because these questions need answers, OpenAI is now convening this preparedness team. They write that the team will connect capability assessment, evaluations, and internal red teaming for frontier models, from the models we develop in the near future to those with AGI-level capabilities. As a little aside, they are clearly saying that we are not developing AGI-level AI just yet.

12:50They go on, the team will help track, evaluate, forecast, and protect against catastrophic risks spanning multiple categories including individualized persuasion, cybersecurity, chemical, biological, radiology, and nuclear threats, autonomous replication, and adaptation. Now that first category, individualized persuasion, is interesting. Given a tweet the other day from OpenAI CEO Sam Altman who wrote, I expect AI to be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to some very strange outcomes. Okay, so coming back to this preparedness team, they're helping track, evaluate, forecast, and protect against these risks.

13:26But I think it's clearly that last one that we're most interested in, right? How do you actually help protect against it? One of the specific things is that they're developing what they're calling a risk-informed development policy. This, they say, will quote, detail our approach to developing rigorous frontier model capability evaluations and monitoring, creating a spectrum of protective actions and establishing a governance structure for accountability and oversight across the developmental process. Basically, to me, this reads like the mandate of the preparedness team vis-a-vis this risk-informed development policy is to actually integrate how the company deals with these risks to the actual development and deployment process.

14:05In other words, rather than having it be totally disconnected or something that you just look at once the model is ready to go, it seems at least they're giving the indications that this preparedness team will actually integrate those questions into the process of how OpenAI works in general. If that's the case, and certainly I could be optimistically reading into it, that I think would be a huge and meaningful development. One of the big concerns that many have pointed out, including a piece that we will read for long reads this weekend, is that there is an incredible disparity between the amount of money and just the general resources being spent to add capabilities to AI as opposed to align and minimize risks of AI.

14:41If that becomes a more integrated process within OpenAI, and especially if that sets a template and a model that others can follow, or even that governments mandate others follow, that could be a meaningful change to how the industry is evolving. So as part of this launch, they also announced their AI preparedness challenge. They say to identify less obvious areas of concern and build the team, we're offering 25 ,000 in API credits to up to 10 submissions on catastrophic misuse prevention. This is now available on the website. There's a preparedness challenge. It's just openai.com slash form slash preparedness dash challenge.

15:14and if you're interested, there's not a lot of words. You give them your name and your LinkedIn, and then basically you have around 600 words total, along with a three-page max PDF, to explain a particular misuse you're concerned about, why it might lead to catastrophic harm, and a plan for how to actually deal with it. I'll include a link to this in the notes so that if any of you have these ideas or want to get involved with this team, you have easy access to it. Now, as I mentioned, OpenAI said that as part of this announcement, they were also sharing an update around the UK AI Safety Summit on their approach to frontier risk.

15:44This is sort of building off those voluntary commitments that came earlier in the summer. So what have they done around those commitments? Well, they say when they launched Dolly 3, which is their first major public release of a new frontier model within the scope of those voluntary commitments, they, quote, did critical safety work, including pre-deployment safety evaluation and red teaming. In addition, they say we are working towards new methods to empower people to track the provenance of AI generated media and, quote, have continued to invest in responsible practices through our rollout of voice and image analysis capabilities in chat gbt although they don't say what those responsible practices are they also said that they've met their voluntary commitment to quote establish or join a forum or mechanism through which we can develop advance and adopt shared standards and best practices for frontier ai safety that is the frontier model forum which we discussed the other day as it just got itself its first executive director still it's quite clear that when it comes to what they think is significant here it really is this risk-informed development policy the vast majority of this blog post is in fact spent on that policy.

16:40They basically expand upon what they said before, that the RDP will detail their approach to evaluations and monitoring, as well as establishing a governance structure, and that it will provide for a spectrum of actions to protect against catastrophic outcomes. Now, they do note that this is meant to extend their existing work, including the work of their safety systems team to conduct research, their super alignment team, which has this moonshot mandate of aligning super intelligent AI systems with human intent in the next three and a half years now. And they also discuss a joint deployment safety board with Microsoft, which they say, quote, approves decisions by either party to deploy models above a certain capability threshold.

17:15The DSB focuses specifically on deployment decisions rather than on earlier steps, such as deciding whether or not to train models of a certain scalar capability level. Now, another part of this note talks about just how much of a scientific challenge alignment really is. They put it really bluntly. Our current techniques for aligning AI, such as reinforcement learning from human feedback, rely on human ability to supervise AI. But these techniques will not work for superintelligence because humans will be unable to reliably supervise AI systems much smarter than us. Now, I will also share a link to this because there is some really interesting information about how their model evaluations and red teaming work.

17:49There's also an acronym that they're using a lot, which is CBRN, which refers to chemical, biological, radiological, and nuclear uses and risks of use of AI. And I think if you take anything away from this write-up and from the preparedness team announcement, it's that while there isn't a substitute for real actions, the presence of this larger conversation around AI risks is having an impact in how people are thinking about their development. There is quite clearly serious effort being put into this. Now, we can argue about whether there is enough and about whether that effort needs to be matched by resources in terms of both money and compute, but this certainly looks different than the conversation did even just six months ago.

18:26Now, two other sort of related announcements heading into the AI Safety Summit next week. The first is that the United Nations has created an advisory body addressing AI governance. This was announced yesterday by the Secretary General, and it is a 39-member advisory body that can include tech company executives, government officials, and academics. Said Secretary General Antonio Guterres, The transformative potential of AI for good is difficult even to grasp. And without entering into a host of doomsday scenarios, it is already clear that the malicious use of AI could undermine trust in institutions, weaken social cohesion, and threaten democracy itself.

18:59So basically, this body is not one that is actually going to have any sort of real governance oversight, but is one which is meant to help make initial recommendations for intergovernmental cooperation around these issues. The UN said that the immediate tasks including building a global scientific consensus on those risks and challenges and strengthening international cooperation on AI governance. The first meeting of this body is taking place actually today. Lastly, yesterday, of course, the British Prime Minister Rishi Sunak had a speech about AI heading into next week's summit, where frankly a lot of the conversation was around the decision to invite China.

19:34But he also announced, as reported by Reuters, that they are setting up the world's first AI safety institute. Sunak said that the institute will, quote, advance the world's knowledge of AI safety and will carefully examine, evaluate, and test new types of AI so that we understand what each new model is capable of, exploring all the risks from social harms like bias and misinformation through to the most extreme risks of all. So like I said, if you are trying to step back and get a sense of what's happening right now, it's that the world is very clearly taking these threats seriously. Fascinatingly, you have people on either side of this question who are upset about this for one reason or another.

20:07The accelerationists are of course concerned that anything that stands in the way of AI is a failure to realize how beneficial it can be to humanity, and so of course they don't want to see open AI slow down because they're worried about these risks. And then on the flip side, you have plenty of people who say either A, it's not enough, and that even more serious commitments are needed, that more government power is needed to exert control on these big AI labs. Or another dimension of that is that we're focusing on the wrong issues, that there's too much time spent on these big future theoretical scenarios and not enough time on the existing problems like biases that show up in LLMs right now.

20:38My general take is that if a lot of people on lots of sides of an argument are upset about something that's happening in the middle for different reasons, then it might, might just be the start of a good compromise approach. But of course, this is an extremely dynamic and fast-moving space. We will hear a lot more, I'm sure, about all of this next week with the Safety 7 actually happening. However, for us today, we will wrap it there. I appreciate you guys listening or watching, as always. Until next time, peace.

21:14The questions Guestown span