OpenAI Sets Moonshot Goal of AI Superalignment in 4 Years

0:00Today on the AI Breakdown, we're talking about OpenAI's massive moonshot goal to reach superalignment in four years. Before that on the brief, Google's new data policies and why AI is killing the open web. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information. Is it the end of the open internet? After a number of recent moves by tech giants, it certainly seems that way. Welcome back to the AI Breakdown Brief, all the AI headline news you need in five-ish minutes or less. We kick off today with something that is, in many ways, a continuation of the story of Reddit and Twitter's changes in their API considerations from last week.

0:44Over the weekend, Google updated its privacy policy, and it basically says that the company reserves the right to scrape pretty much anything that anyone posts online. Now, specifically, it says that it's allowed to use this data to train its AI models. So, for all intents and purposes, if you create words on the internet that Google can see, Google says it has the right to train AI on them. Now, in terms of how this changes things, part of it is that it's expanding the types of AI that Google says that it could use this data for. As Gizmodo writes, previously Google said the data would be used for language models rather than AI models.

1:16And whereas the older policy only mentioned Google Translate, BARD and Cloud AI are now listed as well. Gizmodo also sums up the stakes pretty well. They write, The practice raises new and interesting privacy questions. People generally understand that public posts are public, but today you need a new mental model of what it means to write something online. It's no longer a question of who can see the information, but how it could be used. Now, of course, these sort of policies are exactly what's behind recent changes to Twitter and Reddit's API. In April, Reddit announced that it would now be charging companies who scraped its data, and Elon on Twitter has been on an anti-scraping tirade recently.

1:52The biggest manifestation of that so far is Twitter's move to cap the number of tweets that any account can read, with the specific numbers based on whether you're verified or not. And to many, the implications of this are quite a bit larger than just which internet services you use. Bob McElrath writes, there is no more open internet. It's all walled gardens under the lie of free speech. Rob Sheridan writes, the thing we warned about way back in the early days of the net neutrality fight was that media and telecom companies wanted to turn the open internet into an entertainment platform of walled gardens, paid tiers, and proprietary content.

2:23Now, that call is coming from inside the net. The way The Verge sums it up is this. Add it all up, and the social web is changing in three crucial ways. It's going from public to private, it's shifting from growth and engagement, which broadly involves building good products that people like, to increasing revenue no matter the trade-off, and it's turning into an entertainment business. It turns out there's no money in connecting people to each other, but there's a fortune in putting ads between vertically scrolling videos that lots of people watch. So the social media era is giving way to the media with comments era, and everything is an entertainment platform now.

2:54Now, of course, part of the downside of these walled gardens isn't just that they create balkanized experiences, but that they give platforms much more control than they previously had. Professor Ethan Mollick writes, Everything the early open internet hackers were worried about turned out to be true. If you can't have your own local copy of something, it can be disappeared. Any corporate online platform can become a walled garden overnight and your content taken away. Axios sums it up, public web unravels an AI-driven storm. The old web is coming apart at the seams faster than efforts to shape a new one can fill the gap.

3:25The Axios piece continues, Any site that depends on contributions from the public, text messages, product reviews, photo or video uploads, is preparing to be swamped with AI-generated input that will make finding signal in the noise even harder for human users. At the same time, these sites are trying to shut their technical gates so others can't gobble up troves of data for AI models to study. The tech world has built 30 years of growth on the idea of the open web, But the emergence of ChatGPT and other AI tools trained on this stockpile of human expression, along with a financial downturn that's made firms scramble for revenue, has imperiled the old ideal of the web as a public resource.

3:59Now, of course, one of the other battles that this sets up is the one that has been anticipated in the Web3 space for some time. As Axios puts it, many of the users and volunteer moderators who contributed all that valuable data to these sites are chiming in too to say, hold on, we created all that value for your company. We should have a say in how it gets used and maybe a share of the profit. I think we are just at the beginning of this battle, but it is remarkable to note how fast the rise of generative AI has fundamentally shifted the way that people think about the open web. Or perhaps to put it more accurately, the way that platforms think about their opportunities and challenges in the context of the open web.

4:35I do think data-hungry AI is in this case combining with the fallout of the shift away from zero interest rate policies and the incredible financial pressure that internet companies now find themselves under. Sadly, from where I'm sitting right now, I don't totally disagree with Axios' conclusion that whatever happens, the online world is going to end up with less commons and more silos. Next up today, another area where the challenges of that move away from ZERP policies has been very clear. Venture capital funding is down 49 % year over year in Q2 and 51 % for the first half of 2023 as compared to 2022.

5:11As Crunchbase puts it, AI was quarter two's big hope to reverse the global venture funding slowdown. It wasn't enough. Now, the picture that this research paints is just a significant decline in venture capital funding. And by and large, this has been expected for some time. The venture capital space is just finally now starting to deal with the fundamental shift in interest rate policy and how that impacts where investors are looking for yield. And the byproduct is simply less money for startups. Now, many think that this is going to create an extinction level event over the next 18 months, as companies that raised at high valuations over the last few years are forced either to get to profitability or have to raise money at what will in many cases be down rounds where the valuation is lower than it was in previous fundraising.

5:52Now, when it comes to AI, companies that are categorized as AI in Crunchbase raised$25 million in the first half of this year, which was down from$29 million invested in the first half of last year, but up as a portion of total funding. AI companies represented 18%, so almost a fifth of total global funding over the first half of this year. Now, that does include$10 billion in funding to OpenAI that was led by Microsoft, so the statistic might be a little bit warped by that. A couple more quick ones before we get out of here. The trend of entertainment unions adapting to or trying to adapt to AI continues as IATSE, which is a union that represents around 160 ,000 professionals in the entertainment space, has released their core principles for applications of artificial intelligence and machine learning.

6:33If I had to sum up these core principles, it would be that one, artificial intelligence is here and it's not going away. Two, as they write, it threatens to fundamentally alter employers' business models and disrupt IATSE members' livelihoods. And so three, this union is going to work hard to prepare its members for this new future. IATSE tweets, The stakes involved are high. Therefore, our approach as a union must be comprehensive, focusing on research, collaboration, education, political and legislative advocacy, organizing, and collective bargaining. One of the things here that they discuss extensively is upskilling or reskilling.

7:06They write,

7:20Now, speaking of upskilling, another little Passover news item from Reuters, India's Tata Consultancy, which is one of the biggest IT consultancies in the world, has announced a new generative AI offering, as well as a plan to upskill 25 years. thousand of their engineers on Microsoft Azure's OpenAI. One of the interesting dynamics of this new era of AI is the way in which it levels the playing field between highly skilled workers in places like America and their engineering counterparts in other countries. In other words, this could be a big moment for global competitiveness. Finally, a story from Bloomberg, which I definitely think is going to be reflective of a larger trend.

7:53The piece is called The U.S. Military is Taking Generative AI Out for a Spin. It focuses on how the Defense Department is testing five different large language models to see how they do when trained on proprietary secret military data and can be used to quickly help military organizations ask questions, access information, make predictions, test scenarios, and more. Bloomberg writes, The military exercise, which runs until July 26th, will also serve as a test of whether military officials can use LLMs to generate entirely new options they've never considered. For now, the U.S. military team will experiment by asking LLMs for help planning the military's response to an escalating global crisis that starts small and then shifts to the Indo-Pacific region.

8:32In other words, they're testing to see what would happen if we got in a conflict with China around Taiwan. Speaking of that, in reaction to the U.S.'s growing controls around AI chips sent to China, China has announced plans to curb exports of AI chip-making materials. On July 3rd, the Chinese Ministry of Commerce issued a statement saying that beginning on August 1st, export of a couple products that are used in chip manufacturing will now require a government-issued license. Now, I don't think given the US's aggressive anti-China moves as it relates to AI that this is all that surprising, but it does help demonstrate just how tense and geopolitical the AI issue really is.

9:07Anyways, guys, that is going to do it for today's AI Breakdown Brief. If you're enjoying it, go check out the AI Breakdown newsletter. You can find that at the AIBreakdown.Beehive. That's B-E-E-H-I-I-V.com. Thanks for watching and listening, and I'll be back soon with the main AI Breakdown. welcome back to the ai breakdown today we are talking about a new initiative from open ai that is meant to help make it so that ai doesn't take over the world over the past few months concerned around ai safety and in particular the extinction risk or existential risk that comes from a super intelligent ai or an advanced general intelligence have really rocketed into mainstream consciousness One of the big drivers of this has been Jeffrey Hinton.

9:50At the beginning of May, the New York Times published this piece called The Godfather of AI Leaves Google and Warns of Danger Ahead. Now, the idea that someone could be ludicrously well compensated as someone at Google is, and still leave, and effectively turn one's back on their entire lifetime of research and work, has really captured people's attention and put the spotlight on these questions of AI risk. Now when it comes to AI risk, part of the way that people in the space try to address or think about it is what they call alignment. In other words, aligning AI with human values. The presumption is that an advanced general intelligence wouldn't necessarily share human values a priori.

10:30And so in order to avoid the various scenarios that end up with AI exterminating the human race in one way or another, we need to figure out ways to align advanced AIs with our own societal norms and values. The problem historically has been that there has far less time, energy, compute, and other resources dedicated to alignment than there is to advancing AI capabilities. This makes sense in the context of our market system, which is going to reward people more for their advances in capabilities than it is for their advances in alignment. But that gap has been hugely problematic, and part of the reason why AI doomers are so, so scared.

11:01It was notable then yesterday when OpenAI, the company behind ChatGPT and pretty much the most influential company in the space, published a blog post called Introducing Superalignment. The lead of the post reads, We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. To solve this problem within four years, we're starting a new team co-led by Ilya Sutskever and Jan Leakey and dedicating 20 % of the compute we've secured to date to this effort. The post begins, Superintelligence will be the most impactful technology humanity has ever invented, and could help us solve many of the world's most important problems.

11:35But the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction. While superintelligence seems far off now, we believe it could arrive this decade. Managing these risks will require, among other things, new institutions for governance, and solving the problem of superintelligence alignment. In other words, how do we ensure AI systems much smarter than humans follow human intent? Currently, we don't have a solution for steering or controlling a potentially super-intelligent AI and preventing it from going rogue. Our current technologies for aligning AI, such as reinforcement learning from human feedback, rely on humans' ability to supervise AI.

12:11But humans won't be able to reliably supervise AI systems much smarter than us, and so our current alignment techniques will not scale to super-intelligence. We need new scientific and technical breakthroughs. So there is a lot here just in this first section. One, once again, OpenAI notes that the extreme possibility of human extinction is real. I've said it before and I'll say it again that it is quite significant that the leading company in this space speaks so openly about that risk. It's exactly the type of thing that in most historical contexts a company would try to tamp down or even malign and castigate as crazy.

12:45A second interesting thing about this opening is what OpenAI's sense of timeline is. They believe, they say, that superintelligence could arrive this decade. Given that historically, people's estimates of how fast AI was going to develop have been off but in the other direction, as in AI has developed much faster than they thought, this prediction that superintelligence could be here before 2030 is super, super notable. Finally, this acknowledgement that today's techniques are not sufficient is put really plainly and really clearly. There is an acknowledgement from OpenAI that in the context of a superintelligent AI that is smarter than us, we simply won't be able to monitor it like a zealous parent would.

13:21So to me, right from the beginning, it's clear that there is a serious level of intent behind this new initiative. So what is OpenAI's approach going to be? They write, our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts and iteratively align superintelligence. To align the first automated alignment researcher, we will need to, one, develop a scalable training method, two, validate the resulting model, and three, stress test our entire alignment pipeline. They also say that they're expecting their research priorities will evolve substantially as we learn about the problem.

13:53And so this solution is not necessarily the one that they'll end up with, but it is where they are starting. From there, they get into the resources that they're going to dedicate to this. First of all, it's a new team, and it's being led by their chief scientist and a co-founder of the company. More than that, however, is the announcement that they're dedicating a fifth of the compute that they've secured to solving this particular problem. Now, you may say a fifth isn't all that much, but to put it in context, OpenAI has had to delay features this year because they simply don't have enough compute to push them forward.

14:22They had intended, for example, to release a multimodal version of ChatGPT, and it appears that that's just not going to be possible because of lack of access to compute. So to take 20 % of that and move it away from non-commercial resources is not an insignificant move. Still, maybe the most notable thing about this is the moonshot nature of it. OpenAI writes, our goal is to solve the core technical challenges of super intelligence alignment in four years. Going from something we have no idea how to do to actually solving it in four years is an incredibly, incredibly ambitious goal. And this is absolutely something that people in the community have taken note of.

14:58And I think that's a good segue to start talking about what people in the community actually think about this. Let's start with the AI safety community, who are the folks who are most likely to be skeptical of it. Zvi Maushiewicz, who's one of the best writers on AI safety right now, sent out a Twitter poll saying, If your top priority is AI not killing everyone, joining the OpenAI Super Intelligence Alignment team is, the options he listed were a great play, a good play, about neutral, or an actively bad idea. In reverse order, of the 500-plus voters, 12.7 % said an actively bad idea, 19.1 % said about neutral, 38.2 % said a good play, and 30 % said a great play.

15:36So almost a third of people think that this would be a great approach for someone who wanted to focus on eliminating AI X-Risk, and overall about two-thirds of people thought that it was a great or good play. Eliezer Yudkowsky initially had concerns that there was a disparity between how the people who were focusing on the alignment side of things were being compensated as compared to those who were focused on the capability side, tweeting, based on previous reports that a typical ML person at OpenAI earns$930 ,000 per year. It looks like you're planning to pay alignment people one-fourth to one-third.

16:04What capabilities people earn at OpenAI? Am I in error? To that, a researcher at OpenAI said, you are. The alignment research scientist role page says the annual salary range for this role is$245 ,000 to$450 ,000. The generic research scientist role page says the annual salary range for this role is$200 ,000 to$370 ,000. To that, Eliezer said, okay, I'm temporarily relieved. And even tweeted out, I'm in error. Yay. Daniel F. writes, Glad they're not dancing around this issue or using euphemisms here. As an aside, super alignment sounds like trying to align superheroes, which makes me think of the TV show The Boys.

16:36It's a goofy term, but one that seems hard to co-opt or water down. And maybe the most telling response to me comes from Leron Shapira, who wrote, Hot take, that's awesome. More than I was expecting and more than any competitor is doing. Now, most importantly, Leron points out, they didn't have to make themselves vulnerable to accusations of failing to meet the standard they set for themselves, but they did. And what he's talking about is what it means to set this four-year goal. In the absence of a four-year goal, in the absence of identifying a percentage of compute that's being used for this, it would be easy for this to be a sort of corporate philanthropy type effort that is ultimately greenwashing or just some PR-focused thing.

17:11By announcing that they're using 20 % of their compute for this goal, and by announcing their goal of four years, those are benchmarks that people can measure against to see how OpenAI is doing at its own initiative. Now, of course, this isn't to say that there is some skepticism. Autism Capital writes OpenAI owning alignment research is like the cigarette companies owning cancer research. TK Rengrenajan writes OpenAI has no incentive to avoid even known downsides of the AI tech they create. The super alignment is simply a checkbox feint to avoid regulations. Maybe more middle of the road, Rohit asks, Practically, does OpenAI's announcement about super alignment meaningfully change anything from what they were already doing before?

17:4819.3 % said yes, 60.6 % said no, 20.1 % said a secret third thing. For what it's worth, I am definitely in the yes column on this question. I think that putting a co-founder and chief scientist on a team that is going to have access to a fifth of the compute resources of a company, where compute resources are inevitably and inexorably scarce, and where there is a four-year moonshot goal for one of the hardest problems in the world, is significantly different than just having a bunch of people who focus on alignment kicking around the office. Now, from the less cynical side, a lot of people recognized that moonshot nature of this and appreciated the ambition behind it.

18:21Riley Goodside, who was featured on that Cognitive Revolution episode that we had on our feed a few weeks back, wrote, A statement of purpose in giant font toward an ambition like no other. Meanwhile, others like Dr. Jim Phan from NVIDIA are jumping in with their own ideas already. He writes,

18:45He then gives an example of writing malware and how a human system could be redesigned to have humans be able to supervise much better than they could today. Now, what about the wisdom of the crowds? On manifold markets, there are a number of prediction markets around this particular issue now. One of them asks, will OpenAI's super alignment project produce a significant breakthrough in alignment research before 2027, which currently stands at a 62 % chance of yes? Now when the question was asked a little bit more concretely, i.e. not just a generally important breakthrough, but actually achieving the goal of super alignment, the confidence interval was a bit lower.

19:19Victor Lee created a prediction market, will the OpenAI super alignment team believe that their goal has been achieved after four years, and only 26 % say yes. Now of course the big question that overhangs is what happens if the alignment team gets to the end of four years and hasn't made the breakthroughs that they want? What happens if instead they've convinced themselves that there aren't good ways to align super intelligence? Nathan Young on Twitter asks, Will OpenAI stop developing AGI if their alignment team is pessimistic in four years? I think that's a really good question to ask, and one that's worth having discussions about now.

19:50You have to think that OpenAI is having some of those conversations, but ultimately I'm sure they're going to reserve judgment until they're actually able to make some progress on this challenge. Now, I definitely find myself in the optimistic camp around this. Not optimistic in the sense that I'm convinced that in four years we can solve alignment to superintelligence, but optimistic in the sense that this is a serious effort at it. It's an attempt to align their company at least around a moonshot goal, and I wouldn't be surprised if this is influential in how other companies have to follow suit as relates to some of these issues.

20:21Anyways, I am interested to know what you guys think. Do you have that cynical take on this, that this is just for PR or an ability to avoid regulations? Are you less cynical about the intention, but skeptical of the capacity to actually make progress? Do you think that an even broader effort is needed? And if so, what do we do to encourage it? I think if nothing else, this is a good moment to ask those questions. So hit me up in the comments, come join me on Twitter, and let's have the super alignment conversation. That's going to do it for today's AI Breakdown. If you enjoyed it, go share this episode with one person who you think would be interested in these issues.

20:53I'd love to have them join the community as well. Until next time, guys. Peace.

21:00Thank you.