Do AI Lab Employees Have a "Right to Warn" The Public About AGI Risk?

0:00Today on the AI Daily Brief, we're asking if AI Lab employees have a right to warn the public about AGI risk. Before that in the headlines, the latest in the AI chip wars. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show notes.

0:24Hello friends, quick note, apologies for yesterday not delivering a show. I was traveling and then came back and got walloped by some very weird sickness. You can probably hear a little bit left of it in my voice. Hopefully we don't get interrupted again, and hopefully you enjoy today's shows. Welcome back to the AI Daily Brief Headlines Edition. All the AI headline news you need in around five minutes. It has once again been a busy week in the world of AI chips. There was a big event this week called the Computex Trade Show. You probably have seen this already iconic image of NVIDIA CEO Jensen Huang signing a woman's chest.

0:56Then Bloomberg summed up the event saying NVIDIA's pitch in AI chips holds echoes of Apple and iPhones. Here's how they summed up the presentation. NVIDIA's proposition to the world? Sign up with us and you'll get the best hardware, software, and services you need to train AI models. We'll build an ecosystem of developers and give them the best tools to make apps. We'll upgrade the key components once a year. It'll be pricey and you'll sacrifice much compatibility with our rivals, but you won't have to sweat any of the details. Said against that is basically everyone else who sells chips for a living.

1:23AMD CEO Lisa Su put forth this case in the opening keynote. Adopt the open standards we're promoting and you won't be tied down to us. We'll let you plug and play tech from all providers, craft your own apps, and customize your hardware mix as you see fit. Now, another interesting thing from this event is that while there was a lot of discussion around AI chips for laptops and smartphones, as Bloomberg writes, From conversations with industry observers and executives in Taipei, it's obvious that the data center is where the highest margin business is, and NVIDIA's lead there, as of now, is insurmountable.

1:51That, however, isn't stopping people from trying. Bloomberg writes, Intel CEO takes aim at NVIDIA in fight for AI chip dominance. Intel showed its new Xeon 6 data center processors with more efficient cores that will allow operators to cut down the space required for a given task to a third of prior generation hardware. Like rivals, Intel touted benchmarks that showed its new silicon as significantly better than its existing options. While Intel's not giving up without a fight, so far the market is unconvinced. Another legacy infrastructure provider, Cisco, also announced a new AI initiative, this time a billion-dollar investment fund.

2:23This announcement came at the Cisco Live conference in Las Vegas, and in addition to investing in companies like Mistral, Scale, and Cohear, Cisco, quote, plans to team up with these companies, which may mean using their technology in its own lineup and helping sell the services to corporate clients and government agencies. Said Chief Strategy Officer Mark Patterson, AI is reshaping every industry across the globe at an unprecedented pace. Some of the investments that we're going to make will be followed by some partnership agreements. A big funding round that was in the news around this was, in fact, from Cohear.

2:52The company has raised a fresh$450 million, which comes from returning investors including NVIDIA and Salesforce, as well as new investors like Cisco. Reuters writes this concludes the first tranche of Cohere's months-long fundraising efforts and marks a jump in valuation from its last private raise when it was valued at$2.2 billion to now with a valuation near$5 billion. Another big raise in Gen.AI comes from Pica. The Washington Post reports that the company has raised$80 million for its video generation software. The post points out that although a lot of the buzz in the video generation space has belonged to OpenAI's Sora and Google's more recently announced VO, those tools aren't currently available yet, while competitors Pika and Runway are.

3:29This new round values Pika at$470 million. And the question will be whether this is enough of a war chest to compete with much better funded competitors. It's not just the startups in the world that are competing for resources in AI. According to emails obtained by CNBC, Elon Musk has ordered thousands of NVIDIA-made H100s, to be diverted from Tesla to X and XAI. Writes The Verge, Tesla is supposed to be stocking up on NVIDIA's H100 AI chips in order to power its transformation into a leader in AI and robotics. But emails by NVIDIA employees obtained by CNBC suggest that Musk is exaggerating the purpose of AI chips for Tesla.

4:05Instead, many of those processors are now en route to X and primarily its AI subsidiary XAI. After CNBC published the story, Musk said on Twitter, Tesla has no place to send the NVIDIA chips to turn them on, so they would have just sat in a warehouse. The south extension of Giga Texas is almost complete. This will house 50k H100s for full self-driving training. Ultimately, of course, the question is whether investors get nervous and see this as an example of Elon's freight attention. But for that, we're just going to have to wait and see. For now, that is going to do it for the AI Daily Brief Headlines Edition.

4:34Up next, the main episode. Today's episode is brought to you by Fractional. Fractional AI is my go-to AI dev shop. When we wanted to build an AI product feature for our company, Superintelligent, We hired Fractional because they're some of the best and fastest AI engineers on the planet. The feature they built turned out great. It's already been released. And I'm about to hire them for another project, so I highly recommend them to anyone looking to build AI product features and workflow automations. The Fractional team is a group of senior engineers in San Francisco working on some of the most exciting projects in applied AI.

5:06They work with everyone from startups all the way through the Fortune 500. To request a free consultation, head to fractional.ai. If you want help identifying and building AI projects for your business, then I highly recommend that you go check them out. Hit pause on the show, open a web browser, go to fractional.ai and get your free consultation. Today's episode is brought to you by Super Intelligent. Regular listeners know that Super is our platform for helping people learn how to actually use AI tools. These are not long, laborious courses. These are fun, fast tutorials that get you actually using the world's most interesting and useful AI tools within minutes.

5:41If you want to build a web application with no code, We've got tutorials for that. If you want your presentations to look better than ever and take you less time than ever, we've got tutorials for that. If you want help brainstorming, writing social media copy, and just generally working smarter, faster, and better, we've got tutorials for that. We've worked really hard to make it so that there is no better place on the internet to learn how to actually put AI to work for you, and I'd love for you to check it out. Go to besuper.ai and use code podcast for 50 % off your first month. Once again, that's besuper.ai.

6:17Welcome back to the AI Daily Brief. It's been very interesting to watch the trajectory of the AI safety conversation in popular society. This is a conversation that had been ongoing for some time. But before ChachiBT, no one was paying attention. At least no one in the mainstream was paying attention. For that reason, it took the AI safety advocates by surprise when a few months later, everything that they were saying was making it into the news. Time magazine ran a cover story about how AI needed to be shut down because it was going to kill us all. And of course, we had the six-month pause letter, which, while ineffective in pausing things for six months, certainly was effective in getting concentrated attention on these issues.

6:55In some ways, this seemed to culminate with the firing of Sam Altman from OpenAI last November. However, subsequent to that, and specifically subsequent to his rehiring, the AI safety discourse has lost a lot of steam. Now, how much that is based on specific tactics within the AI safety space, or is just a natural ebb and flow, is an open question. But I think all of this matters as we contextualize a new note that just came out, where a group of current and former OpenAI employees are asking for a right to warn about risks they see emerging from their labs. The letter was published at RightToWarn.ai.

7:28It reads, We are current and former employees at Frontier AI companies, and we believe in the potential of AI technology to deliver unprecedented benefits to humanity. We also understand the serious risks posed by these technologies. These risks range from the further entrenchment of existing inequalities to manipulation and misinformation to the loss of control of autonomous AI systems potentially resulting in human extinction. AI companies themselves have acknowledged these risks, as have governments across the world and other AI experts. We are hopeful these risks can be adequately mitigated with sufficient guidance from the scientific community, policymakers, and the public.

7:58However, AI companies have strong financial incentives to avoid effective oversight, and we do not believe bespoke structures of corporate governance are sufficient to change this. AI companies possess substantial non-public information about the capabilities and limitations of their systems, the adequacy of their protective measures, and the risk levels of different kinds of harm. However, they currently only have weak obligations to share some of this information with governments and none with civil society. We do not think that they can be relied upon to share it voluntarily. So long as there is no effective government oversight of these corporations, current and former employees are among the few people who can hold them accountable to the public.

8:29Yet broad confidentiality agreements block us from voicing our concerns, except to the very companies that may be failing to address these issues. Ordinary whistleblower protections are insufficient because they focus on illegal activity, whereas many of the risks we are concerned about are not yet regulated. Some of us reasonably fear various forms of retaliation given the history of such cases across the industry. So what are they asking for? Well, they want AI companies to commit to the idea that they will not enforce agreements that prohibit disparagement when it comes to risk-related concerns, that the companies will facilitate a verifiable anonymous process for current and former employees to raise these concerns, that the companies will support a culture of open criticism, and that the companies will not retaliate against former employees who publicly share risk-related confidential information.

9:09The letter is signed by nine former employees of OpenAI, Google DeepMind, and Anthropic, and four that are currently employed by OpenAI. Of those 13 overall, six are anonymous. It's also endorsed by Yoshua Bengio, Jeffrey Hinton, and Stuart Russell. One of the signatories who was not anonymous, Daniel Cocotaglio, did a Twitter thread about this as well. He wrote, in April, I resigned from OpenAI after losing confidence that the company would behave responsibly in its attempt to build artificial general intelligence. I joined with the hope that we would invest much more in safety research as our systems became more capable, but OpenAI never made this pivot.

9:41People started resigning when they realized this. I was not the first or last to do so. When I left, I was asked to sign paperwork with a non-disparagement clause that would stop me from saying anything critical of the company. It was clear from the paperwork and my communications with OpenAI that I would lose my vested equity in 60 days if I refused to sign. My wife and I thought hard about it and decided that my freedom to speak up in the future was more important than the equity. I told OpenAI that I could not sign because I didn't think the policy was ethical. They accepted my decision and we parted ways.

10:07The systems labs like OpenAI are building have the capability to do enormous good. But if we are not careful, they can be destabilizing in the short term and catastrophic in the long term. These systems are not ordinary software. They are artificial neural nets that learn from massive amounts of data. There is rapidly growing scientific literature on interpretability, alignment, and control, but these fields are still in their infancy. There is a lot we don't understand about how these systems work and whether they will remain aligned to human interests as they get smarter and possibly surpass human-level intelligence in all arenas.

10:32Meanwhile, there is little to no oversight of this technology. Instead, we rely on the companies building them to self-govern, even as profit motives and excitement about the technology push them to move fast and break things. Silencing researchers and making them afraid of retaliation is dangerous when we are currently some of the only people in a position to warn the public. I applaud OpenAI for promising to change these policies. It's concerning that they engaged in these intimidation tactics for so long and only course-corrected under public pressure. It's also concerning that leaders who signed off on these policies claimed they didn't know about them.

10:59We owe it to the public who will bear the brunt of these dangers to do better than this. Reasonable minds can disagree about whether AGI will happen soon, but it seems foolish to put so few resources into preparing. Now what's really interesting to me is how people responded to this. It was effectively a Rorschach test for what people already think about AI safety questions. In other words, for those who are already concerned about these issues, this was yet another indication of how much was going wrong in OpenAI, yet another piece of evidence about why we should have much more governmental oversight of labs like OpenAI.

11:30However, not only do I not think that this convinced anyone who was on the fence to be more inclined towards AI safety questions, I actually think it might be having the opposite effect. Joshua Achayem pointed out some of the problems in a long thread on Twitter as well. He writes, there is a letter circulating now from former and current AGI Frontier Lab staff advocating for a particular policy around whistleblower protections on safety and risk issues. I will preface this by saying I like the people who have signed this. I like them a lot. Some number of these people I would consider not just a colleague but friend.

11:58To the signatories of this letter, I am speaking directly to you. I think you are making a serious error with this letter. The spirit of it is sensible in that most professional fields with risk management practices wind up developing some kind of whistleblower protections, and public discussion of AGI risk is critically important. But the disclosure of confidential information from frontier labs, however well-intentioned, can be outright dangerous. This letter asks for a policy that would in effect give safety staff carte blanche to make disclosures at will based on their own judgment. I think this is obviously crazy.

12:25The letter didn't have to ask for a policy so arbitrarily broad and underdefined. Something narrowly scoped around discussions of risk without confidential material would have been perfectly sufficient. Or narrowly scoped to protecting disclosures made to regulators. Freedom to report concerns containing confidential information to the public with no guardrails is an invitation to the worst and most avoidable InfoSec failures. It's an invitation to every interested party, state actor, or otherwise to exploit that vector of information for myriad purposes. Crucially, this letter disrupts a delicate and important trust equilibrium that exists in the field and among AGI Frontier Lab staff today.

12:55I don't know if you have noticed, all of us who care about AGI risk have been basically free to care about it in public since forever. We've been talking about P-Doom non-stop. We simply won't shut up about it. This has been politely sanctioned and supported by lab leaders, despite what are frankly many structural forces that do not love this kind of thing. The unofficial official policy all along has been to permit public hand-wringing and warnings. Just one red line. Don't break trust. Don't share confidential info. This line is red because the only way an organization can functionally achieve its goals as if there is an adequate basis of trust for cooperation between its many elements.

13:25Just by introducing the idea into the ecosystem that folks concerned about risk should have a special privilege to disclose whatever confidential information they feel they should, you have made it infinitely harder to build trust between tribes. Good luck getting product staff to add you to meetings and involve you in sensitive discussions if you hold up a flag that says, I will scuttle your launch or talk shit about it later if I feel morally obligated. Now, I think there's a lot that's important about that critique, but one of the most important pieces is what I expect to be a shift in the public stance from labs like OpenAI.

13:52I tweeted that I thought that OpenAI has two possible positions if things like this continue. The first is that they can continue to do what they've been doing, which is basically say, yeah, no, we totally agree. It's important to speak about these issues, not really comment on them. And as Joshua pointed out just a minute ago, basically sanction this public hand wringing. On the flip side, they could say, you know what? We reconsidered. And the reason that we sandbagged our super alignment team is that we just don't think the risk looks like what we thought it looked like before. And if they wanted to go farther, they could say, in fact, we think these are a bunch of Manic Street preachers who are screaming doom.

14:23My instinct is that it's increasingly likely that OpenAI and other companies like them head towards number two, basically no longer giving even lip service to the AI safety movement, at least not the X-Risk human extinction version of it. And part of the reason that I think that they might head that direction is how the public support for the AI safety movement is shifting. I saw a lot of commentary where people were basically nonplussed at the idea of asking permission, effectively saying that if these things are as bad as these safetyists say they are, are you actually going to wait before saying something?

14:53Eris at Ornai says, how big of a spineless idiot does someone have to be to stare down the approaching disaster and still follow laws and rules regarding non-disclosures? Were these people completely stripped of their agency at birth? Another common critique I saw was the genericness of this. Avikday writes, all these anonymous folks should get together and publish something more specific than these generic FUD. Till then, most serious people aren't going to take these seriously. Trust us bro attitude doesn't help if they truly believe the risk is as high as they speculate here. And indeed, I think this is the exact same response that we saw in the wake of Sam Altman's firing.

15:23The fact that even with these quote-unquote bombshell interviews that have been happening recently, no one is actually pointing to any real evidence of specific ways in which Altman broke the board's trust seriously undermines their arguments overall. And then there's the perspective represented by Danny Knott Jr. on Twitter, who said this was incredibly underwhelming. Anyone trying to warn of this sort of risk was always going to have to face some amount of criticism for being chicken little-ish or boy who cried wolf, and I think that that has taken hold in a huge way right now. The problem, of course, is that if you think these conversations are important to have, even if you find yourself on the other side of them, is that space for these conversations is getting crowded out by these sort of publicity stunts.

16:02I don't have a good answer for how to do it better, but I do know that from where I'm sitting and from the commentary that I've seen, This letter is the latest in an example of things from the AI safety movement that not only seem to not have the impact that they wanted, but to in fact have had the opposite impact. Then again, we also got a very long book-level essay from Leopold Aschenbrenner, formerly of OpenAI, about AGI that seems to be doing a little bit better. So we'll come back to that one later in the week. For now, though, that is going to do it for the AI Daily Brief. Until next time, peace.

16:37Thank you.