Llama 3.1 405B Eliminates Gap Between Open and Closed Source AI

0:00Today on the AI Daily Brief, Meta's Llama 3.1.405b may have closed the gap in the state of the art between open and closed source models. Before that on the brief, OpenAI updates their safety policies without actually updating anything. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show notes. Welcome back to the AI Daily Brief Headlines Edition, all the AI daily news you need in around five minutes. We kick off today with the latest announcements from OpenAI on their safety strategy.

0:33They tweeted very early this morning, East Coast time. Making sure AI can benefit everyone starts with building AI that is helpful and safe. We want to share some updates on how we're prioritizing safety in our work. We believe that frontier AI models can greatly benefit society. To help ensure our readiness, our preparedness framework helps evaluate and protect against the risks posed by increasingly powerful models. We won't release a new model if it crosses a medium risk threshold until we implement sufficient safety interventions. We're developing levels to help us and stakeholders categorize and track AI progress.

1:03This is a work in progress and we'll share more soon. Editor's note, this is, I think, what we got some information about last week with their various levels of AI proceeding towards AGI. This tweet thread continues, however. In May, our board of directors launched a new safety and security committee to evaluate and further develop safety and security recommendations for open AI projects and operations. The committee includes leading cybersecurity expert, retired U.S. Army General Paul Nakasone. This review is underway and will share more on the steps we'll be taking after it concludes. Our whistleblower policy protects employees' rights to make protected disclosures.

1:34We also believe rigorous debate about this technology is important, and it made changes to our departure process to remove non-disparagement terms. Safety has always been central to our work from aligning model behavior to monitoring for abuse, and we're investing even further as we develop more capable models. Now what's notable to this, at least to me, is that there's nothing new that's actually announced. They frame it as we want to share some updates on how we're prioritizing safety, but there's nothing actually new here. This suggests to me that there was some reason that they felt they wanted to remind everyone of these efforts and put it in a nice, easy to point to place, but not that they had something big and new that they wanted to highlight.

2:08As I was wondering about what the reasoning was that might be behind this, it started to pop up on X that a group of five senators had sent Sam Altman a note. The note, which was dated yesterday, July 22nd, reads, Dear Mr. Altman, we write to you regarding recent reports about OpenAI safety and employment practices. OpenAI has announced a guiding commitment to the safe, secure, and responsible development of artificial intelligence in the public interest. These reports raise questions about how OpenAI is addressing emerging safety concerns. We seek additional information from OpenAI about the steps the company is taking to meet its public comments on safety, how the company is internally evaluating its progress on those commitments, and on the company's identification and mitigation of cybersecurity threats.

2:44The letter goes on to ask 12 questions, requesting that information be shared by August 13th of this year. One of the questions that many on Twitter have commented on is this one. Will OpenAI commit to making its next foundation model available to U.S. government agencies for pre-deployment testing, review, analysis, and assessment? Now, it's likely that I'm going to dig deeper into each of these questions in an episode later this week. So for now, I will just note that it seems like there is an increasing moment of scrutiny around OpenAI from the U.S. government, and that that might be driving some of this increased conversation around safety and preparedness.

3:15Next up, perplexity is back in the news, although this time not for a funding announcement, amazingly. No, unfortunately for them, this time it's because Condé Nast has sent a cease and desist letter to the company. The information reports that Condé Nast, which owns publications like the New Yorker Vogue and Wired, has sent a cease and desist demanding that Perplexity stop using content from these publications in its search results. This follows a letter last month from Forbes, where Forbes accused Perplexity of infringing on its copyright. Condé Nast similarly claims that Perplexity is plagiarizing its content.

3:44It seems that the ire around Perplexity isn't around training data, as that's not what Perplexity does, but around the way that Perplexity's AI summarizes news articles. Now, there are a lot of specific technical details in here around whether and where the bots that companies like Perplexity use to allow publishers to block their crawlers actually work, but ultimately this comes down to this new format for search. It feels to me fairly inevitable that the sort of combination of AI-generated summary plus links is going to be the default and norm in the future, but the path to get there may be littered with a lot of legal battles, proving once again that when it comes right down to it, the only group of people who always win are the lawyers.

4:20Moving to some science, a new Google model has, quote, helped make a breakthrough in accurate long-range weather and climate predictions. Writes the Financial Times, using a hybrid of machine learning and existing forecasting tools, a model led by Google called Neural GCM successfully harnessed AI to conventional atmospherics physics models to track decades-long climate trends and extreme weather events such as cyclones. A recent paper said Neural GCM proved faster, more accurate, and used less computing power in tests against a current forecasting model based on atmospheric physics tools called X-Shield.

4:49In one trial, they say, Neural GCM identified almost the same number of tropical cyclones as conventional extreme weather trackers did, and twice the number of X-Shields. In another test based on temperature and humidity, the error rate was between 15 and 50 % less. One of the things that often gets lost as we debate the big legal and ethical issues of AI is how much it's likely to impact scientific discovery, but that ultimately is going to be the subject for another show. That is going to do it for today's AI Daily Brief Headlines edition. Next up, the main episode. Today's episode is brought to you by Superintelligent, the platform for fun, fast AI learning.

5:23Super has a ton of new things going on. We recently announced our partnership with Spotify, through which users of that app can now access Superintelligent content directly from their mobile apps. We've also just launched the AI learning feed. In addition to seeing the tutorials that we're dropping, there are polls, news items with related lessons, and a chance for people to show off the projects and use cases that are making AI come alive for them. We've also just kicked off the Super Summer Challenge, where each week we'll share a new challenge that you can use to discover new AI tools and use cases.

5:53Go to besuper.ai and use code SUPERFUN for 50 % off your first two months. That's besuper.ai. Today's episode is brought to you by Venice. Venice is a private, uncensored generative AI app. It accesses open source models to enable text, image, and code generation without the fear of being spied on or having your data exploited. Discuss anything with Venice without concerns about it being monitored, sold, or given to advertisers and governments. Venice is different because your conversations and creations are kept securely within your own browser, never stored or accessible by Venice. Unlike other AI apps, Venice won't tell you what's okay to say or not.

6:27Venice won't patronize you. It simply provides direct access to machine intelligence. No topics are off-limits, no ideas are taboo. With Venice, you're in control of the AI, as you should be. Pro subscriptions are available for$49 a year or$8 per month. Try it for free without an account at venice.ai. Welcome back to the AI Daily Brief. Today we are talking about some leaks around the forthcoming meta Llama 3.1 405b model. As Didi from Menlo Ventures puts it, this is potentially the biggest news in AI in several weeks. Llama 3.1 405b leaked day before on 4chan and obliterates GPT-4.0 on most benchmarks.

7:05Writes Runway's Siki Chen, for the first time ever, an open source model is state-of-the-art, outperforming OpenAI's GPT-40 and Anthropic's Sonnet 3.5 across multiple benchmarks. So today we are going to talk about what we have learned so far about LAMA 3.1-405b and what the implications are of the full closure of the gap between open source and closed source. Now to go back, we have been waiting for some time for Meta's largest LAMA 3 model. About a week and a half ago, we got reports from the information and others that Meta was planning on releasing the 405 billion parameter version of Lama 3 today on July 23rd.

7:41Back in April, we got two smaller models from Lama 3, including the 8 billion and 70 billion parameter models, but this was always going to be the big show. In a post last week about why we should care about this release, Tom's Guide wrote, the Lama 3 400B model is particularly exciting as it approaches performance parity with OpenAI's GPT-40 model despite using less than half the parameters. Apart from the potential benefits to cost and energy efficiency, there's another significant advantage. One of the most compelling aspects of Lama 3 is its open license for research and commercial use. If the 400B model is released under the same open license, it would democratize access to state-of-the-art language capabilities, allowing researchers and developers to leverage this powerful tool for their projects without relying on expensive proprietary APIs.

8:22Still, as Tom's Guide pointed out, there had also been some scuttlebutt that Meta was not going to exactly open source this model. Notorious OpenAI leaker Jimmy Apples wrote, Meta plans to not open the weights for its 400B model. The hope is that we would quietly not notice and let it slide. Don't let it slide. He followed up saying Dustin Moskovitz, who is one of the big funders of the EA space and the AI safety movement, is quote, having a loud voice to the ears of lab CEOs behind doors. Then later, however, Jimmy Apples updated to say apparently at the moment they do plan to open source it despite Dustin's objections to Zuck.

8:55Open source AI advocate and AI CEO Bindu Reddy made this prediction. In a couple of days, we will stop talking about politics, hopefully. LAMA 405b will be the topic of the day. The gap between the closed and open-weight models will finally close. And indeed, that is what seems to have happened, at least with the information we have so far. Matthew Berman says, suddenly the world has access to an open-source model considered state-of-the-art. It beats GPT-4-0 on many benchmarks. What a time to be alive. Data Economy writes, in a dramatic turn of events, early benchmarking data for the forthcoming LAMA 3.1b models, including 8b, 70b, and the colossal 405b, were leaked on the local Llama subreddit today.

9:33The preliminary results suggest that Llama 3.1405b could potentially surpass the performance of the current industry leader, OpenAI's GPT-4O, across several critical benchmarks. Should that happen, it would represent the first instance of an open-source model eclipsing a leading closed-source LLM. A couple other things that the community has honed in on. Hold aside the question of whether it is the state-of-the-art and which of the benchmarks we should care most about, if these benchmarks are anywhere close to accurate, the thing that is for sure is that it is a GPT-40 class model. In the same way that we can debate around whether GPT-40 or Claude 3.5 Sonnet is exactly the better or more performant model, this would add Llama 3.405b to that conversation.

10:11It's important to note here that in addition to Llama 3.1.405b, we're also getting the 3.1 updates to the 8b and 70b models. Maxime Laban writes, the new 70b also looks insane with a significant boost of performance compared to the previous version. And that I think is a part of the story that is actually flying a little bit under the radar comparatively, i.e. how much better these smaller models have gotten as well. Chris at HingeLoss also noted that there seemed to be an updated license which removed the prohibition on using Llama 3 to improve other models. He used DiffChecker to compare the 3.1 versus 3 Llama license.

10:44Going back to this idea that the big story once everything settles might be the smaller models, Kyle Corbett points out that if the leaked benchmarks are correct, Llama 3.170B beats GPT-40 Mini. Iden McLeod really sums it up when he says, if these Llama 3.405 benchmarks are real, this will be the world's best model, in the hands of everyone to tune, cheaper than GPT-40. It's hard to overstate how fast everything is changing. Picking up on some of the themes that we were discussing in yesterday's show about the move of competition to the smaller end of the spectrum, Leighton Spaces' SWIX called what's happening the under-100B model red wedding.

11:17He writes, I do not think that people who criticize OpenAI have sufficiently absorbed the magnitude of disruption that has happened because of 4.0 Mini. He points out that both Lama 370B and GPT-4.0 Mini score an 82 on the MMLU, but GPT-4.0 Mini is priced at a sixth of the price of Lama 370B. He points out that many of the models that were state-of-the-art just three months ago are now being dominated by new counterparts. Swix writes, What's the depreciation rate on the flops it took to train them? GPT-4 took$500 million to train and it lasted a year. Intelligence too cheap to meter, but also too ephemeral to support greater than five players doing R &D.

11:51Is there an angle here I'm missing? SWIX also compared the new 3.1 benchmarks to Llama 3.0 and came to the conclusion that there was a huge bump for the 8B and that the Instruct 70B is mildly better, but that the 405B is still behind flagship models. Now we are slated to actually get this model today and should be able to independently start verifying these benchmarks. And until then, we won't know exactly what the situation is. However, if things are close to what they appear to be, we really are, it appears, living in a new paradigm where open source has by and large caught up and closed the gap with closed source.

12:23One thing I will be watching is whether that prompts OpenAI and Anthropic to release new models that redefine the state of the art once again. For now though, pretty interesting times, lots to pay attention to. And of course, I appreciate you hanging out and listening or watching as always. Until next time, peace.

12:43Thank you.