The State of AI In 13 Stats

0:00Today on the AI Breakdown, the state of AI in 13 charts. Before that on the brief, Mistral is raising at a$5 billion valuation just months after raising at a$2 billion valuation. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our Discord, our newsletter, and our YouTube channel. Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. Today, we kick off with a fundraising story and exclusive from the information. France's Mistral, which is of course an open source darling and one of the hottest LLM builders, is apparently in talks to raise fresh capital at a$5 billion valuation just a few months after closing money at a$2 billion valuation.

0:46Mistral closed on$415 million at that previous valuation back in December. So why might there be appetite for this next fundraise at a higher valuation? I think there are a couple pieces of this. First of all, in the subsequent months, Mistral inked a big deal with Microsoft. That included releasing their first non-open source model through Microsoft's Azure, which could easily have suggested to investors that the company had a pathway to real revenue. More relevant though, in my estimation, there are only a tiny handful of companies that have any credibility as a plausible leader in the frontier and foundation model space.

1:19You've got OpenAI, Anthropic, Google, Meta, Grok, and Mistral. Maybe you could nudge it open to Databricks and Cohere, but the point is that this is a very, very small group. What's more, it's a group that has gotten smaller. Inflection, which raised more than$1 billion last year to get the compute to actually compete in this area, still ultimately decided that it was too big a mountain to climb and ended up mostly going to Microsoft with that recent announcement. I believe that investors have a broad sense that the value for winning even a piece of this market is so tremendously huge that the valuation for credible competitors in the space, that very small handful of credible competitors, is just not going to be the barrier.

1:59That doesn't mean that every investor is willing to sign up at any valuation. I know several investors, for example, who have balked at the valuation of other companies on that list. But I think by and large, what this reflects is the sense of just how significant the pie is for winners in the LLM space. And despite being only a year old, Mistral is a serious contender. Now, in addition to that unconfirmed funding news, we also got more model updates from Mistral, who uploaded their 8X22B model, as well as a brand new Instruct 8X22B with function calling on Hugging Face. The model is fluent in English, French, Italian, German, and Spanish.

2:33It's a$141 billion parameter model. It has a 64 ,000 token context window. And it's released under the same Apache 2.0 license that they've released previous models on, which includes, of course, an OK for commercial use. Mistral also dropped a chart showing the relationship between performance and cost with three of their models, Mistral 7B, Mixtral 8x7B, and Mixtral 8x22B, all being at the very best end of the performance and cost ratio. Bindu Reddy of Abacus writes, apparently the new Mistral model beats Claude Sonnet and is a tad bit worse than GPT-4. In a couple of months, the open-source community will fine-tune it to beat GPT-4.

3:08This is a fully open-weights model with an Apache 2 license. I can't believe how quickly the OSS community has caught up. And if you're wondering why Mistral would be pushing hard right now, one of the reasons might be that, whether it's this week or very soon, Meta's first versions of Llama 3 are soon to arrive. Mistral's big competition in the open-source LLM space when it comes to developer Mindshare is, of course, the Meta Llama models. So, to me, overall, not that surprising to see a big fundraise, although going from a$2 billion to a$5 billion valuation over the course of four months by any previous industry standards would be slightly insane.

3:42Next up, another big theme right now is the move to try to get models to work on local devices and local hardware. This is, of course, Apple's great pursuit as it tries to bring AI models to the iPhone and its other suite of devices, but it's happening everywhere. You've heard lots and lots of companies talking about the AI PC era, for example. AMD has this week unveiled a new set of chips that are specifically designed for these AI PCs and will be included in devices from HP and Lenovo. In making these chips for the AI PC market, AMD joins both Intel and NVIDIA who are also looking at the space.

4:15Another random little area in which AI is finding its way into hardware, Logitech is now shipping a mouse that has a dedicated AI button. The Verge writes, Tomorrow's AI PCs may not only have a co-pilot key on their keyboards, Logitech is introducing its own way to summon ChatGPT2. It's called the Logi AI Prompt Builder, and it'll use a dedicated button on your mouse or keyboard. What's more, it appears that this prompt builder doesn't just pull up a chatbot when you press the button, but also offers an interface to actually improve prompts. So are AI buttons the next big thing in devices? I suppose we will have to wait and see.

4:49Finally today, as every company tries to figure out how to participate in the AI revolution without supporting or encouraging misinformation, companies are trying to figure out what their solution to letting people know that something was generated with AI. Basically, every big social network is thinking about how to do this on their platforms, and Snap is reportedly planning to add a watermark to AI images that are created with Snap tools. Interestingly, this will not be an invisible watermark that can be picked up by a specific reader, as we've seen the approach being from some others, but instead will be a visible mark, a translucent version of the Snap logo with a white outline.

5:24The company says that removing the watermark will be a violation of their terms of service. The question, of course, will be whether that means they've just incentivized people to go to Midjourney or Dali or somewhere else to create an image that doesn't have a watermark. Right now, all these companies are still taking baby steps in this space because there's no regulatory pressure so far to do anything more. How much longer that stays the case remains to be seen. For now, though, that is going to do it for today's AI Breakdown Brief. Next up, the main AI breakdown. Hello, friends. Quick note before we get back to the show.

5:54I'm so excited to share that Super Superintelligent is now live. Superintelligent is a platform for fast, fun, and super practical, useful AI learning. We have something like 300 video tutorials adding 30 to 50 each week, covering every topic in AI you can imagine from LLMs to image generators to case studies, use cases, basically everything that tells you how to use AI and what to use it on in short, fast, four to seven minute tutorial videos, which are paired with step-by-step instructions that help you actually use these tools as well. It's$20 a month for unlimited access, and I would love to see you there.

6:30Check it out at bsuper.ai. That's bsuper.ai. Attention, AI Breakdown listeners. Consensus 2024 marks the 10th gathering for all things crypto, blockchain, and Web3. However, importantly, this year's agenda will also dive deep into AI-driven transformation. And the speaker lineup includes the leading minds and innovators at the forefront of this digital renaissance. Don't miss the Consensus AI Summit to cut through the hype to find where true transformation and opportunity lie. Listeners to this show can get 15 % off registration with the code AIBreakdown. Visit Consensus2024.coindesk.com to learn more.

7:06Some of the folks who will be at Consensus this year include Guillaume Verdun, aka Beth Jezos, founder and CEO of Xtropic, as well as spiritual leader of the accelerationist movement, Neil Stephenson, co-founder of Lamina One, and Brendan Eich, the CEO of Brave Software. Again, go to consensus2024.coindesk.com to learn more and get 15 % off registration with the code AIBreakdown. Welcome back to the AI Breakdown. As you probably have garnered at this point, I am a sucker for a big think piece, especially if it has some numbers or data or anything that we can comment and react to. So today for our episode, we are looking at a recent piece by the Stanford University Human Centered Artificial Intelligence Center called the AI Index, the state of AI in 13 charts.

7:50This is a sum up of the Institute's larger AI index, which is a 300 plus page report, and I think does a good job of giving the high levels. So what we're going to do is we're going to look at the charts, discuss Stanford's conclusions, and then I'll share if there's anything that I think differently around or just see it in any sort of different way. Their first note, they call a move towards open source. In 2023, they counted 98 open foundation models, 23 limited foundation models, and 28 with no access. They noted that of the 149 foundation models released in 2023, which was itself double the number released in 2022, 65.7 were open source.

8:27That compared to only 44.4 % in 2022 and 33.3 % in 2021. If you've listened to this show at all, I mean, heck, if you listen to the Mistral section in the brief today, this will probably not surprise you. There has been a huge push towards open-source LLMs, with companies like Meta initially leading the charge, and in the process surprising Google and OpenAI, I will say, and then the banner being picked up by others like Mistral as well. However, Stanford's second chart notes that Open comes at a cost to performance. They write, Closed-source models still outperform their open-source counterparts.

9:00On 10 selected benchmarks, closed models achieved a median performance advantage of 24.2%, with differences ranging from as little as 4 % on mathematical tasks to as much as 317.7 % on agentic tasks. This is a chart that I really don't like. It's not that I disagree that there is still a difference between closed and open models, but I think that this is an area where medians and averages really fall apart. Basically, who cares about not-state-of-the-art models? Certainly no one that I know that's actually working with them unless they're making a choice for specific cost reasons. So a better comparison would be, what's the difference between the most performant open source models and the most performant closed models?

9:40In other words, what is the gap between open source and GPT-4 class performance? I think the overall result would still show that gap, but it might be less dramatic than it seems here. Next, Stanford notes that, quote, industry dominates AI, especially in building and releasing foundation models. They point out that since 2019, Google has led in releasing foundation models with a total of 40, followed by open AI with 20, and academia far behind, with UC Berkeley releasing three and Stanford releasing two. Once again, no big surprise here given how much money is at stake with these models. Putting a fine point on that, they note that industry released 108 models last year as compared to academia's 28, industry-academia collaboration's nine, and government's four.

10:21Next, Stanford notes that the prices for training models has gone up significantly. They write, One of the reasons academia and government have been edged out of the AI race, the exponential increase in cost of training these models. They note that Google's Gemini Ultra cost$191 million worth of compute to train, up from OpenAI's GPT-4, which cost an estimated$78 million. In comparison, they point to the original Transformer model from 2017, which cost around$900 to train. The next one might be a little surprising given how much we talk about geopolitical competition around AI, but here's how Stanford sums it up.

10:55What AI race? They write, at least in terms of notable machine learning models, the United States vastly outpaced other countries in 2023, developing a total of 61 models. That compared to China's 15, France's 8, Germany's 5, and Canada's 4. I do think that these numbers are telling and important. However, I think that what's more of interest to people who are looking at the race dimension of this is less the number of models, and more, once again, the state-of-the-art and the performance of those models. In other words, to people who care about this race, it wouldn't matter if the US released 100 times more models than China if China released the best models.

11:30Next up, they talk about how much more performant these models have gotten. They call the chart move over human. They note that when it comes to image classification, basic level reading comprehension, English language understanding, visual reasoning, and multitask language understanding, AI has in general exceeded human performance at this point, and competition level mathematics is getting very close. Of course, this brings up the question of what superintelligence and AGI actually mean, which is of course a debate that you see constantly on AI Twitter. Here's another one that might surprise you, given how often we talk about the big money going into AI on this show.

12:03Overall, Stanford notes that total AI private investment has actually gone down between 2021 and 2022, and again between 2022 and 2023. Specifically, 2021 saw$132.36 billion invested in AI, down to$95.99 billion in 2023. However, generative AI has seen a massive surge, going from$4.17 billion in 2021 to$2.85 billion in 2022, to a 10x increase to$25.23 billion in 2023. So clearly the emphasis in what in AI is being invested in has made a big shift, commensurate with the change in the focus on ChatGPT, MidJourney, and the like. Once again, the United States is seeing the biggest portion of that investment.

12:45In 2023, it saw 67.22 % of all private investment. Next, almost 60%. percentage points behind was China, which saw 7.76 % of investment, the UK, which saw 3.78%, and then a set of companies including Germany, Sweden, France, Canada, Israel, South Korea, and India, which all saw between 1 % and 2 % of total private investment. Here's one that's particularly interesting to us as we are building Superintelligent, a platform for practical learning around AI. Based on a McKinsey and Company survey, Stanford reported on how businesses are using AI currently. 26 % are using it for contact center automation, 23 % are using it for personalization, 22 % for customer acquisition, 22 % for AI-based enhancement of products, and 19 % for creation of new AI-based products.

13:31Surveys also saw an increase in organizations that said they were using AI in 2023, reaching 55%, which was up from 50 % in 2022. What about concerns around AI impacting jobs? Overall, Stanford writes that globally most people expect AI to change their jobs, with more than a third expecting AI to replace them. Gen Z and millennials are anticipating more substantial effects, with 66 % of Gen Z compared to 46 % of boomers believing AI will significantly affect their current jobs. Individuals that have higher incomes, more education, and decision-making rules also think that AI will have a bigger impact on their employment.

14:06The numbers they quote, 57 % believe that AI will change how they do their current job in the next five years, and 36 % see AI replacing their current job in the next five years. Stanford also looks at different countries' attitudes towards AI, and specifically where people were worried about it. At the top end of the concern is Australia, with 69 % of people saying that AI makes them nervous, while at the bottom end of the spectrum is Japan, who only had 23 % worried. Lastly, they looked at the change in regulation. In the United States, the number of AI-related regulations jumped from around 15 to 25 between 2022 and 2023, although of course none of those are any sort of comprehensive regulation, which remains the big question going forward.

14:47Overall, I think a lot of this probably reflects what you might assume if you're paying a close attention to this space, but it's still really interesting to see this data captured in this sort of way. So big thanks to Stanford University's Institute for Human-Centered Artificial Intelligence for publishing this report. And for now, that's going to do it for the AI Breakdown. Until next time, peace.

15:11Thank you.