The LLM for Coding Competition Heats Up!

0:01Today on the AI Breakdown, we're looking at the latest from NVIDIA and the state of the AI chip wars. Before that on the brief, some new competition in the LLM for coding space. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our newsletter, our Discord, and our YouTube channel. Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. Today, we kick off with one of the hottest parts of the AI space, which is, of course, the competition around LLMs for coding.

0:38Now, it is perhaps no surprise that developers are some of the earliest adopters of artificial intelligence. I mean, it's the cohort that built the tool, so it's not surprising that they're using them. What might be surprising is just how profligate they already are. In June, GitHub released the results of a developer survey where they found that 92 % of developers in the US are already using AI coding tools both in and outside of work. On top of that, 70 % of developers say AI coding tools will offer them an advantage at work, thinking that it will lead to better code quality, faster completion time, and an easier ability to resolve incidents.

1:12And four out of five developers also think that AI coding tools will help make their team more collaborative. Given all that, it's not a surprise that there is a lot of competition around which LLM is best for coding. Two interesting announcements on that front. The first comes from Stability AI, who keep up their absolutely relentless release schedule with the announcement of StableCode. The company claims that StableCode is the first LLM product entirely focused on coding. The announcement post writes, StableCode offers a unique way for developers to become more efficient by using three different models to help in their coding.

1:43The base model was first trained on a diverse set of programming languages from the stack dataset from BigCode, and then trained further with popular languages like Python, Go, Java, JavaScript, C, Markdown, and C++. After the base model had been established, the instruction model was then tuned for specific use cases to help solve complex programming tasks. 120 ,000 code instruction response pairs in alpaca format were trained on the base model to achieve this result. Still, the big sell for stable code is what they claim is the biggest token context window for any open LLM coding model. The version of stable code that is available as their long context window model, has a context window of 16 ,000 tokens, which they say can handle two to four times more code than previously released open models.

2:24Now, of course, as soon as this was released, developers started looking at how it compared in terms of coding benchmarks. Prince Kanuma tweets, The Instruction-Toon stablecode variant performs competitively with the Lama 2 75 billion params variant on the HumanEval benchmark. It also outperforms all other Lama 2 variants while significantly smaller at only 3 billion parameters. Interestingly, Stability AI CEO Ahmad Mostak didn't love what evaluation models the company had presented. He said, Human eval isn't the best benchmark for a code completion model. Team are working on better ones for this and other code tasks, as well as specific models for use cases.

2:58Still, can only be the benchmark in front of you. Clem, the CEO of Hugging Face, however, responded and suggested that Stability AI add the Stable Code model to Hugging Face's multilingual code evals, which Ahmad dutifully agreed to. That said, Lubna Ben-Ala, an ML engineer at Hugging Face, wrote, I added stable code completion alpha 3b to the leaderboard. It's competitive with smaller size models on Python, Java, JavaScript, and C++, but has poor performance on the rest. Now, this is less than 24 hours old, so I think it's not exactly clear where this sits relative to other LLMs that are used for coding.

3:31But what is clear is just how big an area of competition this is. Another AI-related coding announcement yesterday came from Google. Lior at AlphaSignalAI writes, Just in, Google announces a new browser-based code environment. It will bring the entire full stack and app development workflow to the cloud. It also includes generative AI features based on Palm 2, code generation, code completion, translating code between languages, code explanation. To learn more about this, you can go to idx.dev. Google describes the project by saying, These days, launching applications means navigating an endless sea of complexity.

4:05We felt this pain at Google, so we started Project IDX, an experimental new initiative aimed at bringing your entire full-stack multi-platform app development workflow to the cloud. And while this isn't just about AI, that is definitely one of the big features they're pushing. They write, Work quickly and efficiently with AI assistants from Google built-in, including code generation, code completion, translating code between programming languages, explaining code, and more, all powered by Kodi, a foundational AI model trained on code and built on Palm 2. Project IDX is currently available by waitlist only.

4:34Next up on the brief, what has to be one of the most obvious and predictable about faces in history. After throwing an absolute hissy fit around the Drake track that came out earlier this year, Heart on My Sleeve, which was of course an AI track, now it appears that labels including Universal Music and Warner are in conversations with Google about licensing artists' IP, including their voices, melodies, and more, to create an IP-approved platform for people to develop AI music. The reporting comes from the Financial Times. The way they characterize it is discussions between Google and Universal Music are at an early stage and no product launch is imminent, but the goal is to develop a tool for fans to create these tracks legitimately and pay the owners of the copyrights for it, said people close to the situation.

5:15Artists would have the choice to opt in, the people said. Now this reminds me quite a bit of an idea posted by Product Hunt founder Ryan Hoover in April. Back then he wrote, Free startup idea that will likely get you sued. AI Spotify. How it works. AI Spotify hosts AI-generated music of your favorite artists. Anyone can submit music and the best songs surface based on listens and likes. Music with the most listens earns a pro rata share of subscription revenue reserved for the original artists. For example, Drake could claim money generated from his likeness on the platform. Artists that do not want to participate can opt out entirely banning any music that uses their likeness or individually allow songs they endorse.

5:49Of course, there are many ethical and legal issues with this model, especially with labels, but maybe this is a germ of a shower thought that has potential. Now, this to me seems absolutely like the only path that makes any sort of sense. TLDR, you can't put the genie of AI music back in the lamp once it's released. There's simply going to be too many models, too many tools, and too much publicly available content to train those models on to not see just a ton of AI tracks. In that context, to use the classic phrase, if you can't beat them, join them. It appears that Warner and Universal are thinking in exactly the same way and are out to get their cut.

6:24A quick one from the world of policy and health AI. Reports have been that Google has been testing its MedPalm 2 model in hospitals for some of this year, and at least one senator, Senator Mark Warner, is not happy about it. The Virginia Democrats sent a letter to the CEO of Google on Tuesday, basically warning them off a further rollout. The letter said, while the technology has shown some promising results, there are also concerning reports of repeated inaccuracies and of Google's own senior researchers expressing reservations about the readiness of the technology. Warner basically accuses Google of sort of the same thing that Jeffrey Hinton has accused the entire big tech space of, which is racing to get AI models that aren't fully tested and aren't fully safe out because of the pressure of competition.

7:06So far, Google is standing its grounds, saying that the rollout is extremely limited and that the company doesn't control any private data. Lastly, a nice positive one to end the AI breakdown brief today, a new AI model called Heliolink 3D has been used to spot asteroids that could, in the future, pose a threat to Earth. The team from the University of Washington claims that the new AI model can identify asteroids with just half the observations that were needed before. Mario Jurek, the director of the DRAC Institute at the University of Washington that developed the model, writes, The solar system is home to millions of rocky bodies ranging from small asteroids a few feet in diameter to dwarf planets the size of our moon.

7:42Most of them are distant, but a number orbit close to the Earth. These are known as near-Earth objects. The closest of these, whose trajectories take them within 5 million miles of the Earth's orbit warrant special attention. The large, very nearby objects are known as potentially hazardous asteroids, PHAs. They're systematically searched for and monitored to ensure they won't collide with Earth. Astronomers search for PHAs using specialized telescope systems. The NASA-funded ATLAS survey is a prime example. To find asteroids, ATLAS takes images of parts of the sky at least four times every night.

8:10A discovery is made when they notice a point of light moving unambiguously in a straight line over the image series. Now from there, Jurek explains the development of the Rubin Observatory, which is an observatory that will be based in the Chilean Andes and which could increase the discovery rate of PHAs. Jurek writes, to be even more efficient, it will visit spots on the sky just twice each night rather than the four times needed by present telescopes. But with this novel observing cadence, we need a new type of discovery algorithm to reliably spot space rocks. And that is, of course, where the Helio-Link 3D model came in.

8:38The big deal, Jurek says, is that it can identify asteroids with much fewer, up to 50 % and more dispersed observations than required by today's methods. Now, Jurek says that the asteroid they discovered, 2022 SF-289, while coming close to the Earth, its closest approach brings it within 140 ,000 miles of Earth's orbit, which is closer than the moon, it currently poses no danger of hitting Earth for the foreseeable future. Still, the urgency is that while we've identified 2 ,350 PHAs, scientists believe there are more than 3 ,000 yet to be found. And so perhaps AI is not just for stealing our jobs, but for avoiding a real-life replay of Michael Bay's 1990s classic Armageddon.

9:17Thanks as always for listening or watching to the AI Breakdown Brief. I'll be back soon with the main AI breakdown. Before we get into the main AI breakdown, I want to tell you about today's sponsor, Supermanage. If you work in a professional setting, you probably have some version of a one-on-one meeting, either with the people that work for you or the people that you work with. Unfortunately, all too often, those one-on-one meetings become glorified catch-up calls. Don't you wish you could jump right to the stuff that really matters? That's where Supermanage comes in. Supermanage AI magically distills your team's public Slack channels into a real-time brief on any employee, any time.

9:54Catch up on contributions, work in progress, challenges they're facing, sentiment, everything you need to show up ready for a truly meaningful conversation. And it's completely free. Visit supermanage.ai forward slash breakdown today to start making the most of your one-on-ones. And thanks again to Supermanage for sponsoring the AI Breakdown. Welcome back to the AI Breakdown. Today we are talking all about the state of the AI chip wars. And obviously any conversation about said chip wars has to start with NVIDIA. Yesterday, NVIDIA CEO Jensen Huang made a set of announcements in LA about new generative AI initiatives from NVIDIA.

10:32And so let's start by looking at what was announced and how it relates to this larger AI chip battle. First up, for those of you who think the metaverse is dead and just a hype cycle gone by, it's live enough that a company as big as NVIDIA is still investing resources in it. NVIDIA had some updates about its Omniverse platform yesterday. Specifically, they say they're advancing the development of OpenUSD. USD stands for Universal Scene Description, and NVIDIA describes it as a 3D framework enabling interoperability between software tools and data types for the building of virtual worlds. And they actually make an analogy to help understand the comparison point for these initiatives.

11:09Jensen said at the event, just as HTML ignited a major computing revolution of the 2D internet, OpenUSD will spark the era of collaborative 3D and industrial digitalization. NVIDIA is putting our full force behind the advancement and adoption of OpenUSD through our development of NVIDIA Omniverse and Generative AI. The second big Generative AI announcement from NVIDIA was their AI Workbench. The NVIDIA AI Workbench is, they say, a unified, easy-to-use toolkit that allows developers to quickly create, test, and customize pre-trained Generative AI models on a PC or workstation, then scale them to virtually any data center, public cloud, or NVIDIA cloud.

11:42So effectively, this is another entrant into the enterprise AI workspace. If you're a regular listener of the show, you will have heard me talk about, for example, Amazon's Bedrock, which is making a bet that there won't be one winner-take-all model, but that enterprises who have a keen sense of needing to control their data end-to-end, and a real fear of losing that data for training purposes to some third party, are likely to opt for something that's much more customized and built on either A, open-source tools, or B, enterprise-grade AI platforms that are built by partners they already trust with their data.

12:14That's the play that Amazon is going for, and it seems like that might be something that NVIDIA is trying to do as well. Manavir Das, the vice president of enterprise computing at NVIDIA, said, Enterprises around the world are racing to find the right infrastructure and build generative AI models and applications. NVIDIA AI Workbench provides a simplified path for cross-organizational teams to create the AI-based applications that are increasingly becoming essential in modern business. Giving more detail about what it actually does, they write, Access through a simplified interface running on a local system, NVIDIA AI Workbench allows developers to customize models from popular repositories like Hugging Face, GitHub, and NVIDIA NGC using custom data.

12:51NVIDIA's Dr. Jim Phan, one of the mainstays of AI Twitter, tweeted yesterday, happy to share that NVIDIA is partnering with Hugging Face. We love open source software community. NVIDIA DGX Cloud will be accessible with Hugging Face to create and customize generative AI models for the enterprise. Yes, we do have a cloud. The official announcement reads, Integration of NVIDIA DGX Cloud and Hugging Face platform to speed LLM training and tuning simplifies customizing models for nearly every industry. CEO Jensen Huang again said, Researchers and developers are at the heart of generative AI that is transforming every industry.

13:23Hugging Face and NVIDIA are connecting the world's largest AI community with NVIDIA's AI computing platform in the world's leading clouds. Now, as part of this, Hugging Face is spinning up a new service they call Training Cluster as a Service, which will help simplify the process of creating new custom models for the enterprise. But maybe the biggest announcement, at least when it came to what Wall Street was thinking about, was around the forthcoming next-generation GH200 Grace Hopper Superchip. Basically, we had gotten information about the chip in the past, but at yesterday's event, we learned more about the platform built around it.

13:53NVIDIA says that this platform will deliver 3.5x more memory capacity and 3x more bandwidth than the current generation offering. The other big thing that we got was information about availability. NVIDIA says leading system manufacturers are expected to deliver systems based on the platform in Q2 of calendar year 2024. Now, one of the big use cases that NVIDIA is looking at is, of course, the next generation of data centers. In fact, Huang said in the address, this processor is designed for the scale-out of the world's data centers. Now, NVIDIA is by far the dominant player in the AI chip space.

14:27Estimates I've seen suggest that they have between 80 % and 83 % of the AI chip market. Their biggest competitor is, of course, AMD. And just a couple months ago, AMD announced its own new chip, the MI300X, which is meant to challenge NVIDIA's place at the top of the AI heap. Now, not surprisingly, AMD has said that this new chip and the architecture surrounding it was specifically designed with large language models and other AI models in mind. When AMD premiered the chip, they pointed out that it can use up to 192 gigabytes of memory compared to NVIDIA H100's 120 gigabytes of memory. The argument is that this would improve inference, and that by adding memory on AMD chips, developers might not need as many GPUs in total.

15:07Importantly, AMD also added a software package around its AI chips, which is something that NVIDIA offers and has historically given them a lead. Now, outside of these chip giants, there are also a number of startups that are trying to elbow their way into the space. One that made news recently for a nine-figure investment from Hyundai and Samsung was TenStorrent. Part of what makes TenStorrent notable is the extremely high-profile team that they brought on, including CEO Jim Keller, who has previously worked on AI chips at places like Apple. And one of the interesting things about this most recent funding announcement is that the participation of Hyundai was more than just financial.

15:41Hyundai Motor put in$30 million and Kia put in$20 million, and part of their plans are to partner with TenStorrent to jointly develop chips that can be built into future vehicles. Other chip startups that have recently raised money include SEMA.ai, IR Labs, and Ethernovia. However, the other big chip player might be the big tech companies themselves. In April, reports came out that Microsoft had been working on its own AI chips for years. From The Verge, Microsoft is reportedly working on its own AI chips that can be used to train large language models and avoid a costly reliance on NVIDIA. The information reports that Microsoft has been developing the chips in secret since 2019, and some Microsoft and OpenAI employees already have access to them to test how well they perform for the latest large language models like GPT-4.

16:24The project is apparently codenamed Athena. Google is also apparently making a push for its own chips. Just a couple weeks ago, the Wall Street Journal published a piece called, In Race for AI Chips, Google DeepMind Uses AI to Design Specialized Semiconductors. Google has been working on their Tensor Processing Unit chips since at least 2016. And the new report had researchers from Google's DeepMind claiming that they had, using AI, discovered a more efficient and automated way of designing these chips. Amazon has also discussed making its own AI chips. In a conversation with CNBC, he said, I think of generative AI as having three macro layers, and they are all really big and important.

17:00The bottom layer is the compute, all the machine learning training and inference. What matters in the compute is the chip in there. There has really been one chip provider. Supply is more scarce, and it's expensive. It's why we've invested over the last few years in our own customized training chips and inference chips, which will have much better price performance than anywhere else. We are quite optimistic that a lot of the machine learning training and inference will be done on AWS chips and compute. So TLDR, NVIDIA is the 800-pound gorilla in this space, but everyone is coming after them. They're established competitors like AMD, novel startups trying to use partnerships to get a leg up like TenStorrent, and of course all the biggies like Amazon, Google, Microsoft, OpenAI who don't want to be at the behest of any other company.

17:43And frankly, right now, it appears that there may be enough market to go around for everyone. Just three days ago, CNN wrote, The crushing demand for AI has also revealed the limits of the global supply chain for powerful chips used to develop and field AI models. CNN points to Microsoft's annual report, which identifies for the first time the availability of GPUs as a risk factor for investors. During his testimony before the Senate in May, OpenAI CEO Sam Altman joked about how ChatGPT was struggling to keep up with all of the demand for its service. At that time, Altman said, We're so short on GPUs, the less people that use the tool, the better.

18:16Now, part of the issue isn't just that the demand for AI increased so fast, but that the makers of GPUs also have bottlenecks around a number of the key supplies for actually manufacturing more chips. And finally, there is the political dimension of all of this. Three weeks ago, representatives from NVIDIA, Qualcomm, and Intel were at the White House to discuss US-China policy when it came to the export of AI chips. Reuters writes, The chip industry is keen to protect its profits in China, as the Biden administration considers another round of restrictions on chip exports to China. Last year, China accounted for$180 billion in semiconductor purchases, more than a third of the worldwide total of$555.9 billion, and the largest single market.

18:55Now, in the wake of the existing export restrictions, NVIDIA started selling a tweaked and depowered AI chip specifically designed for the Chinese market. But now some politicians in Washington are saying even that should be restricted. As wonky and technical as it might seem on the outset, the story of the chip space from an industry perspective, a competitive perspective, and a political perspective is going to be absolutely integral to how AI develops and evolves. So hopefully this gave you a better sense about where the chips lie right now, pun absolutely intended, and I of course will keep you posted as more developments happen.

19:29If you're enjoying the AI breakdown, please click that like or subscribe button. And if you're listening to the podcast, I would so appreciate it if you would take the time to leave a five-star rating. Those ratings go a long way to helping people discover the show, and I appreciate you for taking the time to do so. Appreciate you guys as always. Until next time, peace.