Game of Thrones, but for AI Chips

0:01Today on the AI Breakdown, we're looking at a set of news in the AI chip wars. Before that on the brief, OpenAI fires some researchers for alleged leaks. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown.net for more information about our Discord, our YouTube, and our newsletter.

0:24Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. We kick off today with a couple pieces of OpenAI news, the first being quite the tea. According to the information, OpenAI has fired two researchers for leaking sensitive information. The two researchers include Leopold Aschenbrenner and Pavel Ismailov. Both of these researchers have been on the team dedicated to AI alignment and safety research, and according to the information, Aschenbrenner was also an ally, their word, of OpenAI chief scientist Ilya Sutskever, who of course we have no idea where he actually is or what he's actually still doing with OpenAI.

1:00We also have no information about what the two allegedly leaked. Right now, it's all a big bucket of speculation. The information talks about Ashenbrenner's ties to the effective altruism movement, the most notorious proponent of, of course, being Sam Bankman-Fried. And the intimation is that the leaks probably had something to do with the disagreement around how OpenAI was proceeding with regard to safety. But again, that's all speculation right now. Whether these two actually talk or if they just show up at a competing lab remains to be seen. But that is the story. Next up, earlier this week, OpenAI announced a more advanced GPT-4 Turbo with Vision model, but it was only available initially through the API.

1:35Well, now they've brought that new advanced model to ChatGPT. The company writes, Our new GPT-4 Turbo is now available to paid ChatGPT users. We've improved capabilities in writing, math, logical reasoning, and coding. For example, when writing with ChatGPT, responses will be more direct, less verbose, and use more conversational language. We continue to invest in making our models better and look forward to seeing what you can do. So the example they gave of it being less verbose, The prompt was SMS reminding friends to RSVP to my birthday dinner invite. The previous response was, hey, friend's name, just checking in to see if you've had a chance to RSVP for my birthday dinner.

2:09I'm finalizing the table reservation and really hope you can make it. It wouldn't be the same without you. Please let me know by RSVP deadline. Looking forward to celebrating together. The new response is, hey, friend's name, just a friendly reminder to RSVP for my birthday dinner. Hope you can make it. Let me know soon. Now, of course, the proof is in the pudding. And so far, I've seen a lot of people say that this does feel like GPT-4 is now once again caught up with what they were seeing with Claude 3 Opus. There's some disagreement around whether that's the case with coding, but I've seen some people talk about how the coding has gotten significantly better as well.

2:37Of course, there is a larger question, which is why we're only getting these very incremental improvements rather than some big model launch, and a lot of speculation that OpenAI is still very much holding things back. Next on the brief, Amazon has added well-known computer scientist Andrew Ng to its board. Andrew is an extremely well-known figure in the AI space, having led projects at Google and Baidu, and seemingly showing that Amazon is here to compete in this area. Finally, the Humane AI pin is getting lots of reviews, and they're not all that great. For example, the Verge's review is called Humane AI Pin Review.

3:09Not even close. For$6.99 and$24 a month, this wearable computer promises to free you from your smartphone. There's only one problem. It just doesn't work. The reviewer writes, I came into this review with two big questions about the AI pin. The first is the big picture one. Is this thing anything? In just shy of two weeks of testing, I've come to realize that there are, in fact, a lot of things for which my phone actually sucks. Often all I want to do is check the time or write something down or text my wife, and I end up sucked in by TikTok or my email or whatever unwanted notification is sitting there on my screen.

3:39Plus, have you ever thought about how often your hands are occupied with groceries, clothes, leashes, children, steering wheels, and how annoying slash unsafe it is to try to balance your phone at the same time? I've learned I do lots of things on my phone that I might like to do somewhere else. So yeah, that is something, maybe something big. AI models aren't good enough to handle everything yet, but I've seen glimmers of what's coming, and I'm optimistic about the future. That raises the second question. Should you buy this thing? That one's easy. Nope. Nuh-uh. No way. The AI pin is an interesting idea that is so thoroughly unfinished and so totally broken in so many unacceptable ways that I can't think of anyone to whom I'd recommend spending the$699 for the device and the 24 monthly subscription.

4:13Now, there really was not just one nasty reviewer. There were a bunch of people who said something similar. However, for a more generous and let's call it historical point of view, I'll read this tweet from investor Kala Jalanbo, who writes, With AIPin, Humane will go down in history as the company that created the world's first ever AI computer. Yet, as history proves, version 1.0 is just the beginning of a founder's vision. Those who understand what it takes applaud entrepreneurs that have the guts and imagination to challenge giants, rather than scrutinize them like they're already trillion-dollar mega caps.

4:40The Humane team are pioneers that catalyze the conversation around how we apply AI in consumer hardware. And that, in itself, merits recognition and respect. I think that's a great way to put it, and very clearly we are at the beginning of a flowering of new, interesting experiments when it comes to AI hardware. Who knows where that will all land? That's going to do it for today's AI Breakdown Brief. Next up, the main AI breakdown. Welcome back to the AI Breakdown. Today we are catching up on the AI chip wars, which, if they get any more intense, are really going to start being reminiscent of Game of Thrones pretty soon.

5:12So let's start with the most recent story, although just one of many from this week. Yesterday, Bloomberg's Apple whisperer Mark Gurman reported that Apple is planning to overhaul their entire Mac line with a new custom silicon chip they're calling the M4. It's obviously the fourth in their line of M chips. And this one is specifically focused on AI. Now, what's not surprising is that Apple is thinking about AI and leveraging their custom silicon to try to win an advantage in that space. In fact, a lot of our conversations about Apple's strategy, or likely strategy when it comes to AI, have been about how it is trying to, on the one hand, increase the power of chips in its devices, and on the other hand, shrink powerful models such that they can be running on device without having to touch the cloud.

5:53What is surprising about the announcement is that the M3 was first released just five months ago. That's an extremely short turnaround, even for Apple, which has a reputation for making its old technology obsolete very quickly. According to sources, the M4 chip will come in at least three varieties, and Apple is planning on updating every single Mac model with it. Now, obviously on this show, we are focused on the AI dimension of this, but just to understand the broader business dimension for Apple, their Mac business is tough right now. Mac sales fell 27 % during the last fiscal year, and even during the holiday period, revenue was flat.

6:25Between that and their seeming sort of strategic flailing, given their recent ending of Project Titan, their electric car project, their unclear AI strategy, overall it feels like Apple needs to get things back on track. Investors seem to like this report. Apple had actually been down 13 % on the year, very different than the other AI-focused tech stocks, but on Thursday saw their share price go up 4.3%, which was their biggest single-day gain in 11 months. The names of the M4 chips include Donnan, which is the entry-level version, a middle-powerful version called Brava, and a top-end version called Hydra.

6:57These are their code names, so they might not be called that when everything comes to light. In terms of getting more information. Sources expect us to hear more about these chips and the devices that they will run in at the Worldwide Developer Conference, which is coming up in June. Bloomberg also writes, As part of the upgrades, Apple is considering allowing its highest-end Mac desktops to support as much as a half terabyte of memory. The current Mac Studio and Mac Pro top out at 192 gigabytes, far less capacity than on Apple's previous Mac Pro, which used an Intel processor. The earlier machine worked with off-the-shelf memory that could be added later and handle as much as 1.5 terabytes.

7:28With Apple's in-house chips, the memory is more deeply integrated into the main processor, making it harder to add more. So, one of our competitors in the Game of Thrones for chips is Apple, but an old player seems to be coming with some new tricks as well. Ars Technica writes, Intel's Gaudi 3 AI accelerator chip may give Nvidia's H100 a run for its money. Intel claims 50 % more speed when running AI language models versus the market leader. So basically, this week, Intel held its Vision 2024 event, and their big reveal was a new AI accelerator chip that they're calling the Gaudi 3. Now, Intel is pursuing a number of different strategies simultaneously.

8:02They're increasingly competing with TSMC as a chip fabricator, i.e. a builder of other people's chips, but the Gaudi 3 suggests that they're not giving up on their own chip business either. Now, it is important to note that although Gaudi 3 is projected to have faster performance in terms of training time and inference than NVIDIA's H100, the H100 is no longer NVIDIA's most powerful chip. The H200 is not out yet, but is expected basically any day now. And then there is of course the Blackwell B200. One of the big questions is what the price of the Gaudi 3 will be. Analysts think that it could be an attractive alternative if it can come in meaningfully under the$30 ,000 to$40 ,000 in H100 costs.

8:38I think the big takeaway from this Game of Thrones perspective is that Intel seems willing to fight on multiple fronts at once and is putting out some pretty compelling products. Moving on in our battle, Google has announced its own ARM-based CPU to support AI work in data centers and is also introducing a more powerful version of its TPU or Tensor Processing Unit chips. The new CPU is called Axion and will be initially used to support Google's internal AI workloads before rolling out to business customers of Google Cloud later this year. Google said that the Axion chips are already powering YouTube ads, Google Earth, and other Google services.

9:10Said Mark Lohmeyer, Google Cloud's vice president, Axion is built on open foundations, but customers using Arm Anywhere can easily adopt Axion without re-architecting or rewriting their apps. Reuters reports that Axion Arm-based CPU will offer 30 % better performance than general-purpose ARM chips and 50 % more than Intel's existing processors. Google also discussed their overall cloud strategy in terms of what chips they offer, specifically announcing that they are not offering AMD chips. Again, Mark Lohmeyer said, At this time, we're deploying NVIDIA GPUs and our own TPUs, and he added that they are also supporting Intel's new central processing unit, which he claimed was, quote, good for inferencing certain workloads.

9:45Lohmeyer said, We offer those three and we feel good about that combination for our customer base. Meta also made a chip announcement, which was perhaps understandably a little bit overshadowed by the fact that they revealed that Llama 3 was coming soon. But the company says that their new Meta Training and Inference Accelerator, MTIA, is a quote, big piece of its long-term plan to build infrastructure around how it uses AI in its services. Meta said in a post, meeting our ambitions for our custom silicon means investing not only in compute silicon, but also in memory bandwidth, networking, and capacity, as well as other next-generation hardware systems.

10:17Meta had first announced their V1 of the MTIA in May of 2023, and said at the time that they were focused on providing those chips to data centers. That V1 was not expected to be released until 2025, but Meta now says that both that first version and a new MTIA chip are in production. Writes The Verge, Right now, MTIA mainly trains ranking and recommendation algorithms, but Meta said the goal is to eventually expand the chip's capabilities to begin training generative AI like its LLAMA language models. Over in the world of Amazon, CEO Andy Jassy discussed the chip wars in his annual letter. He wrote, To date, virtually all the leading FMs have been trained on NVIDIA chips, and we continue to offer the broadest collection of NVIDIA instances of any provider.

10:56That said, supply has been scarce and cost remains an issue as customers scale their models and applications. Customers have asked us to push the envelope on price performance for AI chips, just as we have with Graviton for generalized CPU chips. As a result, we've built custom AI training chips named Tranium and inference chips named Infercia. In 2023, we announced second versions of our Tranium and Infersia chips, which are both meaningfully more price-performant than their first versions and other alternatives. This past fall, leading FM maker Anthropic announced it would use Tranium and Infersia to build, train, and deploy its future FMs.

11:25We already have several customers using our AI chips, including Anthropic, Airbnb, Hugging Face, Qualtrics, Ricoh, and Snap. So the point here is not necessarily that Amazon announced anything new, but that the battle is significant enough that it warrants mention in this annual letter. But what about the geopolitics of chips? Obviously, this is a huge global issue, and we've gotten some news recently in that respect as well. The Biden administration is continuing to distribute money through the CHIPS Act, with the latest recipient being TSMC, who are receiving a$6.6 billion grant to increase their U.S.

11:56manufacturing. TSMC is, of course, building out their first major U.S. plant in Phoenix. It's actually two plants with this money being used to build a third. And as part of the announcement, TSMC said that they're increasing their total investments in the U.S. from$40 billion up to$65 billion. This is so far the second biggest grant under this program, with the first being the$8.5 billion in grants, plus$11 billion in loans, that were announced for Intel a couple of weeks ago. As an aside, TSMC is doing well right now in general. They saw a 16 % rise in quarterly sales, which outstripped projections and also was the fastest growth in more than a year.

12:29Another chip-exporting nation, South Korea, is investing around$7 billion to stay competitive in chips as well, said South Korean President Yoon.

12:59So, lots and lots going on in the AI chip world. Like I said, this is a battle that is not slowing down even a little bit. That is going to do it, however, for today's AI Breakdown. Until next time, peace.