AI Accelerators Could Change Everything - DTNS WEEKEND

0:02This is another Daily Tech News Show Weekend Edition. Happy weekend, everybody. Thanks for joining us. uh we're gonna talk chips today with andrew main andrew welcome back to the show it's good to have you buddy hey thanks for having me here tom how do you uh how do you give your title these days uh i'm a partner at zero shot which is an investment firm i started with some other open ai alumni but you were also uh the first prompt engineer at open ai um former magician Well, current, I guess you're still a magician, right? Never goes away, Tom. Yeah, I also host the official OpenAI podcast for OpenAI.

0:39So that's, so do that too. And then with a couple of our friends, Justin Robert Young and Brett, we've got Interdimensional, which is the consulting firm, which is like growing like crazy. But most of my time is spent now with zero shot talking to founders who are trying to build. Well, thanks for taking time to chat with us today. I heard you talking about Grok on the Attention Mechanism podcast that you do with Justin Robert Young. Grok is one of a series of companies that's doing chips to accelerate services. Cerebris is another one providing infrastructure that kind of speed up how LLMs work.

1:15Can you explain a little bit about what these companies are doing and what these chips are versus what everybody else hears about, which is NVIDIA and maybe Google's TPU? Yeah, and we have to, of course, specify for audio listeners. We talk about Grok, we mean G-R-O-Q. and not XAI's, Elon Musk's AI system, which it gets so confusing. So yeah, Grok, GROQ. Isn't that what trademark law was created for? Anyway, sorry. Well, you know, Robert Heinlein deserves some credit somewhere along the line. Yeah. Oh, yeah, right. I think first coined the term. So what you had was Grok was founded by some former engineers, some of which worked at Google on the TPU.

1:58And Google had realized early on that, you know, GPUs, graphical processing units, are very, very good for doing a lot of different, you know, tasks, you know, because they literally were designed for video game graphics, etc. And then people found out the same kind of math for that. It's really good for certain A applications and doing tensor mathematics, etc. But they weren't designed for that. But, you know, you already had an ecosystem building out, you know, NVIDIA GPUs, building this out. But then Google said, you know, what if we built something that was more specific to that? And the kind of math they do is called, you know, tensor math.

2:34And they said, let's create the TPU, the tensor, basically a tensor processing unit. And decided to focus on doing that internally, building it out. It was a very, very long haul for Google. And for years, people were like, is that really worth the effort? Is that going to pay off? And then a few years ago, they started training the Gemini models. It's funny, people paid attention to it now because like, oh, look, they trained the Gemini all on top of the TPU. I'm like, yeah, they did that. Press releases said they've been doing that for years, but it shows you how successful that's been for them.

3:00But other people left and said, hey, we have different ideas. Grok was founded with the idea of like, hey, we think we can actually make something even more specific because they looked at there's training, which is one part of it. You have to train a model to train, you know, the next ChatGPT or Gemini or Cloud model. But then there's inference, which is when you and I ask a question to the chat bot, and that doesn't need to have the same hardware. And so Grok said, well, let's just build our own specific chip They call it an LPU, a language processing unit, basically based upon the idea of instead of having a thing that has to be networked with thousands of other pieces and all this other architecture, a simpler architecture that gets rid of everything on the GPU you don't need and just focuses on inference.

3:40And it was very, very fast, extremely fast. And they could take a conventional model. Like right now, you could go to Grok and you could run. I mentioned this into a letter to my investors I talked about a couple weeks ago, was that the funny thing was the fastest models that OpenAI has aren't served by OpenAI. They're served by Grok and Cerebrus. And I use examples in there. But so you could go to Grok and run OpenAI's open source model, the GPT OSS 120B model, which is a great model. And it's super fast, way faster than you're going to get in chat GPT. And that was the idea that they were exploring was saying, like, let's not use the kludgy GPU hardware, which is really dynamic and robust, but I can get rid of everything I don't need and focus on the thing I want.

4:22So it's basically, we're just whittling down, right? Instead of using a GPU. Yeah, you're moving your cache around, you're focusing on the area. You know, the GPUs are great, but there are many, you know, many computers that computers and we actually run a bunch of tiny little computers to do stuff. And they can say, we can make this a lot more efficient. We don't, we can get complexity. We can put more storage here. We can move us around to do that. Yeah. And then Grok said, well, if you only need inferencing, you can get even more efficient. What about Cerebris, though? Because they're doing it in a different way, but also speeding things up.

4:51Cerebris is another one of these companies. It had been a long bet for a very long time and a lot of skepticism around it. And so their idea was what we call wafer scale. Instead of when you look at when you make a microchip, you take your little wafer and you go in there and you have a bunch of little microchips that you punch out from the wafer. They're like, what if we use the whole wafer? What if we put all of the chips onto the wafer, have a much, much larger, you know, sized processor? So they were working on a wafer scale. The theoretical upside of it is, is you've reduced the size of everything you need down to this, which means I don't need to have five cooling units.

5:26You should have one really powerful cooling unit. You know, it means I don't have to have all this other stuff, hardware going into it. I can build everything, most things I need in there. The hard part is building things at wafer scale is challenging. It's a new area. We've explored it for years, but it's been a very big challenge where we've got decades of really good machinery, lithography, everything else like that. You need to go from tape out to building your conventional chips. But they said, hey, we think this is a bigger bet for the future. I just spoke about them at a conference a couple months ago about now we're starting to see some of the benefits of this because Grok is fast, Cerebrus even faster.

5:59Cerebrus is even a faster, much, much, and crazy sort of fast. And so, you know, I brought up Grok because, as you mentioned, NVIDIA just made this deal basically acquiring Grok. Like it's a licensing deal. It's a thing probably designed to avoid any kind of antitrust. But the leadership of Grok are now going to go work with NVIDIA. And now they're going to probably make this part of their pipeline. But you'll still be able to get inference from there. I said, hey, this is very interesting. This is where everything's headed this year. This is the trajectory. Speed matters. It's not just how good is the token.

6:29It's how fast I get the token, you know, and how cheap it is. And then I mentioned the thing that's interesting, it's even faster than Grok is Cerebris. And then just a few days ago, OpenAI announced a deal with Cerebris to buy 0.75 gigawatts of inference from them. And so with inference, you're talking about the processing on the end, even though the Cerebris chips can also do the training as well. And actually, I think this, and let me clarify, they said compute. So it may not have actually been a difference. All right. Yeah. But where this comes up to, the point I've been trying to make to people is that right now you have the three – I would say that the coding models – I think Gemini is good.

7:09But when you go look at the boards, people tend to think Claude by Anthropics is great.

7:18OpenEye's GPT 5.2 codex is actually getting really, really – is very, very capable in many cases. Some people say it's better, what have you. But it's slower. It's a little bit slower. You know, and OpenAI has a different approach to how they do stuff. They really wrap their heads around how to do reasoning and do this and get more performance out of that. But in both systems, whatever you're using, you still have to wait a while to get a response. But when you go use something now, like you can take, you can go play with this, get an API. You can go into, you know, sign up with your Google account in either Cerebrus or Grok, and you can run your own little test and go to like a GPT-OS S120B model, which is a capable model, and see how blink of the eye it is and compare it to ChadGPT, Gemini, or Claude.

7:57and it is mind-blowing. I do this demo for people all the time. I just do side-by-side, show you how, I think I've got on my Twitter account right now, show you how fast it is. And that makes you think, if you're working at a software development firm and you're trying to make code updates and all that, 15 seconds may not seem like a lot, but you add that up throughout the day and that's a big difference. And one of the things that helped Anthropic was they built a great code model, but they also charged a lot for it, right? And so if you look at the cost per token on Opus, It's much higher than some of the other standards.

8:28And that really helped Anthropics margins to say, hey, they're doing great profits. But it showed, too, that sometimes it's not about cheaper. It's about efficiency. It's about the quality of the token, but also the speed of the token is now going to be that important. Yeah. And I've seen people compare this to the coming of high bandwidth internet in the early days of the dot-com era. The difference between having a dial-up versus a DSL versus a cable broadband connection really changed what you could do. And a lot of people at the time were like, why does it have to be faster? And it turns out we found out lots of reasons why it was good that it was faster.

9:10You kind of gave me a little bit of an insight into the question I'm about to ask. But can you think of some examples of things that you think might happen because people have this faster response time, this faster training and inferencing? Sure. Think about – this last year, the biggest paradigm was back in – remember like late 2024, there was this open eyes working on something secret. What is it? And they came out with the O1 model. The O1 model is a reasoning model. And what made reasoning models really important was there had been some question. If I asked a model, if I asked GPT-4, can it ever give me a better response than the response it gives me the first time?

9:49And it turned out actually it can. If you let it break a question down, you feed those parts of the question back into it and let it reason through those things, it actually can do much more, even though it's the same model. It's why you can now download, you know, you know, a GPD OSS, you know, the 12 billion parameter model on your own laptop and get GPT four level answers, which wasn't supposed to be possible. And so other companies have excelled at this, too. But because this process means you spend more time letting the model think about it means it takes more time. One of the measures at which we measure, you know, computing compute capabilities, how long an AI can think about a problem, come up with an answer.

10:26And you'd think, well, why would I care how long it works? Because actually length of time suggests, if it length of time and it comes with an answer means it doesn't give up, it can find paths, but that takes time. So if we said, okay, it took three hours to answer this, this math Olympiad question, but now I can go put it into Cerebrus and I can get it in 45 seconds. Now the cost is really high, but I can get that answer sooner. and that's a really big value proposition to say, yes, I can give you the same answer if you're willing to wait time, but if you don't want to wait, you need that answer now.

10:58Great. We can do this. And would it also be able to give you a higher quality answer because it could spend more time because you could say, okay, if I'm not worried about cost and I say, okay, I'm willing to wait an hour. One of the things, when you go look at the leaderboards and you'll see what these different models are performing at, you might see a number that says like five X or whatever. That means they're doing like five parallel processes and going to choose the best of answer for that. So I can ask a really thorny question about proteonomics or something or some finance question and say, you know what?

11:29Run it 10 times, but run it 10 times sequentially. And because it's faster, it's just, you know, I'm not as timed down before. There's a lot of interesting things happening in this area of chip architecture. At CES, we talked about the Halo chips coming to Raspberry Pi. There's some RISC-V stuff being done by Jim Keller called TenStorrent. Sambanova is out there. I don't know as much about them, but there's a lot of this sort of thing. It does seem like Grok and Cerebris are grabbing the headlines more. Grok's path seems pretty clear to me now. It's going to be an element of NVIDIA's offerings.

12:07If you had to guess, where do you think Cerebris goes? Well, the deal with OpenEye is a pretty big deal. you know, when they're agreeing to provide 0.75 giga, that's a huge amount of compute. I mean, that is an incredible amount of compute, which sounds, it sounds like that's also going to be a large part of their capacity because, because they have to figure out the timeframe to build these chips and how long they'll be able to do it. You know, I, I don't think it's necessarily an exclusive deal, but you know, some things can for a while be basically effectively because you just, they're just buying your capacity.

12:44I think that it could be a big win for Waferscale. I think that it's really what's exciting about this was they made a bet. There have been a lot of people for years who've been saying that Waferscale is the way to go, but we hadn't seen any really big high returns on that. But this is now, you know, I think I may be correct, but it was like another$10 billion deal or something just incredible for that. And I think that now that's showing for other people working on that scale of making those kinds of bets that there's actually some profitability in there. Yeah. And I think it's worth repeating that Grok is an LPU.

13:23And as such, it's a piece that NVIDIA can use. With wafer scale, you mean like that's the whole dinner plate of silicon, right? That's crazy that that's the entire chip. Yeah. Yeah. Yeah, I mean, it's a crazy thing to think about, like, you know, what's possible when you start building stuff? Because so much of what you have to do in when you want to run, you know, either if you want really fast inference or you want to do really fast training or anything like that, you're tying a bunch of different, you know, systems together. When I was at OpenAI and we worked with Microsoft, you know, we had a lot of these guys who were a few guys who were from network architect guys, you know, the guys with the beards, you know, gnome-like guys that would go into these centers and figure out how to wire up all this stuff and to kind of put together these supercomputer systems.

14:13And getting a lot of that reduced onto one wafer is a huge, huge, huge deal because it reduces a tremendous amount of complexity. it in theory makes things a lot more energy efficient but it's just that early stage and that's been the thing of like is it really you know economically feasible maybe so now you know again at 750 milliwatts of compute is an incredible amount of compute like just to give you an example of like if you look at you know uh i'll pull up a number here but like you know how much open i was using, you know, for compute when they first launched, you know, chat GPT, you know, it was a number related to that.

14:53Yeah. And if I understand this right, the wafer scale means, depending on the size of the model, you could conceivably have the entire model in the chip, right? Yeah, you can, you can, in theory, you could have your memory, you could have, you know, you have your memory, you have your, your, your, all of your different units that go into there doing it, you're not having to break things across a board, all that's in size there. And you can, you can put, you can do an incredible amount of density in there too. Yeah. And then if you like, that's just one. So, so now what can you do when you add a bunch of different ones together and all the compute that you're talking about from open AI, that's, that's a lot of power at your hands.

15:31Yeah. I'm, I'm friends with George Gilder, who's been talking about this for years. And he talks about like the whole, like, you know, the data center on the chip, you know, the idea of getting rid of data centers and it all being there like and i'm i love george but i'm like ah i don't think we're getting rid of the data centers i just think we're gonna make them even more powerful yeah but but but but it does distribute that i mean that's the other area we're seeing is that you know um uh where this is going and just for an example so in 2024 opening i had 0.6 gigawatts so which is about you know you know 600 you know uh i'm gonna get my my term terminology incorrectly here but like literally uh you know getting um uh the amount of compute that they're getting you know for that is you know um 750 it's like it was like you know more than the total compute budget they had in 2024 is what they just made a deal with three risk for yeah uh and that am i wrong am i wrong is that that actually helps with power efficiency?

16:34I mean, because you can do more with that chip? In theory, you know, the, the, yeah, 750 megawatts, I said kill it. So again, apologies to everybody here. Um, what you've done is that imagine that I have, uh, I want to, I have like 10 NVIDIA H100s, right? These, you know, the very, very incredible capable chips or the grace hoppers and new ones, whatever, whatever. Okay. Which means I've now got to have basically 10 of these, 10 of these cards or 10 of where they are. And I've got to have a cooling system for each one. Okay. Which means all of my plumbing, everything else I got to go in there complete, you know, increases the complexity of this.

17:10Whereas if I can have one cooling system that go in there and figure that out, I'm going to reduce that. I have to have my network cards. I have to have all these other things that go into there. A lot of other hardware I don't necessarily need. And part of it is because I got to send data further than I should have to. So in theory, it should reduce that considerably. Yeah. Well, this is wonderfully helpful. Thank you, Andrew, for walking us through this and sharing your expertise with us today, man. I always appreciate it. It's exciting. And I think that what we're going to see this year, you know, for the relates to all of us, that you think of how long you wait to get a video from Sora or VO and, and, or what it costs for a high quality image.

17:49And that's good. You're going to see downstream effects of that. You're going to see really, really highly intelligent models give you instant answers. When you talk to one of the problems people have with voice models right now, when you talk to a voice model is that they're not using the most state-of-the-art model there because of the latency. If you're using a thinking model, it's got to take 10 seconds to think about it. You're not going to connect that to voice because you're going to get impatient. But now you can take these really high-speed models and put it in a voice, and you're going to notice this when you start talking to these AI assistants, and they seem a lot smarter.

18:17Fantastic. Andrew, if folks want to follow you, what you do, get the podcast and other ventures you're involved in, what should they do? I would say on X, at Andrew Main, easiest way. Fantastic. Thanks again, Andrew. My pleasure. And thanks everybody for supporting us. You can get these shows ad free, dtns.substack.com or patreon.com slash dtns. Have a great weekend, everybody. Talk to you on Monday.

18:55The DTNS family of podcasts, helping each other understand. Diamond Club hopes you have enjoyed this program.