Grok Bots and How CPUs are used in Agentic AI

0:00So check this out. So this week was actually the release of Grokbot, which is an agentic AI platform. So you think about it like an open claw, but like extremely easy to use so that like everybody can run agents all the time right now, even if your computer is closed. So this is a very important development, I think, because it's not so much for what the GrokBot is as a product, but what it means from an underlying substrate of compute and specifically what it means for CPUs.

0:38All right. Hello, listeners. Welcome to another Semi-Doped episode. I'm Austin from Chipstrat, and this is Vic from Vic's Newsletters. And if you're watching this on YouTube or something, you'll notice Vic is in a different location. So Vic, tell us, where are you? And then I also saw that you are trying this GrokBot, which I saw on X, but I have not tried. I honestly don't even know anything about it. So I want to hear, where are you? And then tell me about GrokBot, because I know you've been playing with it. Awesome. I am in the Bay Area. I came here for like hot chips next week, which is very exciting.

1:14A lot of people are coming down for this conference, but I decided to show up a week early because I was actually in Taipei attending the Open Compute Project APAC, like the Asia Pacific region version of this conference. And I was like, yeah, I think it makes just sense to like come here a little bit early because I've got so many people to meet in the Bay Area. It's been amazing. I've met so many amazing people and so many amazing things have happened that we should, eventually it'll all come out at some point and we'll discuss it on the podcast or whatever when the context is right. But yeah, it's been an amazing week.

1:53And so I am in the Bay Area. I got my little camera rig with me, but I don't have my semi-doped sign, which I'm very bummed about. So I'm going to travel with my neon sign next time. Yeah. So like Grothbot. Grothbot is an interesting thing because this is an agentic AI software. Like, I don't know. They call it an agentic harness is the term for it. It's a piece of software you just download off the website and you install it. And you've got an interface where it pretty much everything works out of the box. Right. Right. So what you do is the first thing that comes up and you go, oh, hey, here's the, you know, what do you want it to do?

2:37And I said, hey, I want you to be a chief agent. And then you are the chief of my agents and you're going to talk to all the other agents. And then it's like, oh, what other agents do you want? I'm like, oh, I want an agent that does this. I want an assistant agent. I want a, I don't know, a news monitoring agent, something like that. And it just spins it all up automatically. Right. It's fancy. It's really nice. So I think this will open up the doors to a lot of people starting to use agents now because the whole OpenClaw approach and the Hermes approach was like a little bit difficult. Like you've got to install it and you've got to go into the command line and you've got to set up an API key and you've got to like set up all of this stuff.

3:21It is a little bit involved for everybody to use, but GrokBot takes care of everything. Nice, amazing. Okay, so like OpenClaw and Hermes, Hermes, however it's pronounced, those ran locally on like your own machine, right? And this Crock-Bot, this must, how much of it runs locally on the machine versus like in the cloud? You said you downloaded something. Yeah, this is just a piece of software that essentially communicates with a computer in the cloud. this is the nice part of it right uh which is what i really like about this is it is actually running your own little dedicated computer in some server somewhere so you have like a little virtual machine that's running somewhere and everything is within that box all your tools are installed there all your sign-ins are installed there like you don't need all these mcp connections and these plugins and those plugins it's just think about it as your computer but somewhere in the cloud it's a vm it's a virtual machine right the nice thing about this is that all of the security practices and everything that matters for these agents to run in its sandbox environment is all contained within there so if your agent has like signed in into one tool it is available easily across all your agents and everything because it's it's a local computer you have.

4:49That's the difference compared to like OpenClaw and Hermes. Nice. Amazing. That does sound so convenient. So in this, would you say it's better than just like having a bunch of cloud code agents running locally on your machine that you have to manage? So it's a give and take, right? There's a lot of control when you have your own cloud code agents. you can specify a whole lot of stuff to the finest detail like which model you should use for doing what and you can token optimize and be like saying hey for these queries you can do you know so you can set up open router as a model provider through the cloud core interface and you can in open router you can say okay use the auto router or use this hierarchy of models such that these requests are routed here and those requests are routed there.

5:42This takes all that out of the equation. You don't know what's happening. You just have tokens and you use those tokens. Nice. That's it. You don't know what's being used. You know, I think the easy button is how it's going to go. Agents will go to the masses where they don't need to go buy a Mac mini. They don't have to install something. They don't have to know what open router is, which maybe is getting acquired by Stripe now. But they don't have to configure and token max or, you know, get token efficient, token min, you know, they can, they can just hit the easy button. So then does that mean I should buy a bunch of CPU stocks?

6:22If Grok is going to spin up all these VMs? We'll get to the not financial advice part really quickly. But you mentioned Mac mini. So I just wanted to like touch upon that. Like there was this when agents were rolled out not so long ago, it was like only December last year that we even heard of this thing. It was such a rush to get compute that even Intel was caught unawares of the demand for CPUs, right? In their earnings call in February, they said, oh, we have a lot of CPU demand. We had no idea it was coming. So the question is, why did all that happen? If people observed keenly enough at the time, you could kind of tell why what's happening because the moment agents showed up, everybody ran and bought a Mac Mini.

7:07So what exactly were they buying? Were they buying a Mac Mini for its memory? Were they buying it for its NPU based, you know, the graphics processing they have in there? Or were they buying it for just compute or a CPU, right? So that question, if you revisit and ask yourself why people bought the Mac Mini, that will tell you something. and the reason is people didn't want to run OpenClaw on their existing laptops because you're giving it access to your whole computer and that's dangerous because you don't know what it's accessing, you don't know what it's going to do with the information on your computer, does it access sensitive information, medical records, you have no control.

7:54So the idea is what if I could buy a separate machine, a Mac Mini, keep it only for this purpose install open claw on it and then have that as my agent computer and then there i don't put all the sensitive information i give it only what it wants and then it works all the time now the problem with that setup and this is familiar to anybody who's run a home server setup like i've run one for like 10 years but keeping that up and running is a real big pain because you are responsible for power right like if there is an outage or if your computers i don't know like some part of the psu blows up or you know something happens power goes down or if your internet is out uh that's a problem because you don't have connectivity into your own machine at home but you have to set up networking like how are you going to access your PC at home when you're traveling like I am right now, how are you?

8:55So you've got to set up some kind of a VPN network. Like we, you know, you and I set up tail scale for all this stuff all the time. It's not that simple for everybody to go off and be like, hey, if I tell my mom, hey mom, all you need to buy is this Mac mini and then set up tail scale and, you know, set up a UPS so power doesn't go down and have some failover internet so nothing happens. She'll be like, I'm out. I can't use this stuff. Okay, you keep your agentic PC. I'll just go back to my regular laptop. Yeah, 100%. Yes, it is all about the easy button for 99 % of people, seriously. Yeah, yeah.

9:29So that was the problem. So people were buying these computers just to sandbox their environment and have its own, you know, area for agents and stuff to run. At that time itself, I thought about it and thought like, wait, what is the right way to do this? Like, do you have to buy a PC and go do this? No, at the time too, what you could do was you could get on a VPS, you could rent out a little virtual machine on a server, and you could install OpenClaw on that one. And then you just go off and run it on the cloud, which is a much better solution because you don't actually have to buy a, I don't know, $600,$800 machine, which got so much more attention because like everybody's buying it.

10:12So it was like a craze and prices went up and all that. Assuming you don't want to run models locally because some people actually bought the mac mini the high version with the maxed memory version great decision in hindsight i think you could pull out the memory and sell it for a profit compared to the cost of the machine itself now but consider it an investment but yeah if you wanted to run models locally you could do it right but i don't think models could all be run locally they're really big so you want to run a frontier model like the fable class model with your agents there's no way you could do that right right yes i have an agent computer by the way and i do run a few models locally but it's obviously smaller things for the non-frontier tasks including um like audio uh basically like transcription you know voice to text like so i'll run this podcast through on that when i post it to our semi dope newsletter to give the transcript to people that's like the simple kind of thing that can run on a local box.

11:15But for most people, when they want to spin up agents and they want to do things, I think that there's a lot of work that you would want an opus or a fable quality, you know, frontier model. Yeah, it depends on the workload. You know, you might want a better quality model. So what is the right way to do this, right? Now, that's what Grokpot does all of this. It does all of this for you because you don't actually have to, when you get Grokbot, this is not a sales pitch and they're not really a sponsor or anything, but what problem it solves is all I'm trying to articulate. I'm not suggesting anybody go and I'm not shilling this product.

11:50Okay. So don't get it. Actually it's$200 a month. Most people probably won't get it. Okay. It's very expensive. So I'm not saying you should, but what this, what Grokbot solves is you just get the virtual machine with all the integration tied in it asks you when you install like hey what integration do you want i said i i want my google calendar um i want my google drive you know some some of these things you can just say add add add and all those things are natively installed and ready to go for you you don't have to do anything like there's canva there's figma there's so many tools out there you don't have to set up each one so you set it up there into your grog bot instance and it's available across all your agents.

12:32And then it runs on a virtual machine and you don't have to be responsible for anything, including the security of that virtual machine, because that's Grok's problem. I hope they have experts taking care of that. I don't have to become like a cybersecurity expert now. Right. I mean, I don't know. I thought Elon fired all the people at X or slimmed it way down. But anyway, carry on. I'm sure they've got their ducks in a row. Maybe they don't. Okay, fine. and then deploy Fable to be your security expert. There you go. Got to be a creative problem solver. Yeah, whatever. Like all this is, you don't have to worry about anything.

13:09You get a virtual machine that sandbox securities supposedly ensured. Then you've got all your tools installed. You run your agents. And it's amazing. So now we go to the question of CPUs. So do we need them? Right? Like let's now go into the question of do we, why do we really need CPUs? because we never even mentioned GPUs and all of this stuff. We are sure for doing the inference, we want to do GPUs, but we really never haven't even touched upon what CPUs have really done for us in the last six months since agents showed up. And I thought that's what we should like dig into now. Let's dig into it because from your example, okay, so we talked about running cloud code on like your laptop.

13:52So there's going to be work done on the CPU there as well as calls to a GPU in the cloud. We talked about running OpenClaw on a Mac Mini. So now you've got another CPU on your desk. It's not your laptop, it's your Mac Mini. It's in its own little contained environment. It's got a CPU and yet it's still going to the cloud for the GPU. And then of course, now you're talking about using GrokBot, which is actually spinning up a VM in the cloud. So a CPU in the cloud. And so it's a CPU in the cloud talking to a GPU in the cloud and then communicating results back down to you. So there's actually, and we'll get into this, but what I'm alluding to is there's different work happening.

14:38In some scenarios, it's happening on your local CPU. Some it's on your home server CPU. In others, it's on a rented cloud CPU. And so there's obviously interesting implications. But yeah, let's get into it. Let's first talk about what do the CPUs even do versus what is the GPU doing? Yeah, yeah, yeah. Before the show, we were talking about a nice analogy we could come up with this. Yeah, I'll let you explain that analogy because this is a very nice framework for which people can think about where GPUs lie, where CPUs lie, and how we should think about this going forward. yes okay so ultimately like i i think of a good way of like where the value happens is the gpu the gpu is like the the genius you know um now it's like the ph the the genius that has a phd in everything which of course you don't always need um but the the gpu is like the genius and then but at the end of the day once like the brain the genius like thinks of all the work that needs to be done, which I mean, this is what I'm doing with Fable is I'm trying to say, like, you know, give me your very thoughtful approach or Opus 2.

15:48And then once it is decided, like what work needs to be done, ultimately, most of this work, or it depends on the workload, but a lot of the work can actually be given to like assistants, like little doers, little task carry outers. So ultimately, you know, I think about the GPU is like the genius that is doing all the thinking and delegating work and then ultimately the CPU as the assistants that once told what to do, they can go carry out all the work that the genius told them to do. Yeah. So ideally you don't want a genius doing all the work. Like the genius is thinking about it. Genius has paid a lot of money per hour to come up with this brilliant ideas.

16:32And you don't want the genius going off and I don't know, putting paperwork in the drawers or you don't want the genius looking up some information from a file that's in the drawers or you don't even want the genius going out to fetch the mail like ideally in an ideal world you want to have the absolute genius aka the gpu just doing what it does best actually there's one we can break this analogy down into a little bit more detail So if all the workers are like CPUs, like CPU cores doing their thing, for example, I think we can break down what kind of worker that is too. Because there is something called the head node CPU.

17:21And the analogy to the head node CPU is an assistant to the genius. The assistant to the genius, which is the GPU, the job of this assistant is to keep handing the genius work. You don't, you know, if the GPU, if the genius needs coffee, you give the genius coffee. If the genius needs water, you give the genius doesn't stop working because it's lost, you know, you're losing money if the genius stops working. So you want to make sure that once the genius is done with this task, the next folder of materials is ready for the genius work to be done. And the host node CPU will be like, here you go, sir.

18:01This is what you've got to do next. And the genius is like, let's go. let's do this. So that's the host node. And it has to have a, I'd say it has to have a pretty fast per co-performance because it's constantly monitoring what the GPU is doing. Are you done? Are you done? No. Okay. Like, let's get this ready. Let's keep it ready. But as soon as the GPU is done, this has to hand it in like, you know, no delays, no wasting time. Do not waste the genius time. So that's the host node CPU. Yes. Yeah. I love it. I love it. It's, it shows that you want like a high clock speed uh or a very responsive cpu but also obviously you want like the communication between the cpu and the gpu between the assistant and the genius to be as fast as possible right so that um you know that there's no so for example you ideally you wouldn't have the assistant like uh be in a different building and then they have to come over and check in and then walk back to their building and then come back and check in right you want them in the same room talking with each other.

18:59And so maybe to take it a layer deeper, there's differences between a coherent host and a standard host with these host nodes. So a coherent host would be like Grace Blackwell, where you've got the GPU and CPU, and they can talk over a shared really fast network, so like NVLink C2C, versus a standard host where you know maybe you have um just some hoppers and you take an x86 processor and you put it on the board but it's still talking over pcie so that that would be a little bit more of an equivalent to like uh you know they're in separate buildings or separate rooms so the the the chip to chip the c2c protocol that the gpu and the cpu uses the coherent protocol that you know the coherent cpu would be somebody who's standing right next to the genius and looking over.

20:00I'm like, oh, here you go. Here you go. Right. Right there. Right. And if they, yeah, exactly. And if they have like memory coherency, it's like I can see the, I'm the assistant. I can see the genius's notes and I could like jot down on his notes too or something. Yeah. So I was thinking of what HBM is in this analogy and this is where we push it too far and we shouldn't, but we'll do it anyway. but I think of HBM as the stack of papers next to the genius's desk right the genius is not going to get up off the chair and go and get it from the cupboard or anything HBM will be the stack of papers on the desk and the genius can just take it and keep working on it and putting it back on or whatever the the coherent memory coherency comes from the assistant having access to the same pile of papers on the desk because they're in the same room so if the assistant wants can You can also access the memory directly.

20:50Exactly, exactly. When you zoom out, to your point, this CPU assistant, this host node, it has one job, and it's keep the GPU fed. That's ultimately what the host CPU needs to do, keep the GPU fed. Then it starts to raise the interesting question. That made a lot of sense in the chat GPT area where everything's just a chat bot, and it's just like, yo, I'm asking questions, and it's just like, just keep the genius fed so we can respond to questions. Now, all of a sudden, when there's all this extra work to be done, like the genius is spewing out code and you need to compile the code, see if it compiles.

21:25Maybe you need to run the code. Maybe the genius is like, I need information from 50 different sources. Go hit SEC filings and go Google the web or search the web and go do all this stuff. Now, all of a sudden, the question is, can that CPU assistant go fetch all these filings? Can it take all this data? Does it have the memory capacity and bandwidth to analyze all this data and keep the GPU fed? Or is this too much work for that assistant standing right next to him? And do we need an army of other CPUs to help? Yeah, yeah. So this brings us to the concept of what is it? I don't know. Is there a thing called an agentic CPU?

22:05I don't know. That's like a marketing term probably, but I think it makes sense in this context. Yeah. So, you know, I'll give you my take on the agentic CPU. And actually, I just recorded a podcast recently with AMD, and they were aligned and thinking very similarly, which is, you know, if you zoom way out, sort of breaking this analogy, just going back to how things used to work. Back in the cloud era, obviously we had all these general purpose CPUs. And they back all of the SaaS products that we use in the cloud, right? And so now as we're standing up all of these GPUs so that we can ask them our little chat questions, of course they're not necessarily going to go communicate with a CPU that's already running an API server or running my database for my company or something.

22:57like that already has a job and that CPU is kind of already shaped to fit the workloads of like, I run big databases quickly, or I run tons of small little API servers quickly or whatever. So then the question is, okay, now fast forward, if we've got all these general purpose GPUs that already have jobs, they're already running SAS products or web servers or databases or whatever. And then we're now we've got these new geniuses that are standing up and there's they have a little assistant CPU next to them trying to keep them fed. But if now there's work that it's going to spill out because it's too much for the host node CPU, then the question is what CPU should those fall on and where should they live?

23:38And so there is this term of art that has come up lately, agentic CPU, which is more about, hey, these are CPUs that are dedicated to doing this agentic work, all this spillover work that the host CPU could do, but it's too busy keeping the assistant fed or it's keeping the genius fed. So there should be racks of CPUs that are dedicated to this task. But of course, it then raises the question, well, what should those CPUs look like? Should they look like the host? Should they look like the assistant? Or should they look more like the general purpose? Or do they need to have their own shape? So that's kind of like the framing for this new kind of middle ground agentic CPU.

24:26Agentic CPU. Yeah, so you don't want to use the host CPU, like you said, to do any of this work that the genius is asking. Like, hey, go search the internet. No, because once the host node goes off and does something like this, it stops feeding the GPU and the genius, and then the whole thing goes down. Like, what's the point? Now the genius is idle, which is the worst case scenario. So you need a different kind of CPU that does this stuff. It doesn't even have to be a different kind of CPU. In function, it's a different CPU. You could use the same CPU like the host node CPU. It may not need the coherency that we spoke about because it's not talking to the genius all the time.

25:07But otherwise, it could be the same CPU. But ideally, what you can think of this is like let's take a CPU with like, I don't know, let's say 128 cores or something. So what you can do is you can think of all the cores and as like a floor plan of an office building. And then you can kind of draw little boxes and say, OK, these four cores are the finance department. These four cores are my research group. These four cores are like my, you know, facilities team, whatever it is. Right. And so what you do is then somehow you've got yourself a little company and the genius is saying, hey, I need this stuff to be done.

25:48and the host CPU is going to yell across the room and say, hey, finance team, I need you to go and scrape up the SEC filings from last night. Go do that. And those guys will be like, okay, cool, we'll do that. And the team of four cores will go off and be like, okay, I'm going to do this stuff. And they'll report back to the host node CPU, the assistant, who will then feed it to the genius, the GPU, right? So in this scenario, a multi-core CPU that is either like one of the AMD's 256 core or Intel's more recent 288 core CPU, these are great because you can assign a lot of little departments across doing different functions.

26:35or you could have basically bigger teams of people. So if you have more cores, you can put like eight people in the finance team versus like four people in the finance team. So they kind of get the job done faster, right? But also you want each person to be competent. So each core should be fast. It should not be like an ultra slow. You don't want a team of eight interns versus a team of eight experienced people matters. So single core performance matters because if you get like very slow single cores, It's like you're putting eight interns on the job, which is fine. Depends on the task. Nothing against interns, but it really is okay.

27:11But it depends on what the task is. Totally, totally. And this analogy is good because, for example, your host CPU, the one that's feeding the genius, obviously it may have something like 88 cores, but it may be super fast, and those cores may be dedicated to keeping the genius fed. Now, all of a sudden, if you take that same CPU and you move it into this agentic situation, you might stop and ask yourself like, oh, do I want on my little floor plan with all my cubicles, do I want 88 cubicles of like really fast thinkers? Or sometimes would it actually be better to have 128 or 256 or 288 or 512, right?

27:54And it reminds me of working at lots of other companies where you look around and of course you've got some management tier and senior architects and stuff. But then you also have a lot of fresh out of college hires that are maybe cheaper and maybe they work a little bit slower. But guess what? Some of their tasks may be like go fetch this from the web and it's going to take two or three seconds to respond and you're just going to sit there anyway. So maybe it doesn't matter. and then you go process and you process a little bit slower. But to your point, there's definitely certain IO-bound, network-bound tasks and then just tasks that don't have to finish instantly because they're not blocking the genius.

28:38And therefore, zooming out, then you start to think, oh, maybe for these agentic CPUs, there's probably a lot of companies that are mostly interested in getting as much work done out of a CPU as possible. So having as many cores and as many threads as possible. And, you know, of course, zooming out even further, if you have a fixed amount of power in your data center, you might start to say, like, can I get the most amount of cores or threads per watt, for example, or per dollar, depending on your situation. Yeah, it's about the cost per employee now. Exactly. How many cores can you get and how expensive is each employee in that floor plan?

29:18Yes. That's important. So you want to have the most capable employee and many of them at the lowest possible cost of acquiring them. Yes. And that has to be suited to your workload. You don't want to put like really, really inexperienced people on a very complex task or very experienced people on a boring task, you know. So when people many times ask, what's the best agentic CPU or what's the best host node? I think host node CPU, you can kind of tell what it is. And we've both written about this in quite some detail on our sub stacks. Hostnode CPU, you can actually tell. You need coherency. You need speed.

29:54You know, cores are not all that important, but like the single core performance in, you know, getting stuff to the GPU is very important. So I think you can identify a Hostnode CPU when you see one. However, like you mentioned, the whole floor plan of employees that the CPU cores are, and you're going to fill racks of them in a data center, there is no reason you can't fill several racks with different kinds of chips. Like you've got some high core but low core speed chips in one rack. Then you've got really fast chips but not as many cores in another rack. And the whole problem now comes down to how do you architect your workflow to the workload that you are going to be using it for.

30:41I think that's very important. So there's no such thing as the right CPU. I think they're all right CPUs. it's all about the cost you know that you get per core kind of the total cost of ownership of the chip and how you use it together so that co-op co-design and co-optimization is very important and i spoke to an you know an intel speaker also at taiwan and it was a very good talk and he explained that they actually if you go to intel they actually do explain to you what the right cpu configuration should be for a given workload so they have recommendations that they provide for these kinds of things.

31:17So it's very important. Yes, yes. And this is why Intel had recently launched this past summer, like a P rack and an E rack for agentic AI, trying to make the point that it's not necessarily one size fits all. And there may be particular workloads that you can map to needing more performance, even for agentic AI tasks, or needing more or just wanting as much efficiency and as many workers as possible. There's a lot of analogies with like, uh, org design. And I'm thinking back to companies where it's like, Oh, budgets were bad. And, or, you know, the year was bad, budget was tight. So they had to lay people off.

31:57And so then the question is like, do you lay off the junior workers where they're all really cheap? So you'd have to lay off a lot of them. Or do you lay off like middle management where they're expensive? And, and, you know, a lot of times it's like lay off the middle managers. And this kind of reminds me of like saying, you know, I don't need the host node over here. Actually, I just want a bunch of maybe like junior employees or early career employees that are essentially cheaper, but can still get the work done. And then you've got the general purpose employee, general purpose CPU. I even lost the analogy now.

32:29Sorry, I'm in office mode now. I've even forgotten we're speaking about CPUs. But yeah, the general purpose CPU will be like, I don't know, it's just like good for everything. Like sometimes you've got these people who just have the skill to do quite a lot of different things. Those also can be valuable. So the agentic CPU is one thing, the host node CPU is another, and then you've got general purpose CPUs. These are just people you need, okay? Right, right. The receptionists or the people who maintain the building, you know, all important tasks, but you can't do without them. Totally. I will say at hyperscaler scale, they tend to still have particular workloads in mind and therefore they will buy a skew a general purpose cloud skew but still with a particular shape like we know this is memory optimized because we're going to have like in-memory databases you know running here and so what will be interesting whereas if you're just an enterprise you might buy more sort of generically shaped like yeah it's kind of fast enough and it has enough memory and that you know enough compute that we think it can do a broad set of tasks so it'll be interesting to see also how agentic tasks and agentic cpus if there's if there's any difference at the enterprise level you know kind of can i buy a one-size-fits-all agentic cpu rack versus um obviously hyperscale uh but now that i won't even go there because then you start to ask like are enterprises really going to be buying additional racks of cpus or are they going to be still leaning on the cloud here you know how's this going to play out yeah uh so rack scale ideas is is quite interesting too um i was at this is a different kind of a i realize this is a slight tangent but i just wanted to mention it because yesterday uh considering where we are recording or when we're recording this yesterday was the cerebrus announcement of the their new rack scale solution and so we always thought of cerebrus as a as a wafer scale you know a chip right a wafer scale chip but they are saying no like you know the next next unit of compute could be putting them into a rack so you think of this as a rack scale solution so even cpus could be could be filling into racks and that's nothing new it's been part of the data center uh cloud data center for for decades now like that's how cpus used to be filled into racks and they used to do to compute right so now you've got a rack of gpus you've got a rack of cerebrus gpus uh you've got a rack of you know p core like intel calls it or a rack of e core all of these have different capabilities like fast latency like this this genius's specialty the cerebrous genius specialty it's just speed it is only speed right this this thing this genius can't remember this kind of forgetful genius but it's very good and very fast at doing stuff then you've got the other kinds like the the large hbm based accelerators those geniuses uh have a little bit more context They're not like ultra fast.

35:46They're very smart. But, you know, they kind of have larger context, a little bit more general purpose genius, you know, not just genius. And then perhaps you could have the slow genius, the slow thinker. I'm just pushing this analogy because what if, you know, you really don't need that token speed in the genius? You don't need this fastness of the genius. You just want the genius to think for a long time and come back with whenever the genius has an answer. You know, this is definitely a workload for like science and, you know, medical problems of solving cancer or something. It's not like you want the answer tomorrow.

36:26We would all like it, but it would be much better if this genius could think for a very long time and come back with a nice answer that we could all work with, you know, without blowing the budget, because you can't say like, I will put cerebrus on solving cancer tomorrow, like that cerebrous kind of high token speed. Maybe it will. Maybe that's what it takes because it's a hard problem. But maybe sometimes you're like, you know, I just want to study weather patterns and the inferencing can go really slow. I don't mind. Weather is slow anyway. Yeah. Yeah. And when there's like overnight jobs that could be like, hey, go look at all my transactions from today and summarize them and write them in some log or something.

37:07You might want some intelligence where you can't write deterministic software. I mean, that use case, you probably could write deterministic software, but you might want to extract some insights first from all of those transactions and write that to the log as well. You would need some intelligence. Frankly, that could be an LLM that runs on a CPU too. If you've got like a cluster of P cores, which then it makes me ask the question about orchestration, which is like, do we have the right orchestration software to schedule across super fast geniuses and regular geniuses and even like cpus um or is there actually opportunity there for someone to make it simpler you've obviously got like nvidia dynamo and stuff like that but like even a layer higher yeah yeah the nvidia dynamo thing is all about how to make the genius work better right yeah yes but that's not the orchestration layer yeah so it's all Yes, so we do need a solution.

38:10I think modular is one such orchestration layer, if I get this right. But maybe more so for the genius still. I don't think it's going to do it over everything. I'm not entirely sure. Yeah, we should talk to the modular people. You can write with Mojo, and it can run on CPUs or GPUs, all with one programming language, which was just pretty sweet. But I don't know a ton of details yet about their orchestration level capabilities. But the one thing I understand is that they can talk to disaggregated hardware of different kinds. So you can mix and match various pieces of hardware for inference. We're still talking about the genius.

38:56But even that, you could have orchestration. Do a little bit of tokens here. Deal with a different architecture here. Like, can you mix AMD Rack and NVIDIA Rack all together in a data center and have a software platform that does all this inferencing? But then, yeah, ultimately, then you've got to have a software layer above all of that. Maybe it does exist and we are not entirely software guys. But, yeah, that platform will orchestrate how the CPUs and the GPUs interact with each other and how the data moves between all of them. You know, it's a very, very complicated problem. totally totally and i know gimlet labs who i talked to on this uh podcast i don't know back in may i believe um they were kind of working in this same space which is as a neo cloud could you ultimately have lower costs by having different hardware and being very good at scheduling it across the correct so that the correct slice of work gets done on the correct hardware um But, you know, all these are topics for another time.

40:01I think this is probably a good place to call it quits. So we talked about CPUs. We talked about agentic AI. Where should it run? Oh, let's circle all the way back to GrokBot. So where do you think your GrokBot VM lives? Yeah, it lives. Go ahead. Yeah, it lives in some general purpose CPU somewhere, I think, because this general purpose CPU is just running an OS somewhere. And it has a little computer that's made only for me. And I go in there and it's the only job of it is to do what an operating system does, right? It makes, it's like, hey, go access the memory. And here you go. Like, you know, I'll get data from the internet and all of this stuff.

40:50This is not entirely like agentic per se that, but it's just a computer. like it's a computer like the one you're on watching this on your phone or like your laptop or whatever it's just a computer and yeah so that computer accesses a whole lot of other hardware that mini virtual computer you have has access to the genius and it has access to all these host node cpus which the assistant that's feeding the gpu with you know that entire office building of stuff is accessible to my little virtual machine. And I can spawn as many agents as possible from my little VM and send them off to do various tasks.

41:35So it's a nice approach, I think. And it's going to cause more demand for CPUs. Now, because you're opening up not only the layer that was previously inaccessible to people, because not everybody could actually install open claw now if more people can use tools like grokbot i'm not again trying to say this is one particular product i think more will come out like this it opens up uh the ai world to a lot more people the agentic ai world which means that if more and more people start even regular people not like the high-tech ai using token maxing crowd if the regular old people like who never really wanted to use AI, but now find use in it, start using this.

42:21Remember how many VMs are going to be apportioned for each of these people who use this tool or tools like these. Then you've got those little VMs spawning off so many calls to CPUs and GPUs. Imagine each one can run 10, 50, 100 agents. There's a lot of hardware demand. What can I say? There's a lot of hardware demand. Absolutely. I think 10 million people doing this is not crazy. And that could be 10 million VMs running on 10 million cores in the cloud. And then if people are kicking up, let's just say whatever, 100 agents per VM, all of a sudden you're at a billion cores that are needed. it's not that crazy to imagine that world and that's coming quickly and maybe I guess to contrast that with the Mac Mini craze yes Mac Mini was sending a lot of new requests for tokens from GPUs and so that was also selling lots of host node CPUs but ultimately a lot of that agentic work or whatever is running on the Mac Mini but this, the easy button the fast path is you push that button and it spins up a VM in the cloud And to your point, I do think that's where really we're going to see a lot of proliferation of agentic AI to the normal crowd, to normies.

43:52And therefore, it's obviously going to be good for server CPUs. Agree. Totally. All right, let's wrap there. We'll check back in the future. Anyone who has any interesting thoughts or corrections or interesting orchestration stuff that we should be aware of, send us an email, put it in the comments, whatever. However, Elon and Grok team, if you're listening, we'd love you to come on and explain how it works for us. And I think one maybe last little point is the question is how much of this stuff is actually running on agentic CPUs versus VMs on general purpose machines. And because these are newly marketed things, you know, I would love to hear from Anthropic or OpenAI, if they would be willing to share, how much work are they actually putting on agentic CPUs versus just still regular general purpose CPUs and what are the pros and cons.

44:45But so much that we would love to learn. So anyone who's in this space that's listening, let us know. We'd love to talk to you. With that, we're going to wrap here. Thanks, everyone.