Training AI Models Without a Billion-Dollar Data Center | Steffen Cruz of Macrocosmos

0:00Steffen Cruz:The blockchain, in effect, is a registry of addresses, and it points toward the actual assets that are being registered. There's no training data on the chain. There's no compute on the chain. Today, we are training models. We're training multiple models all at once, using pockets of cheap energy, which translates into cheap compute. By the middle of this year, I would like us to reach 5 ,000 compute nodes. I think this is a respectable sized cluster that you can train a model that will get people's attention and make them understand that there's a lot of utility in this technology. I think that's an important milestone for us.

0:40So my name is Stefan Kruse. I am the co-founder and the CTO of Macrocosmos. I hold a PhD in subatomic physics in the University of British Columbia. So I was a physics researcher for the beginning of my career and then I pivoted to AI when it became apparent to me that there's a lot of opportunity for scientists that are just about to graduate from their postgraduate studies. And there's just this, there's an entirely new science sort of being born and developed in real time before us. And I think my decision is one that a lot of other people have made as well, where they want to go from being in what can feel like the sort of the old school machinery of traditional science in academia, where things move slow and your contributions can be very narrow and sparse to becoming a pioneer of this entirely new exciting thing which is not only abstract and theoretical but it's being applied in new and interesting ways constantly so when i saw that i i concluded my phd and i decided to leap leap leapt out of academia and into sort of the application of um ai and also physics in in various private sector domains, such as manufacturing and systems optimization.

1:53Following that, I discovered BitTensor, which is a very interesting AI blockchain project. And ostensibly the purpose of BitTensor is to allow AI to be developed in a way that is fully democratized and is sort of done in a global way with incentives. And I thought that was such a fascinating proposal that I decided to join the network. I've been within the BitTensor ecosystem for around three years now. And I have been relentlessly experimenting and building within BitTensor the kind of things that I think would make a lot of sense and would bring a lot of value to the world using the blockchain as a prerequisite tool to enable AI research to be done in various ways and scales that would be very difficult otherwise.

2:40Steffen Cruz:Yeah, and specifically, well, first let's talk about BitTensor. but before you began we were talking about how there are a few of these blockchain ecosystems for either training or or marketing AI models or AI applications and I'm familiar with Singularity net bit tensor is much larger is that right is it the largest of these or one of the largest and how many are there i believe so so bit tensor is actually over a hundred projects under a trench coat and i think that's also why it feels so encompassing and so large and this is what is very interesting about bit tensor is it doesn't seek to solve a specific narrow problem in the AI industry.

3:34It's not like trying to solve something like how do we provide people with access to models or how do we train models or how do we do one of a plethora of different things. It actually has done a lot of foundational work on how do we make it possible to build a base layer for people to come and solve just about anything they can imagine in a way that it uses the blockchain as a reward mechanism, as a coordination layer, but then BitTensor does a a very good job of stepping out of the way and letting people's creativity drive them through to build what they want on top of it. So BitTensor feels very big and very sort of like it's everywhere because there are, today there's 128 different teams that build on top of BitTensor and they're all building something different.

4:16A lot of them choose to work within AI, but there's also some that work outside of AI as well. And so for that reason, I think that BitTensor punches well above its weight.

4:25Steffen Cruz:Yeah. And for those who are not familiar with the blockchain, and even me, the blockchain, in effect, is a registry of addresses, right? And it points toward the actual assets that are being registered. I mean, this is the thing about blockchain art that was such a craze for a while, that you're not actually buying a piece of it. I remember Trump was selling these, I think, playing cards or something like that on the blockchain. And you're not actually buying anything. You're registering your ownership of something. Exactly. Yeah. But the actual asset is a digital asset that can be copied millions of times.

5:20Steffen Cruz:It's up to you then if you want to enforce your ownership to track people down and that sort of thing. So in the projects that I've seen related to AI, it's a registry that points to an AI project, or is there something more happening on the blockchain? So in a project like Bitensor, the blockchain is used in a number of different ways. But fundamentally, it's an immutable database that is globally shared. And that in itself enables a lot of things that Bitcoin pioneered some 15 years ago. What becomes possible when you have this sort of shared globally distributed state is you can create a store of value and then you can start building higher and higher order commodities on top of that.

6:15So Bitcoin sort of addressed the most principal version of this. But what was demonstrated as a result of that is you can actually aggregate a huge amount of compute or let's just call it effort. Once you have this interesting distributed sort of system architecture, people come to it and because there's economic incentives involved, people are constantly innovating and optimizing and they're trying to find a way to sort of percolate to the top and to become the most competitive people that are doing whatever this mining task involves. In the case of Bitcoin, the mining task is just guessing random numbers faster than anyone else can.

6:52That's why you just throw a more powerful computer at it and you become a more competitive miner on the Bitcoin network. And other crypto projects have tried to orient that effort, that sort of collective coordinated effort of all the participants have tried to orient that towards different tasks, whether it's distributed file storage or creating a global residential proxy network or solving data scraping from the web and all of these things are really interesting whereas what bit tensor uses the blockchain for fundamentally as you described the registry component is very important it's also sort of a shared synchronization clock the notion of blocks and a blockchain gives everyone sort of a drumbeat that you can actually anchor work to which is an integral part of a lot of our research on bit 10 so we have this sort of shared clock which the blockchain regulates but ultimately you can do as much or as little on chain as they call it or off chain as you want and and i think some of the benefits of doing things on chain is you have complete auditability you can see precisely what the details were that went into a specific piece of work or transaction so it gives you transparency the problem with that of course is if you've got billions and billions of transactions being written to a database, your database gets very big, very fast.

8:08And then it becomes kind of unyieldy for people to actually have a copy of, which can interfere with the true decentralization ethos of it, and you end up with just a few copies. So I think that there's a pragmatic middle ground where you store exactly what you need, but nothing more, to ensure the core parts of the work that you care about are captured and untransparent for everyone to audit, review and verify. Whereas the rest of it you can you can store elsewhere you can store offline you do a million other things with it so i hope that adds a little bit of light to the question craig yeah in your case with um

8:42Steffen Cruz:macrocosmos you're providing compute for training models is that right explain precisely what macrocosmos does absolutely so today we operate three subnets in bit 10 sub each subnet is you think of it as effectively a different project, a different service. And our three different subnets, I would consider them to be three different axes or three different use cases for the blockchain that I think are very potent and powerful. So one of them is just the aggregation of raw compute at scale. And if again, harking back to what I described about the Bitcoin network, I think one One of the most important things of our time is set to be AI itself.

9:28And those who train the models and own the models will have an enormous advantage. And so a big part of the purpose of BitTensor is to provide an alternative way for us to create these models, to train these models, to deploy these models such that they're not held in walled gardens and we're having to pay for the privilege of access to them. But we want to reinvent that in a way that's more democratic and also borderless. something that's censorship resistant, something that cannot be controlled. So our primary, our flagship project within BitTensor involves what's called distributed training of large language models.

10:05And that's quite a mouthful. Distributed training is quite different to how a lot of the frontier AI labs train models today. Typically, you have huge warehouses stacked from floor to ceiling with GPUs, which are specialized computers for training AI models and serving AI models. and you've got hundreds of thousands of them in the largest data centers in the world. And they're all plugged up together with extremely high data transfer speeds. And as a result, you can basically use the whole thing as a single computer. And this is fantastic and it's been demonstrated that you can just scale the amount of compute and the amount of data you throw at a model and they get predictably better and better and better.

10:45And this is called a scaling law. Now, there's an alternative to this, which is called distributed training. Distributed training doesn't rely on having a single warehouse stuffed with computers, but actually it's based on the premise that you can train an equivalent model using computers that are distributed around the world. And now this has a lot of very interesting outcomes. When you don't do everything in a single data center, one of the first things is that the capex of actually building this massive warehouse in the first place is massively alleviated. There's also the local energy that is required to run one of these massive data centers can also now be distributed around the world in a way that creates less impact on local communities, generally much less detrimental to the environment.

11:33And lastly, it allows you to do something that is actually fundamentally not available once you've built your data center and stuffed it with GPUs. use sort of the cost of training models is already baked in to that initial build process whereas in the case of distributed training which is what we care about we can actually perform what is effectively cost arbitrage so if there's if there's a surplus of energy in iceland uh very very cheap energy we can actually use that pocket of energy even if it's only for 12 hours of the day and we can target a lot of that compute and use it in a very elastic way and in fact that's what we do Today, we are training models, we're training multiple models all at once using pockets of cheap energy, which translates into cheap compute.

12:15So it has all of these different economies of scale, which are certainly unusual, but are becoming more and more accepted as a viable alternative. And as our appetite for bigger and bigger models is only going to increase as the years go on, we're going to find that we need to start looking at alternatives before we hit a hard ceiling on a lot of this. because things like the Stargate project and the Colossus project, these are multi-billion dollar GPU build-outs. And so we think that just like the fundamental physics experiments, like the Large Hadron Collider, at some point you require the budget of a nation state, it's 20 billion dollars to build a bigger ring to smash protons or electrons.

12:54You need to start thinking about different experiments because it just becomes unpalatable eventually. So we're trying to get ahead of this problem. I think that by 2028, we're actually, the world, the mainstream, the Overton window about training models is going to shift. And we are going to have to start thinking about how do we do this in a way that arbitrage is more efficient in both cost and energy. So we're trying to do a lot of the early work right now on that. And BitTensor is a wonderful place for us to do this research because we are able to use the blockchain to reward people that contribute their compute to our training experiment.

13:28Steffen Cruz:Let me ask a couple of questions. I had a guy on the program who was doing something interesting. He was, and I don't remember whether it was blockchain based, but he would aggregate spare compute from data centers or, you know, whether on premise or, you know, independent data centers around the world. and then offer that compute to AI projects at a cheaper price than the hyperscalers or the big clouds could offer. And so that's one question. Is it like that and you're just tying it to the blockchain to organize it and distribute it. The other is, you know, for a long time, people were talking about federated computing so that you wouldn't have to lose control of your data, but you could make it available for training models.

14:45Steffen Cruz:and again that that was using the blockchain to register the the data i guess uh so first of all is it related to either of those concepts and and then the the deeper question and the one even talking to the people i've spoken to i've never really gotten a hold of the chain again is just a registry. There's no training data on the chain. There's no compute on the chain. The compute resides, as you said, maybe in Iceland. What is the link between the chain and the compute in Iceland? So starting with your first questions, the relationship between those two examples you gave and what we're doing, I would say they're They're cousins, but they're not siblings.

15:43They're a little bit different. For example, federated learning is built on the idea of anonymizing client data, but being able to continuously train models. We're working with a slightly different version of this, which is not so much the edge devices providing training data today, but the edge devices are providing the training compute, I suppose, would be a way to think about that. And secondly, I think regarding what BitTensor actually does or what the role of the chain is, at its core, it's really not that revelationary at all. At its core, what the blockchain is effectively doing is it's providing a trustworthy, transparent record for everyone to audit, especially when there's code that's being run on the blockchain as well.

16:36people understand how this code is going to interact it can't be tampered with which means that it's predictable and that's a proxy for safety and trustworthiness so if someone had already deployed a smart contract for instance we're not actually building a smart contract in microcosmos but i guess just to illustrate the point if someone has written something in the smart contract then the creator of the smart contract can't later change the terms and conditions on you it the point is it's a it's a modular block of code that will run under these conditional expressions, that makes it something that you can effectively count on.

17:12So I believe I've given you a bit of an illustration of one of the projects that we're working on in BitTensor, which is distributed training. This project is called IOTA. And as I mentioned, I think that it's a precursor to some very powerful downstream technologies that we may find are an instrumental part of the way that we create AI in the next decade, let's say. But we really do believe that there's also a real change in the way that people are thinking about personal agents and personal compute. So I've seen personally in the last few months, a lot of people are starting to stockpile Mac minis.

17:50I'm not sure if you've seen this trend, but basically now that agents are starting to become kind of economically useful a little bit, we're seeing the beginnings of it, I think as a fair appraisal. What we're starting to see now is people want one of these things running for them privately around the clock day in and day out. So they're buying a dedicated computer. And on that computer, your personal agent is running and it's checking your emails. It's maybe doing some online shopping. Maybe it's doing some work for you on the side or some hobbies. And this is a wonderful use case. And this is also a really interesting place where our technology of IOTA developed by Microcosmos comes in because all of these computers that people have now built at home.

18:27Well, there's two ways to think about it. That's your personal agent. That is creating value for you personally. But wouldn't it be great if that compute could earn passive income? Almost like you own a property and you can Airbnb it out when you don't need it. And this is actually what we're offering as we develop IOTA is we've created something called Train at Home, which is people that have unused devices at home. Let's say MacBooks, Mac minis, or even consumer GPUs. They can plug those into our network. And that means that they become part of this global supercomputer. So we can use them for training models, which we can commercialize.

19:03And they can also basically create some passive income just from having that device sitting around that was otherwise underutilized. So I think there's something very parsimonious about that arrangement of things. And the more people that are choosing to buy personal devices to run these agents, you realistically don't need the agent running 24 hours a day. You need pockets of productivity. I want to make sure it, you know, plans my weekly agenda. I want to make sure it checks this, that, and the other. Maybe there's only four hours of the day that you actually need this thing. And you can actually get a return on your initial investment within 12 days, 30 days, by plugging it into a network like this and also being part of something that I believe has a great purpose, which is the democratization of AI itself.

19:43So it's a really interesting intersection of timelines right now with agents becoming demonstrably more useful and engagement increasing. And we basically can use all of that consumer compute. we can glue it together and we can train models at greater and greater scales and greater and

19:59Steffen Cruz:greater utility so yeah yeah yeah as a matter of fact i'm running the mac mini uh stockpiling i think started with the open claw and uh exactly i'm running open claw on this computer which i know is a bad idea and i have an unused mac mini upstairs so uh uh i'll probably go fix that after this call but uh but the uh so so how does somebody use this uh well first of all yeah back to the point of so you've got compute around the world that is plugged into the network

20:46Steffen Cruz:do people who are using uh macro cosmos do they need to know what's how the what the link to the blockchain is absolutely not yeah and and the blockchain is just keeping track of where the compute is and what's available at any point in time is that right yep and it's just responsible for uh rewarding participation so it's making sure that any contributions made by anyone's devices are fairly uh fairly rewarded that's really the way i think about this the blockchain is just a really convenient way of taking care of uh compensating people for their contributions The actual onboarding experience for people is we keep this as low cognitive load as possible.

21:40Right now, it's a one-click app store download. This thing will just run passively in your machine, adding some progressively more useful user experiences, things like telling it, oh, you can only use my computer when I go to sleep. So by default, it's disabled until 10 p.m., and it's only enabled from 10 p.m. till 6 a.m. So that would be a simple user control that will get it out of the way of when you're actually trying to be productive with your machine. That's assuming it's your primary machine. If it's a secondary machine that just sits at home, we're also creating a way for agents to decide, oh, I finished my work.

22:13I'm not going to need to do anything for four hours. So the agent just decides, hey, I'm just going to go and make 20 bucks. And that would be nice to come home from work too, right? It'd be nice to come home from work and you're like, hey, what have you done today, Claude? and it's like, oh, hey, I finished all my work by 9.15 a.m. and you were going to be out till 5 p.m. So I thought, well, I thought I'd just shop around and see how I could make you a little bit of side cash. And here you go. So I do think that as our computers become more autonomous, thinking about our relationship with computation, it's going to change fundamentally.

22:43And our computers are not going to sit there waiting. They're going to be very proactive and they're going to be very resourceful, especially when you've got these clever little agents that are running the show inside of the machine. And as a result of that, I think we need to really rethink what your computer can do for you, you know, our relationship to that world. And I think bringing this back to the top about BitTensor, you can think of BitTensor as just a bunch of places where you can use that machine and you can go and make some income from it. Whether it's providing something like human intelligence or whether it's training models or whether it's detecting if an image is real or fake, You can almost think of it as a mechanical Turk for agents and humans alike.

23:25Well, you can go there, you can monetize your skills and your experience or just your raw compute.

23:31Steffen Cruz:Yeah. And for the training, how does, let's say I have a large model that I want to fine tune. How do I do it with Microcosmos? So today our work is focused on the first stage of training models specifically, which is called pre-training. So pre-training has historically been by and large, the most computationally expensive part of creating a model. It's the part where they systematically inhale the entire internet, which usually takes months and tens of thousands of GPUs. Beyond that fine tuning is comparatively a much shorter computational task. So I think it's the essence of retraining requiring long time horizon workloads that is actually why we're so interested in moving this from a centralized to a decentralized context.

24:31And there's a lot of pioneering research that needs to be done that we hope will be very useful even outside of the Web3 world. we hope that this is really useful research to anyone out there that is interested in economically or capital efficient ways of training models we hope that this is a contribution to the field that's really how i see this because what we're doing at the root of our work is we're taking a network of very unreliable compute people are joining people are leaving it's just constantly churning over and churning over how do we create something that is persistent and stable

25:07Steffen Cruz:out of something that is fundamentally so noisy and unstable. And it's core. That's the problem we're trying to solve. And it just so happens that the use case for this persistent compute fabric we're creating is trading models, but there's nothing stopping that from doing something else. Perhaps there's a world where universities could use a similar decentralized network for their traditional HPC cluster jobs where they're doing academic research that's not necessarily AI related at all. It could be bioinformatics. It could be physics. The same thing still remains to be true, which is if you need 10 ,000 nodes of compute for 12 hours, what's the cheapest way you can get them?

25:51And we're trying to build software that basically orchestrates compute around the world to act like it was all plugged into itself and it's a supercomputer. And I think it's a fundamental thing that we're working on that we're applying towards training models, but I hope has a much broader value to the community. Right.

26:08Steffen Cruz:So on pre-training, how does somebody use it? Absolutely. So we're sort of emerging from research mode right now. We've spent about nine months on IOTA so far, and we've demonstrated that we're able to reproduce a lot of important baseline benchmark performance metrics using this, what we call it, the wonky vegetables, sort of the oakley vegetables that you get in the vegetable aisle that no one necessarily wants to buy. we're going to turn that into a premium soup. And we've basically graduated from research mode right now where we're showing that the two are indistinguishable if you treat them in the right way.

26:42If you use the right spices, it tastes just as good. What we're moving to next is thinking very deeply about what is the commercialization opportunity for a technology like this? Who are our target audiences? We think that the world is going to train more models in the next 10 years, not less. We think that researchers, small startups, especially cash-strapped startups, academia, all of these people are interested in training more models, but perhaps for budget reasons, it's very hard for them to do so. So we're sort of targeting the commercialization of IOTA as two things primarily. One is what we call supply side.

27:19So supply side would basically mean people that have surplus GPUs, so the neoclouds, the hyperscalers, perhaps they have 10 ,000 GPUs in their inventory, but they can only rent out 9 ,000 of them at a given time or 9 ,500 of them. Well, for us, them plugging in their compute into our network allows them to increase their overall utilization, which goes straight to that bottom line. So they'll be happy with that. And even better, if you actually to look at the usage of all of those GPUs that a hyperscaler has, you might find that, well, it's rented out for four hours and then it's not rented for two hours and then it's rented out for four hours.