Ep 96: Nebius CTO on Racing the Hyperscalers Before AWS Wakes Up

0:00If NVIDIA could have bought, could have produced like 10x the chips, this whole market would be shaped differently. Because there is not enough capacity for their scale, this is a relatively small business for them. For the hyperscalers, it's not just a matter of like sales. It's also a matter of changing the infrastructure that was built at scale for other kinds of workloads. This is a massive investment on their part that needs to be justified by the size of a potential business. And it's not that there yet.

0:33the emergence of nebius and the other hyperscaler competitors these past years has been a fascinating part of the ai story it's something i've really been interested in and so it was a lot of fun to have nebius's cto neela on the podcast because we were able to dig into a bunch of interesting things about the business and how the space has unfolded we talked about nebius's really interesting origin story and the original relationship with the index and how the company came to be we talked about what it's like to be inside one of these companies how Nebius works with NVIDIA, what it looks like to get their next generation of chips live.

1:01We talked about why the opportunity actually existed and why AWS, GCP, Azure didn't just capture this entire market. And we also talked about Xeniela's view on a bunch of new technologies like DISAG inference, among other things. All in all, just awesome to get to talk with someone who's at the epicenter of the AI infrastructure market about his thoughts on everything about where the space is headed. I think folks will really enjoy it without further ado. Here he is. I guess I'd love to talk just maybe about the broader like AI cloud market, you know, today. And I think that there's a few maybe larger and prominent public, you know, I guess termed neoclouds.

1:35And then, you know, it feels like there's a new one getting funded every day. I know you don't like that term, but maybe just talk a little bit about how you think about what differentiates, like some of our listeners are probably thinking like, you know, all that really matters is just access to GPUs and like, you know, wherever you can get them, you go. Maybe use that as a straw man and tell us why that's not the case. Well, I'm obviously biased because, first of all, I'm a software person and our product and business was launched on the idea that software is the mode, right? So your ability to serve the customers that consume, I don't know, tens of thousands of GPUs in one interconnected cluster with the same platform as you serve a self-service customer that rents one GPU for half an hour.

2:23was our value prop from the very beginning. So we believe that this is extremely important, being able to build large scale, but have the possibility almost like sell it in retail, in a sense. This is what we think definitely differentiates us from the market, from everyone on the market. Like a small remark, NeoClouds hate that term with a passion because it started as a distinction as in these are kind of cloud companies that are doing this weird, strange new thing, but then everyone wants it in that market. So when we started, there were like, I don't know, five, seven companies that were in this area that were called NeoClouds.

3:11Now, I think it's closer to like three to 500. Yeah. Depending on what you even are ready to call NeoCloud. the scale and we really don't like this stretched umbrella term anymore we are the ai cloud we don't care like let others other guys be neoclods we are the ai cloud and this is our mental model we are not competing with the neocloud markets we kind of believe we are past that our next competition point is taking on AWS and Google Cloud. Yes, we are way smaller. Yes, our revenue is like, what, I think 1 % of AWS nowadays. But we are running fast, we serve a very particular market, we have a bet that this is the market that is going to grow, that is going to explode.

4:09So people are concerned that these investments are huge, people put a lot into the infrastructure, et cetera, et cetera, et cetera. My view, if I compare what we see in the market with what was there two years ago, we are barely scratching the surface. I think the overall demand in the next three years will increase tenfold, hundredfold, maybe. So we are at the verge of another explosive growth. Right now, we see these huge facilities, right? But a lot of those are privately owned for internal needs of certain companies, right? So Elon Musk is talking about his Colossus cluster, and it's immense scale.

4:56Really, it's like real solid, large scale thing. But it's not for the broader market, right? It's for their internal needs, maybe for some partners. Maybe for Anthropic. Yeah. Anthropic and Opener, yeah, I'm missing the capacity, but they do it for their own product. And this is like a wholly separate discussion. These companies will essentially want to be everything. They will want to be the internet and contain everything and every product and every capacity angle within themselves. There is a question of Google, Meta, and Microsoft, who have a lot of capacity but are constantly juggling and struggling between giving it out to the market to rent or giving it out to internal teams.

5:47Because all of these companies are extremely machine learning, AI-enabled. They need a ton of capacity for internal usage, for runtime and for research. So they have to constantly juggle this. So our bet is that there is a niche for independent, as in not affiliated with a large company, with a huge internal consumption player that will serve the markets of the mid-tier players who want to be the large companies. I thought it would just be interesting for our listeners to talk a little bit about the history of Nebius. I feel like it's a really interesting path that you had to end up here. And so maybe just talk a little bit about how the business came to be.

6:27I'll start with the moment I've joined. So I am CTO of the company, but I've been with the company from the very beginning, since before it was called Nebius. So the new company is established, I think, during 2024 also. The deal was finalized. We finally announced our separation from Yandere. no connections to the Russian assets anymore, clean slate, etc., etc., etc. And that was it. You know, people have come to these AI cloud businesses in different ways. And obviously, you know, I feel like the Yandex team was like legendarily really, really strong, you know, on the engineering side. You know, a lot of folks building AI clouds came from like crypto or energy or like not necessarily the same, you know, maybe technical heritage.

7:16To what extent, I guess, was the Yandex experience really relevant? And to what extent was it like, you know, it turns out no one knows how to build a GPU cloud because it's all kind of new and, you know, everyone's learning on the fly. First, we had and still have the core team that supports the hardware side of business that came from Yandex that had experience operating hardware infrastructure at scale, like at real scale. hundreds of thousands, millions of servers span over multiple facilities, etc. These people were the core of our build-outs like initial build-outs and expansion and they're still with the company.

7:57So this definitely helped. I think we would be way less successful if not for that core team. On the software side, it's interesting because it's a two-fold. Because with hardware, it's very straightforward. We needed people with experience. we had people with experience they started building out from the get-go on the software side it's it was a weird one first of all we had to rewrite everything from scratch so our previous experience was kind of like garbled second we had to grow so fast that by for example by the end of 2024 when the new platform launched like a significant portion of our engineering team software engineering team was not from yandex anymore because we had to hire during this period extensively.

8:45As a team, we were working on this cloud platform from the initial idea, right? So we kind of just went with it and doubled down on software. We are not doing a managed bare metal. We are not doing a patchwork of third-party services. We want to build a holistic cloud experience, the holistic cloud platform. And we kind of doubled down on that even with the whole rewriting of the platform, like doing it from scratch, etc., etc., etc. And of course, having a ton of talented engineers on the team obviously helped. Just for the scale of it, right? You've mentioned people who came from crypto or came from other avenues of life.

9:29They are mostly concentrated on hardware. Their idea is like you need to buy GPUs, you need them to run, and then you'll find the buyer. in our case for example in 2024 when our hardware team that actually like procured gpus did the build outs uh we're working on our own servers and chassis writing firmware and stuff building the network that whole team was roughly like 40 to 50 people the software organization was already 200. Right? So we kind of can certainly think of ourselves as a software-first company. We are building the cloud on the solid foundation of hardware. Yeah. We're not a hardware-first company.

10:15Makes total sense. I guess I'd love to take our listeners like inside Nebius, and I feel like there's so many interesting parts of the business, but maybe one place to start would just be, I imagine as you're like getting, you know, new NVIDIA chips that come in, like there must be a bunch of stuff you have to do within the business to get this all up and running. So what does that look like from, I don't know, you're talking to the NVIDIA folks and they're giving you early insight into the next generation of chips to actually getting these live for a set of customers? That is a mess. I mean, in a good way.

10:47This whole market, this whole field is running at such a pace that everyone is trying to catch up. And NVIDIA, despite being in the lead of this essentially leading and showing the direction for this market is not the exemption. So first, NVIDIA folks like way before show you the prototypes of what they want to build in the next year, right? They kind of show you the schematics, like how it will be optimized internally, where is your PCI connections, how would that look like, what is the future platform would look like. Then sometime later, they come to you and said, oh, we've optimized things.

11:34It's now going to be different. You're like, all right. Then they come to you and like, oh, we have the first lab samples. Do we want to have a look? Yes, of course we do want to have a look. Oh, by the way, it doesn't look like the stuff you've showed us two months ago. Oh, we've changed stuff because we found... Because they are so compressed in time, same as everyone else. they are constantly iterating while these chips are being produced, right? So it's not like, you know, there is like an ocean in the consumer market. Like Apple iPhone that you see today was actually planned three years ago and final design more or less was settled like a year ago and they take the final year to polish it to build this product.

12:17Like AI market is running at such a pace like that whatever, they announce it, it's indicative. And then teams are constantly iterating. The architectures and the workflows are changing, right? So I'm sure they're iterating. Yes, they're iterating, going further and further and further. NVIDIA is a behemoth now, right? Whatever they do influences the whole market. Supply chains, everything. So they have to do that. So at some point you get a sample and it's more or less like what you'll get in production. Then you start building, like you are testing everything, you like incorporate support for the new hardware, for the new restrictions or capabilities into the software platform.

13:03You figure out a way to isolate workloads from the chat. We are a cloud, we are not providing bare metal hosts, right? So we need to split, we make sure that our virtualization works, that our storage works, that our networking works. Sometimes we are on the bleeding edge of this. So we like, for some of the generations, we were probably the first team, like maybe one of the first teams that actually built like a cluster of a certain size. So we've hit some issues with like with the software that no one has seen before. So we had to report that back to NVIDIA, like propose a patch, get there, like fix from it, make sure that it runs, et cetera, et cetera, et cetera.

13:48So like we kind of feel like we are at the forefront of it. But all in all, it's a pretty normalized process. It may sound messy, right? There are like multiple iterations. You go back and forth. You're in constant contact with like NVIDIA hardware folks, with NVIDIA software folks. You're trying to solve the issues on your end at the same time, incorporating into a more complex platform, thinking how can you expose that to customers because for example working with like your usual black hole chips like you know the B300s and then transferring that experience to GB300s which I call different architecture in terms of isolation you are talking racks rather than servers you need to find a way how to do maintenance on the rack instead of the server which you just switch off and you move the workloads to another one.

14:42Despite all of that, this kind of works all the time. And every day you push it a little bit further. And then in the end, yes, there is a product and the customers use it. And that's it. One thing I think would be so interesting to get your thoughts on is just the differences in what end customers end up wanting. Obviously, it can be easy to think, okay, everyone just wants to come to you and get similar things, but I imagine you segment your customers or think about differences in what is important to different customers. How do you think about the key differences in these customer desires today?

15:18Hi, I'm Anne Maris, and I'm the producer of Unsupervised Learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership and see you next episode. How do you think about the key differences in these customer desires today? It's not that different, to be honest, because every customer is its own unique flower, right? But at the same time, workloads in principle are not that different. It's either training, and you kind of know how training looks like, or inference, and you kind of know how inference looks like.

16:03Circling back a couple of years ago, for example, we kind of underestimated the storage component. We thought GPUs are the most important thing, and they are. But alongside the whole training angle, you see a massive demand for storage. And it's not for the amount of HDDs or VMUs that you can provide to the customer. It's about throughput. It's about latency. In my prior life, pre-AI workloads and infrastructure build-outs, I was talking about gigabytes per second if we were talking about network-attached storage, right? It was like, it's willing. Right now, we have customers routinely hitting a common share, like one volume, right?

16:46Connected to a number of GPU servers with like, you know, five, six, seven hundred gigabytes per second of throughput. this is extreme like two years ago I didn't even think about that and now this is more or less like something we expect at the same time if you are targeting these numbers for the highest demand like the most demanding customers your baseline is so high you kind of can't serve anyone right so if you hit this like use case where a customer needs 700 gigabytes per second from, I don't know, three petabytes of shared file storage connected to 1 ,000 of servers or like 8 ,000 GPUs, you kind of know how to serve a massive cohort.

17:37So I'd say in terms of performance or features per se, the customers are not that different. Then you meet another subset of what the customers really want because there are definitely different customers in terms of not their scale, not their size, not their workloads, but rather how mature they are. What are their obligations for their clients, right? Because we can talk about like a Neolabs. Like Neolabs routinely have seed rounds in the billions nowadays, right? So most of that budget goes into commute. So you can have like a contract for a billion dollars with the newly founded Neolab, they don't care that much about anything but GPU, storage, performance, availability.

18:32This is it. Oftentimes, they don't even care about the location of those GPUs. They don't care if that's US, Europe, France, or Iceland. Why bother? Internet connection is good, storage is there, GPUs are working, and we are talking about a billion dollar contract. At the same time, we now see and start onboarding enterprise customers. These are companies whose contract with us will be smaller, sometimes significantly smaller, but they are more established. They have customers, and they have applications for their customers. They have procedures. they need to make sure they're compliant to a number of regulations in multiple countries.

19:24So now we are getting into that whole angle of being the real cloud like certifications, compliance, security posture checks. Let us look at your source code. How do you solve problem X? How do you protect your customers from attack vector Y, etc., etc., etc. And I'd say if you are talking about differentiated customers, this is the most differentiating thing as opposed to what you would expect from the size or the performance. Obviously, I think a lot of people are thinking through this trade-off of a trade-off of latency and cost. And I imagine you have some customers that have workloads that are super latency sensitive and they'll do small batches or whatever is required to get to that latency.

20:13And then you have other customers that are much less latency sensitive, maybe background agents or something. Is that all just determined on the software side, I guess, if you have those different customers of basically how the batch size that you're running these workloads in? Or do you actually construct the clusters in any different ways based on those workloads? No, no, no, no, no. This was the premise from the very beginning. We are building the universal cloud platform and we need to make sure that our baseline is high enough to fit most of the workloads. Then there is the SLA play, right?

20:49So customers come, like for example, for the managed inference part of the business, customers come with the different demands. And then you are in this kind of a benchmark game, right? They want this particular footprint based on this particular traffic pattern fits this SLAs in terms of latency. or latency is time-to-frost token or the whole request latency. And you start optimizing for that. But this is never done on the hardware level because hardware is good enough. This is rarely done on the lower infrastructure level because it's universal and, again, good enough, no compromise for performance, et cetera, et cetera.

21:30And you're going further away to the orchestration level, to the model level. way further up the stack all the way to how exactly does the request from the customer, like the endpoint, gets to the inferencers and what are your performance hits along the way. Yeah, that makes sense. So this is like 100 % software. I guess a lot of people have talked, it feels like this DISAG, splitting pre-fill and decode is very in the air these days and a lot of people talking about that. Is that something you guys think about now or at least how do you think about that kind of in the future? I think we're in this stage where everyone's talking about that and no one's doing that properly.

22:11So because like it is a complex problem to solve. And particularly in inference, you have a lot of other problems, like way lower hanging fruits that you can fix address and hit those SLAs. So we are quite cautious about that. We don't have it, for example, as a go-to solution for every customer for every workload because like first it doesn't make sense second it's really hard to operate and work with like provides proper slas around that is like proving quite difficult as of today we have some promising results in our like prototype services but not as of today at the same time we have some like disaggregated setups for the most demanding customers in production right now it's just more you know wide glove manual approach yeah but especially for this newer architectures of like like you know those gray black walls whole node that has really fast interconnect a lot of challenges that you've had before with disaggregated steps are kind of solved if you're able to fit in the footprint of a rack because like the throughput is so much faster like interconnect is so much faster, you kind of stop caring and that this is a way of just balancing the perfil and decode nodes to match the particular pattern that you see from the customer.

23:41What's your mental model for like the workloads that do fit in a rack? I don't know. It's a very case by case. Well, these are definitely significant workloads. Like we are talking like thousands of requests per second, right? We are talking like billions of tokens that flow through. But at the same time, because we are the cloud, we can, for example, share the rack between two workloads and not be that bothered about that. So for example, on the cloud side, for this particular product, we only allow to consume at rack level. You cannot order two Grace Blackwell's nodes. It doesn't make sense.

24:22you have to allocate the whole rack and then structure workloads within it but for example when you're a client of such a solution as we do with managed inference for example you can collocate workloads and you can find a way to do that so in a sense you don't need to fit anything in the rack, it's just is there a workload that will benefit from this disbalance, intentional disbalance of pre-fill and decode nodes. And can you fit more than one into a single rack? And how do you solve the packing problem? Yeah, that makes sense. You kind of mentioned there's like lower hanging fruit on the inference side than doing that maybe.

25:04What are like some of the lower hanging fruits today that you still think we're not, as an industry, optimized on? Well, I think as an industry, we didn't optimize shit. So like, pardon my French, but in a sense, I strongly believe that running inference, particularly in the variety of open-weight, open-source-ish models that are available today, is not a solved problem. If you take something, just run with it as it's published in the initial support in some of the open-source inference servers and run with the novel models, your results will probably be underwhelming. It's not that fast. It's not that cheap, especially with like current prices on the infrastructure.

25:50It's easier to go and use like, I don't know, Fable 5.1, Astro 6, etc. But because of the open architecture, because like of everything that we know as of today, you can optimize the way you process those requests. You can optimize the way the inference server is configured. You put a lot of effort in kernel optimizations. sometimes kernel over-optimizations based on the particular pattern that you see in the traffic. There is a lot of underexplored potential in model weights optimization. And I'm not just talking about stuff like speculative decoding, for example. I'm talking about model pruning, selective pruning, compression.

26:40because I believe that, and we see that within our customer base, 95 % of our customers do not run vanilla open source models. These are modified, fine-tuned, et cetera, et cetera, et cetera. It means that these are the models that start to address a particular use case that the customer has. Particular use case usually means limited domain, scoped domain. Scoped domain usually means that you need to retain frontier intelligence and reasoning for a particular subset of tasks, and you don't care about the rest. And now we are getting somewhere in terms of maybe you don't need such a large model anymore.

27:27Maybe you need just like a model that is selectively compressed in a way where it works for a particular task and it doesn't work as good for the rest of it. Now, of course, that doesn't work with like open-ended, you know, deep research or broad coding tasks anymore. But I strongly believe that for real AI adoption, when we are talking about serious enterprise business, when we are talking about practical AI that actually solves problems, you are getting more and more into this specialized agents, limited domains, where you even stop caring about the costs. You care more about predictability of the outcome, and you care about time as in real seconds and minutes until you get that outcome.

28:21I think there is a lot of potential there, and we'll still see a couple of nice breakthroughs there. There's a ton of work around inference optimization, and a lot of that gets done in open source, right? Or contributions that share throughout the community. There's obviously a bunch of work on smaller purpose-built models and compaction and a bunch of these things. Across all these problems, which do impact how enterprises consume inference, which ones like Nebius should have a solution or offer a product around and which ones you're like, hey, it's great that that exists and we'll benefit from that, but we don't need to be at the forefront.

28:59We want to be at the forefront of everything. in a sense these are additive and even maybe multiplicative right like if you have a right subset of things optimizations that you introduce for a particular use case it means that you can reach the targets easier, better, faster in a more predictable fashion so we invest in all avenues like one of our acquisitions back in like at the beginning of summer was targeting specifically that. We basically bought the company for the talent in model optimization, model compression, speculative decoding, et cetera, et cetera, combined it with existing talent.

29:42So we have a team that is 2x the size. And we have serious manpower, as in tens and tens of people who are working on these problems. We have specialized groups for kernel optimization, specialized groups for model compression problems. because this is, I feel, if you want to talk about running inference as a service at a serious level, this is the only way you can do it nowadays. And I guess maybe along those lines, you guys have done a few acquisitions, and one of them that caught a few people's eyes was Tavoli in the web search space. Maybe talk through a little bit the rationale for that, how you think about that fitting into the broader platform.

30:26So first, we were very impressed by the product and their go-to-market motion. It was almost a year ago now, right? But we felt like we knew how to help them run even faster with our expertise in technology and low-level stuff. Because actually running a web search engine is not a very simple task, neither from the software development perspective nor from the... A lot of you used to run one. And some of us dabbled a little bit in our previous lives, yes. Right? So this was the premise. Like, we have a great team that is running real fast in terms of, like, market penetration and, in general, go-to-market motion.

31:15And we know how to make it complementary with our expertise on the low-level stuff and running large-scale infrastructure and, for example, large-scale web index. It's not exactly embedded into the rest of the offerings that we have now today. We are exploring and we have something cooking for the managed inference combined with agentic tools. Because, for example, we also develop sandboxes nowadays. So this is another building blocks for one of the building blocks that people use to build their agents. You have several other products that are in the earliest stages of development in that regard, too.

31:56Yeah, I mean, it seems that there's obviously a set of tools that folks are going to use to build agents and build on open source. And it seems like you want to have a bunch of those tools to be able to offer to your end customers. And web search kind of being a key primitive for any long-running agent will consume a decent amount of tokens on just pulling information in. Yeah. I guess I'd love to switch gears to just like the broader industry build out. and I think there's lots of questions folks are asking about that. I figured one place to start that actually might just be interesting for our listeners is to what extent are the data centers that you work with similar?

32:29There's a reference design and they all kind of, I don't know, to you it doesn't really matter which one's which. What are some of the differences in them that actually impact the business of running an AI cloud on top of them? To be frank with you, I have no idea what the walls of the facility are made of. So they're probably different, but that doesn't really help. But in all seriousness, what's within the walls is something that we fully control. We do not rent data centers when someone has built the network on their own or brought some GPUs or servers, etc. If we don't own the facility and we rent it, for example, we rent the building, electricity, but that's it.

33:22All the internal build-out is controlled by us. It's done according to our standards. So from the software side of things, there is literally no difference which data center we're running. Is it our own facility? Is it something that is built within a rented facility? Or, for example, we've recently announced the so-called asset light model when a third party brings capital in order to set up the facility and GPUs. And we run our platform on top of that. It's going to be the same. These data centers are built according to our, like, fitted out according to our spec. Hardware is something that is familiar to us because, like, unknown hardware is quite hard to adopt and make sure that you provide proper SLAs.

34:14And that's it. So we don't compromise there. We never rent something from people and present it as our own cloud. If you launch a virtual machine in Nebius Cloud, you can be sure that it runs on our stack top to bottom. Yeah, makes sense. I mean, I guess there's a lot of dialogue around the bottlenecks to the AI buildup today, and I think people talk about a bunch of different ones, whether it's power or the labor to actually make data centers, cooling, networking, HBM, GPUs, I guess, as overall. How do you think about what the key bottlenecks are today? or if you were to kind of rank across all of them, the few that feel most poignant right now?

34:55I'm sorry to disappoint. There is one single bottleneck, and that's your ability to execute. Because, yes, all of those are issues. Like there are GPU shortages in the market, components shortage in the market. Nowadays, it's CPU shortage in the market, like NVMe shortage in the market. There are not enough facilities, not enough electricity, Not enough qualified workforce to help you build it out. Not enough small components like network transceivers, for example. What else is not in abundance? Oh, you know, like those large scale. I'm not sure I know the proper English terminology, but like you know those machines that actually convert electricity in place like transformators transformers yeah like do you know the lead times on those these days oh years like so like everything is a problem and i think the single bottleneck and at the same time the single like mode that you can impose on that is like your ability to execute around all these problems and concerns, your ability to source, your ability to cut corners where appropriate, your ability to execute in general, your ability and flexibility to move stuff between facilities.

36:22Okay, we're going to move this one from here to there, and then we're going to move it back to the kind of land. Yes, we'll postpone this one, but this means we'll be in time with this one. This is important, et cetera, et cetera, et cetera. So I think that in this particular field, like an experienced project manager is probably in highest demand I've ever seen throughout my career. Yeah, I guess the bottleneck is really good project managers. Yeah, exactly. Your overall ability to have this large context of problems everywhere in mind and navigate that becomes a mode. Yeah. It's interesting because you obviously have the hardware itself changing.

37:08You have a bunch of workloads changing. Obviously, we have much more long-running agents. We have more post-training going on, people running open-source models. How does that actually impact the way that the overall build-out of the clusters themselves? To what extent is it like, hey, a workload's a workload and we can run it? No, the workload is a workload. Exactly like you said. It is slightly different, but in a sense, you need a high enough baseline to not be constrained by that. And then the rest is just your optimization problem, the way you pack things, the way you split things. You don't need whole nodes for some of the workloads, so you need to collocate workloads.

37:55You need to have the right primitives. You need to expose those through the platform, et cetera, et cetera, et cetera. But in a sense, differentiated workloads almost never impose any changes on how you build on the lowest level. Yes, of course, nowadays with these new architectures, like what is it called? NVIDIA LPX, like the outcome of the Grok deal, right? This, like, adds to it. But in a sense, if you have experience running other generations, this is just another, like, 0.5 generation, right? Yes, this will be a little bit novel, and it's connected in a weird, compared to the previous one's way.

38:52And there are some restrictions when you can run and not run on that. But there are efficiency gains. And this is just a niche local addition to the platform, to what you offer to the broader market. But in general, no. There is way less custom solutions that you would expect from that. The workload is a workload. Why didn't the hyperscalers move into this market faster or win this market more convincingly early on in this first chapter, at least? First of all, sometimes there is an ocean that they didn't even try. There is a solid amount of business that hyperscalers get from out of this market.

39:39Huge deals. They are very competitive. Sometimes people say, oh, have you seen AWS prices? Well, these are public prices. I assure you, for proper deals, they are doing proper prices and they're very competitive. They're very aware of the market. I think, in a sense, there are two angles to that. First, what I've mentioned before, a lot of these companies need to juggle their internal consumption and what they can put for the broad market.

40:14We are in cheap shortage world. right? So if NVIDIA could have bought, could have produced like 10x the chips probably this whole market would have been shaped differently because they were like Microsoft and Googles of the world would both buy capacity for themselves and for the broader market to operate. This was part, like simply not enough capacity for their scale right? Because there is not enough capacity for their scale this is a relatively small business for them. AWS revenue is what, like 100 billion or something? I mean, a business for like two, three, four billion, it's not insignificant by any margin.

41:00But compared to what their conservative, large-scale established businesses, it's not that easy either. But your point is basically like they get a chip allocation that is not as large as they'd want. They have to use a large amount of that for training themselves. There's only so big that the cloud business can be, given all that. And it's interesting to figure out how to prioritize. Do you go after the few folks with large contracts, or do you try and get this mid-market enterprise up-and-coming segment? But I guess you're doomed either way if you only have a certain number of chips. Yes, exactly.

41:39And then at the same time, this becomes a relatively small part of your revenue portfolio, and you're like, do I even need to allocate resources to that? And that is especially a pressing question, because for the hyperscalers, it's not just a matter of sales. It's also a matter of changing the infrastructure that was built at scale for other kinds of workloads. the networking, the throughput performance, the storage performance, by the way, that I've mentioned earlier. So this requires, if you want to have it universally implemented through the whole stack, this is a massive investment on their part that needs to be justified by the size of a potential business.

42:31And it's not that there yet. I think if we start getting out of this cheap shortage in, I don't know, two, three, five years, maybe, we will definitely see this shift. And they will pay more attention because this will become increasingly large chunk of business. and they will start investing. They will reorganize and redesign their internals to be more accommodating to these demands, et cetera, et cetera, et cetera. And then with their whole sales power, they'll go and try to win this market. And the question is whether the novel players, like us, for example, will be fast enough to grow and represent significant competition when they get there.

43:21I'm 100 % sure that they will pay more attention in the future. Our bet is that we'll be fast enough to be in the league with them and fight as equals, rather than a small player trying to run in their trails. Makes sense. I mean, what do you think happens to, like, you mentioned there's, like, these, like, you know, 300 to 500, like, you know, neoclouds, everyone trying to get into this market. Like, as you kind of, you know, you mentioned, you're kind of thinking the chip shortage comes to an end at some point, like, as you zoom five years, 10 years in the future, like, how many of these businesses are still around?

43:56Or like, what is the end state of this market? Listen, I think the whole AI consumption is going to grow, and it will be universally used everywhere. AI is everywhere. Right now, web services are everywhere. AI services will be everywhere. So in a sense, you have AWS, right? But you also have Hetzner. You also have DigitalOcean. You also have Linode. You also have like tons and tons and tons of local like hosting providers that are like giving you virtual machines, sometimes bare metal, et cetera, et cetera, et cetera. So this market is going to be vast. Will all of the current like three to five hundred companies survive the next five years?

44:47I'm not sure. But I am pretty sure that there will be a place for the very long tail of providers down the line. like some of this the current players will consolidate some of them will die off new new players will be launched etc etc etc it's a like it's not a bad market right now it's not a bad business to be in so it will constantly attract capital and attention and investments etc etc etc and i trust that like in five years time there's still going to be like hundreds of players in this market who found the investment to like and several hundred million dollars and they can offer something to someone in the market.

45:31Same as it is with like with the small scale clouds like DigitalOcean these days, right? DigitalOcean did not die because AWS is growing. You're obviously doing a lot of things in-house like sandboxes and web search and, you know, helping people with their models over time. And, you know, I think one question a lot of people are asking is, you know, what parts of the AI infrastructure stack end up just being subsumed into these future AI hyperscalers and like live, you know, in just one combined offering? And then what are like the standalone pieces of AI infrastructure that will persist as companies?

46:06In a sense, I think it will like, it will kind of mimic what's happened with like the legacy generation of clouds, previous generation of clouds. Right. When your growth kind of dictates that you need to start addressing new and like new sectors, new markets, etc. Because otherwise your growth is like bounded and you can't like can't show your projections, can't justify your valuation, etc., etc., etc. so there will be a natural push for the larger companies to start like increasing their portfolio like trying to get into new products into new services etc etc etc it doesn't mean that there will be a one-size-fits-all solution because in a sense if you're like if you're concerned by that you will like we just need to stop doing whatever we are doing and just wait for for OpenAI to rule the world, Entropic to rule the world.

47:11Because these companies will eventually, you do realize that OpenAI has models, runs inference, they have their own search, they don't exactly expose it, but if you're building the agent on their platform, it has access to search. They have all kinds of sandboxing, local agents, cloud agents, et cetera, et cetera, et cetera. they both OpenAI and Entropic are now investing in forward deployed organizations and are gonna go and do the work that was traditionally like I don't know Deloitte's or like Accenture's piece of pie right so they will be everywhere and I think this is a natural motion for the large company to start getting into a broader markets but it doesn't mean this is they are going to be the only ones.

48:05So even nowadays, there is a place for independent object storage players all of a sudden, right? Or independent compute players or independent small scale inference players for that matter. Independent database players, right? Like one runs like, I've seen like Google has like BigQuery and whatever, all kinds of data stuff. But then Snowflake kind of like appears from out of nowhere. AWS has like all kinds of like managed databases, large scale databases, like Aurora, Redshift, et cetera, et cetera, et cetera. But then MongoDB appears and it's pretty successful. I can't say it's like super large scale and probably by nowadays standards will never be because nowadays standards for large companies like trillion plus, right?

48:58But like two years ago, it was a very solid and very successful company. Or someone like Clickhouse appears with their cloud offering, right? Also, a massive managed database, an independent company that finds its niche and works it real well. So I think, like, larger company, larger clouds will grow. They will have, like, products in every sector and will be pretty successful at selling those because of bundling, because of their sales power, et cetera, et cetera, et cetera. But there will always be a place for niche players. And because the market is huge, niche doesn't mean small. Niche doesn't mean small scale or low revenue or low valuation.

49:42This can be solid companies. Yeah. I mean, it's interesting you mentioned like, you know, OpenAI Anthropic having this kind of broad suite of offerings too. I mean, in many ways, they're obviously building their own AI clouds. And I thought one thing that was interesting recently was, you know, obviously a limitation of those AI clouds versus others is the, you know, having just one model family, right? To be able to call from. But I thought it was really interesting that OpenAI announced that partnership with Base 10. I'm curious what your take was on that. I mean, is that them just trying to get into this general, you know, hyperscalar game across models?

50:09In a sense, we see a lot of push and concerns about people claiming that open AI models or entropic models are too expensive. Everyone is puzzled or surprised by the bills they get at the end of the month. So this might be an answer to that. Well, if you're like, go try open models. At the same time, open models are still behind the closed models, right? However good those models are, they're still lagging behind. And giving people an option to try lower scale with the ability to switch back to the top frontier model might also be a good incentive. Anyway, I think OpenAI is doing a very interesting move with that partnership, and I like it really much because this kind of shows flexibility.

51:09They're not set and set. like they're exploring different options different things that they can or cannot do like looking for different options too like so but i i think in a sense this all plays into one single thing like open ai wants same as in tropical they want to be one single like touch point like they want to be one single thing for everything. Even that is like, okay, but sometimes I want to use open source models within my codex controlled environment, by the way. Right? So like, well, good for them. But if the company, the person, the user is still within your ecosystem, you still have a lot of options to steer them, to upsell them, to introduce them to new services, et cetera, et cetera, et cetera.

52:02So if that alleviates one of the concerns, so be it. Why not? Well, I always like to end these interviews with some overly broad questions that I stuff into the very end. And so if you'll humor me, I guess the first question is just, what's one thing that you've changed your mind on in AI in the last year? In the last year? A lot has happened in the last year. No, the obvious answer would be software engineering. like a lot of people forget but a year ago we lived in a totally different world yeah it's like dog years in the ai world way less capable yes way less capable in terms of the model capabilities and in terms of how good the modern harnesses and tools are it's like uh the ai coding because like a year ago it was total bullshit right now it's it's a solid part of your everyday tooling.

52:59But for example, if you're not talking about the industry trends or talking about more like my personal thing, I've started relying more on the AI-enabled local agents and harnesses to afford a part of my cognitive load, to be honest. I mean, obviously, you run a really large technical organization. And I think everyone's trying to figure out, I mean, obviously, the models only recently got really good on the coding side. And so everyone's trying to figure out what it means to build teams given the new model capabilities and run an organization. I'm curious where you've settled and how you thought about that.

53:39Cautiously. Conservatively. I'm very concerned that modern models and the attractive user experience that they provide, like model agents, modern models, etc etc etc is something that will lead us to two issues you like you see slop like being revealed in like in chats in merge requests all over and i like i try to pace that and i try to like be very observant and vigilant about that so for example like we don't restrict talking usage like people can use whatever they want our budget is quite significant we don't impose like per person limitations or per department limitations but the at the same point we try to like look at the usage uh figure out what it is used for look at the most ineffective ways people use it.

54:49We constructed a set of policies that doubles down on the idea that whatever you've done with the agent, it's still your result. It's your responsibility. It's your pull request. It's your code. It's you who is on call. It's you who's going to wake up in the middle of the night if something goes awry, et cetera, et cetera, et cetera. And if something was created by a completely autonomous agent, it is marked like that. Completely autonomous agents will not touch like the core services for a while. Because like, this is like, this is like out of bounds. Like, sometimes people are surprised that like, I work in a company that's kind of participates in this whole AI revolution.

55:39And I'm very conservative. But at the same time, like, I've seen how this sausage is made. And I like, I'll be cautious. What's the biggest mistake you feel like you've made in building Nebius so far? Obviously, a lot's gone right, but I'm curious, you know, something that in retrospect, you wish you guys had done differently. It might sound funny these days, but we underestimated the scale. Like, and it's not about the growth of the company. We were pretty optimistic from the very beginning. But I personally, that was like my strong bias for the first, like, significant part. I underestimated the scale that a single client operates at nowadays.

56:20Because like five years ago, if you're talking about a 30 megawatt data center, it was a serious facility. It hosted like hundreds of thousands of servers, like run gloriously large for close. By now, 30 megawatts sometimes is not enough for a single client, which I'm still getting used to. The storage I've mentioned, four years ago, when you were talking about network-attached storage, you were talking about hundreds of megabytes, maybe gigabytes per second of throughput. Right now, some of our clients routinely fetch, I don't know, seven hundreds of gigabytes per second from a single shared file system.

57:10I'm still... I underestimated it in the beginning, and I'm still getting used to this scale. Like, I'm an infrastructure person. I've worked and I've seen these numbers throughout my career multiple times. And all of a sudden, you add a couple of zeros to the end and call it a new norm. Yeah. Still puzzling. It's crazy. And I guess lastly, there's obviously a lot of unsettled questions in the AI field today that lots of people are trying to figure out where the world goes. I'm curious what you're paying attention to. And I guess with a lens toward if it goes one way or another, it would really change the strategy for Nebius as a whole.

57:52What are you most paying attention to on these unsettled questions of where the AI world's going? Well, I stopped caring about the new models being larger or smarter. I just accepted that it's a given.

58:09I'm kind of trying to look out for like new novel architectures and approaches but I don't see any as of today I hope somewhere deep down in like a stealth lab something new is cooking but for now I frankly don't see anything that like is a thing that I would be even like would place even a small bet on. But I'm more and more interested in the whole outcome optimization part on the agent side, right? Because like we are used to talking about models and size of models and families of models, et cetera, et cetera, et cetera. But now that we've shifted to this multi-turn, long running agents that we are talking not about requests, but rather outcomes, companies are talking about bidding for per outcome right i'm very interested in how that part will shape up and we have like a small research group internally with the neighbors that's working on this problem actually i was like how do you increase the chances of getting to that outcome not by increasing the model or like changing the model that is of course part of the discussion but i believe there is a lot of optimization in the part where you're introduce more sophisticated or maybe more simplified tools.

59:31You introduce the harnesses that start to evolve according to a particular task. And this kind of gets us back to significant CPU consumption, because a lot of modern AI workloads actually go back to becoming code. It's codified. It's very formal. you see like quality gates, you see sophisticated tools, et cetera, et cetera, et cetera. And sometimes your agent works for minutes and minutes without even issuing a backend request to an LLM because the workloads start to like to sit on this part. And this is the part that really interests me because I think like a lot of future AI adoption lies in how reliable your agentic scenarios are and how reliably you reach those outcomes.

1:00:22And I think the whole optimization loop outside the model is an important part of that. 100%. Well, this has been an awesome conversation. Really appreciate you coming on the show. And yeah, thanks so much for coming on. Thank you for having me. I'm Jacob Efron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint.

1:00:53But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.