Who Feeds the GPUs? Inside AI's Hidden $30B Layer | Renen Hallak, VAST Data

31:57It's our job to make sure that they're happy. The one chart that I look at every morning is our customer happiness chart. That's the only thing we care about. What does that look like? How do you measure customer happiness? It has three colors. It has green, it has yellow, it has red. And anybody in the company is allowed to move a customer from green to yellow or from yellow to red, but very few people are allowed to move them back. And so... But based on what metrics? The only metric that really matters in order for us to move a customer back is that they say so. That they say we're happy now, everything is good, everything is solved.

32:35And yeah, that's what we work so hard every day to accomplish. I think the numbers show that it's working. the numbers that people like you usually care about are net dollar retention gross dollar retention we don't have any churn in the company our gross dollar retention it's not a hundred percent because a few of our customers have gone out of business over the years but no one's actively decided i'm going to stop using vast which is something i'm very proud of and on On average, our customers double and triple the amount of data that they place on our systems every year. Let's get into some of the technicality of how the data interacts with models.

33:23What does it actually mean to move data to the GPU? And is it different for training compared to inference? Training is super simple. training you just need a lot of gpus and you need to feed them with a lot of training data once in a while they put down a checkpoint so that if something crashes they don't have to go all the way back to the beginning but other than that there's not much to it you need large scale you need fast access that's it uh inference is becoming a whole complex world of its own Inference started by obviously being requiring low latency access because people are on the other end prompting and they want fast responses.

34:11That meant that you need a distributed system close to where those people are at the edge. That also means that you need a system that's always up. Somebody once told me, one of the big model builders, that if training is down, nobody notices. If inference is down, there is no service. And so it needs to be up 100 % of the time. Resilience, distributed nature, low latency, very secure environments. All of that is important in inference versus training. But that's just the ground level. Once we start talking about multiple models, once we start talking about agents, once we start talking about reinforced learning, definitely once we expand outside of the world of data centers into physical AI and actual devices, each one of those adds another layer that means that we need to develop a new set of tools.

35:07For example, now that we have multiple models, each model is good at different things. And so I want to make sure that this prompt makes its way to that model because it's good at software development. I want to make sure that this prompt makes its way to that model because it's inexpensive and this isn't a high value task. I want to make sure that no sensitive information is exposed to this model because they're going to use that information for things that I don't want them to. Whereas here I have a contract that protects me. And so just that aspect of it is starting to get complex. once we start thinking in that direction there's all kinds of optimizations that we need to do around things like heavy caching and saving context windows and making sure that we route the prompt not just to the right model but to the right GPU that has that model loaded and that has my context in cache context is just the last few minutes of conversation we need short-term memory the last few weeks or long-term memory what happened two years ago to also be accessible through things like RAG into these environments.

36:18And so all of that requires us to manage data, the raw data, the generated data, the input data, the weights of the model, the context. And we need to make sure that nobody sees information that they're not allowed to. We need to keep track as we do reinforce learning of what information went into fine-tuning each model and make sure that that information isn't leaked through interacting with somebody who's using that model, whether it's a human or an agent, it becomes very, very interesting as you layer one, two, three, four of these things one on top of the other. So if you have enterprise data to play it back, that data could be used by the model through the context window, through the KV cache, which is the model's memory, or through RAG.

37:11Who makes those decisions? Is your job as the fast data provider to just be available and then the model pulls as needed? How does that work? So we need policies to be set. And today we just connect into your organization's authentication and authorization mechanisms. If you use Active Directory, if you use access control lists, we then know who is allowed to see what from a human perspective. We then extend that to agents as those inherent properties from the people that deploy them or from other agents that spawn them. And based on that, we know what's allowed. What are the rules of the game?

37:51Then we need to enforce those rules, which becomes very difficult when you're talking about multiple organizations. most of these model builders won't let you infer within your environment which means that you just need to send your data to them and hope for the best that i think has been slowing down the adoption of ai in a significant way and so we need to make sure that it's secure we need to make sure that it's safe um we need to make sure that there aren't any obstacles uh for ai to achieve its true potential. And to what you just said, the concept of agent memory works the same way. So accessing that vast data storage.

38:38Did you say you effectively give the agent authentication authorization the way a human would? Very similar, yes. So access control lists that apply to a user ID. Now this user ID is assigned to an agent rather than a person. We have access controls based on roles. We have access controls based on attributes. And so it's very fine granularity. And you can, in a dynamic way, set your policies. It's accessing data, but also what tool calls they can or cannot make. Correct. And who they're allowed to talk to, because now you have multiple agents talking to multiple people. And so that conversation happens over a persistent streaming service that we provide That allows us then observability into it because we can query through the database and ask questions about those conversations.

39:30And so the entirety of this, remember, it's not in one data center anymore. It's across geographies. And so the entirety of this new world gets managed by this operating system from a observability and from a control and from a policy setting perspective. especially when it starts to expand outside the data center into cars and humanoid robots and satellites then the stakes get higher because physical elements can create physical damage and so rather than just data safety we need to think of things like actual safety that robot should not pick up that tool or that car should never accidentally hit a person.

40:18And so you need us to be in the data path to not just be able to report on these things, but also to intervene and enforce the policy. And in the meantime, the concept of agent collaboration, which is all the rage now, you know, highlighted in part through the recent security and safety issues, but the the concept of agent swarm is everywhere. So that collaboration layer would be also enabled the same way they would have access to a group, like the way humans would have access to a group access to data. That's exactly right. And I think for a long time, we've been hoping that interaction between AIs, interaction between agents, will give us that next step function in a level of intelligence in the same way that when we talk, we make each other smarter and we give each other ideas Because I think this solving of the Navier-Stokes problem earlier this week proves that that is actually happening.

41:21They set 10 ,000 agents loose at an age-old math problem, and within a few days, they came back with the result. And so that's one aspect that was missing, I think, which is good for theoretical problems. The other aspect that's missing still is access to the natural world, which is good for physical problems. And so hopefully that brings us to a level where the AI can help us solve some of the big problems that we've been struggling with over the years. All right. Amazing. We said we would go back to that private and sensitive data and exposing it to commercial models. What is that called, by the way?

42:03It's called Data Enclave? Enclave, yes. Enclave. Okay. I'm not sure I like the name, but that's what we chose. So how does that work practically in simple terms, but practically? It's super simple. In fact, it all goes back to encrypted memory. And so if today model builders want to keep their weights very close because that's their IP and they're afraid of giving it to a million customers and having one of them take it and do something that they're not supposed to. On the other hand, a specific class of customers, specifically enterprises, cannot just give their data to the model builders in the way that they're expected to today.

42:49What we're doing is we're letting that inference process, rather than running within the premises of the model builder or within one of their clouds, we're letting it run within the premises of the enterprise. whether it's a physical enterprise, a premises or a VPC, a virtual private cloud that they own. And the way that we make sure the model builders are happy is by encrypting those weights and we encrypt them end to end. We leverage hardware abilities underneath us from companies like NVIDIA, who is a big part of this launch, who allow us to bring encrypted data through the network, into memory, from memory, up to the GPU.

43:31And only the model builder's inference application is allowed to access it. And so we have full custody, chain of control, end to end, such that those waits are safe and the enterprise knows that nobody can ever see their information. And why is it possible now that the concept of confidential computing, I'm trying to remember the name, there are two or three names for that concept. It's been around for like, I don't know, a decade plus. Why is it happening now? I don't think there's anything new in the underlying technology. It had to be plumbed because this was, as you say, available for a long time for CPUs.

44:16Now it's available for GPUs. And so that piece of it needed to be done underneath us. Our layer requires us to manage all of this and make sure we understand which model is which, who does it belong to, who is allowed to see it, where should we block it, those types of things. I don't think there's a technological breakthrough that was required for this. I think it's just an idea whose time has come. And it also requires many companies, many parts of the ecosystem to work together in order to provide this end-to-end solution. And so we're working with the AI clouds to do it. We're working with the OEMs to build physical boxes that can sit in the enterprise's data center.

45:02We're working with the model builders. There is an appliance concept. Exactly. Yeah. Companies like Cisco or Supermicro are now collaborating with us on this. And of course, the chip makers themselves like NVIDIA. And beyond the technology, there has to be a trust and auditability layer, presumably, right? Because now you become the bottleneck, like you're in the middle of like this great confluence of just different players from across the ecosystem. So how do you think about that? I think that's where the trust should be. That's the layer that is, on the one hand, malleable because it's software.

45:43On the other hand, it's low enough in the stack such that it can benefit all of the different applications. historically that's where safety was done in the operating system and so we needed to be done in our layer I hope that we are trustworthy we're working very hard to build that trust with everybody else in the ecosystem and yes we're very very humbled to be in the middle and in that position. That's like another huge milestone for the company. So amazing. I'd love to take a step back and just like think through the macro environment from your perspective. You're in this very privileged position as the middle layer of the cake, as we said a couple of times now.

46:37I'm curious what your take is on a bunch of things. So the inevitable question is around the demand side of the whole compute boom. So you work with a lot of the NeoCloud and NVIDIA. So you're intimately familiar with the supply side. You're part of the supply side yourself. what gives you comfort that the demand side is going to materialize in a way that is not going to get this entire AI ecosystem in trouble? I think the demand is there today. I'm not sure it gives me comfort and sometimes it scares me as to how much demand is there and the acceleration in demand because every quarter, our customers are coming back to us and telling us they need a lot more than they thought they did.

47:33We had a customer, one of these AI clouds, one of the smaller ones, come to us a quarter ago and we told them we need to plan ahead for the next three years because there's supply chain implications and you need to be aware of this. And they said, we're probably going to need about 500 petabytes over the next three years. And last week they came back to us and said, we're going to need an extra two exabytes on top of that 500 petabytes. And my expectation is that from what I'm seeing in other parts of the landscape, that they'll come back again in a quarter or two and say, we need a double digit number of exabytes.

48:11We undershot. And we're seeing that across the board, both in the smaller and medium sized environments, as well as in the big environments where they thought they would need tens of exabytes. So now they're talking in triple digits. And how much of it is big labs building versus end customers and end users? Well, the big labs are building because they have end customers and end users that are using it. And the AI clouds are building because they have both big labs and medium labs that are consuming it. And I don't think you'll find infrastructure that's just lying around and not being utilized.

48:52At the moment, in fact, the opposite is true. There is a lot more that people want to do that they cannot do. Some of these AI clouds have stopped selling more capacity just because they're sold out for the next year and a half. And so I think the limiting factor here is physical. It's land, it's power, it's chips. Building fabs takes a long time. And so as fast as we feel this is moving, and it is moving very, very fast, it's still the build outs are happening much more slowly than the demand requires from them. Will it sustain forever? I don't know. I'm sure over time there will be hiccups. but as far as I can tell we're just at the very very beginning of most of the world running on AI and if AI will really be able to do all of these things that we're starting to see it do then it should continue at least for the next five to ten years at this pace so we have a lot of work to do Do the circular deals and the whole like financing debt aspect of the ecosystem, does that make you nervous?

50:18Or do you think that's just like a means to whatever needs to happen for the ecosystem to be actually built? So I think going back to the beginning, the old stack and the new stack, I think there's a new ecosystem getting built around the new stack. And some of the older companies are joining this ecosystem and some have not yet joined this ecosystem. And so it's lacking in liquidity at some points in this supply chain. We see the build outs that need to happen in some points need money in order to finance them. And sometimes you need that money ahead of when the results appear and ahead of when you actually get returns on that investment.

51:08And I think the companies that are in the ecosystem are a natural spot to get that liquidity because they see what's happening. They believe in it. They understand how big this is going to be versus someone who's on the outside looking in that maybe feels this is riskier for them to finance. And that's, I think, what's happening. Speaking of finance, you're in a very interesting and somewhat unique position because based on the little ad, I think you guys are profitable. We are. Which is not the case of just like a lot of other players in the ecosystem. What does that tell us in terms of where risk in the ecosystem?

51:53Like you happen to be in this layer, which is indispensable, and therefore there's no other competitor. Therefore, you can be profitable. Are you more disciplined? What means what, I guess? It's probably a combination, but I think it goes towards our business model. We sell software and so our gross margins are high. And as we grow on the trajectory of AI growth, which is pretty consistently been about 10x every two years, we get more and more efficient and we generate more cash and we generate more profitability the companies that are higher up the stack have a lot of compute costs that they need to do and the companies that are underneath us in the stack are building hardware and so they too have a very different business model than than the one that we have but yeah i think the combination of fast growth and efficient growth is something that we've built into the way we work from the early days.

53:04And it goes back to VCs not necessarily wanting to invest in the space in the early days, so we needed to be self-sufficient. And it also goes back to VCs not wanting to invest in the space in the early days, so we have less competition than maybe exists in some of these other layers. You have a lot of customers in the NeoCloud ecosystem, and I'm curious about what you view might be on that ecosystem. I mean, there's literally hundreds of NeoClouds. What do you think separates the one that will be around in two to three years as dominant forces in that ecosystem versus the ones that will not? So they need access to power.

53:50They need access to hardware. They need access to money. And I think the ones that will succeed are the ones that build, that know how to build. I find the end users come, what NVIDIA calls off takers, and they want immediate access to compute. and if you have it, then they'll come to you and they're willing to pay. And if you don't have it and you tell them, let's build it together, we'll have it ready for you next year, then they'll go to somebody else. And so I've seen the ones that are most successful be the ones that build in anticipation of demand rather than behind the demand. Build physically or build capacity through rentals or?

54:34Everything. And you don't have to own all of the layers of the stack. There are multiple layers of the stack underneath us and the NeoClouds. But yeah, and I find the ones that have that combination of good relationship with the hardware vendors, access to power and land, and that are savvy in the way that they finance their operations, be the ones that are most successful. Yeah, and to the last point about money, one can debate whether that's still current or not, but the case against Neoclass for a long time was that, well, effectively, they're kind of like a financing vehicle for NVIDIA GPUs.

55:26The primary skill is finance, and eventually hyperscalers will beat them because hyperscalers will always have a lower cost of capital. is that in the three aspects that you mentioned, is financing skills as important as the rest? What do you make of all of this? I think that the reality has proven to be the opposite of what you just said. The fact that we have so many of these new AI clouds and the fact that the hyperscalers have not beaten them yet, I think it goes back to the innovator's dilemma. When you have something new to build and you have this old cash cow to focus on, it's a lot harder than when you have something new to build and you don't have that legacy.

56:16But yes, regardless of where it started, I think these AI clouds have built up a skill set and a specialty. And it's not the same to build this new stack and these AI factories versus to build the old stack and the way that the hyperscalers built clouds five and 10 years ago. And every day that passes, these AI clouds that are actually in the trenches doing the work, they're learning. And the neoclouds that are sitting on very cheap financing, but not yet, the hyperscalers that are sitting on very cheap financing, but are not yet doing this work, are not learning that. And so the gap in skills just keeps increasing.

56:59Now, I think the hyperscalers have realized that. And if two years ago they were saying, yeah, we have this, we have storage, we have compute, we have networking, we know what we're doing, we don't need to worry about this new thing. Today, they're singing a very, very different tune. And I think they're aware of the fact that their lunch is being eaten by someone else. And I think they're reacting to it. So it'll be very interesting to see how that dynamic evolves over the next couple of years. What do you make of sovereign AI? It seemed to me that the concept of AI factory that we discussed at the very beginning was largely around this concept of sovereign AI like a year or two ago.

57:44As we discussed at the beginning of this conversation, it seems to have expanded now to also include enterprises. But from your vantage point, what's the reality of that concept around the world as it applies to nations? I think this becomes a basic need over time in the same way that electricity is a basic need or that water is a basic need. AI intelligence will be a basic need. We will not be able to live in the way that we would like to live without it. And so in the world that we are in today, countries are adversarial one to the other and they don't trust each other. And in the business community, as much as we have competition, I think we collaborate very, very nicely.

58:32But nations like to have their own. And so every nation wants to make sure that the U.S. can't shut down its AI or that somebody else can't shut down its AI. And so they're trying to build sovereign AI solutions. Obviously, not every nation can build the software. Not every nation can build the hardware. Not every nation can build all the different layers of the stack. But I think they get comfort from knowing that it's within their borders, at least. And within their jurisdiction from a legal perspective. Zooming out a little bit from that whole stack that we described with the five layers, where do you think the value accrues over time?

59:20and which part of the stack commoditized? People have been talking about model commoditization for a very long time. Does that commoditize? What about the rest? I think the best way for me to think about this is to look to history. And historically, hardware tends to commoditize relatively quickly. We have, because it's standard, not because it's easy, but maybe it's easier to copy. I don't know. The software layer tends to accrue a lot of value. If you look at the most valuable companies in the world today, if you look at Amazon, if you look at Microsoft, if you look at Apple, if you look at Google, those are the companies that built those software layers of the cloud, of the PC era, of the mobile era, of the internet.

1:00:08And they are worth trillions of dollars as a consequence. The application layer, there tend to be a lot of applications, some of which will have a lot of value and others will not have that much value. But I think consumer loyalty is something that is also valuable in those previous revolutions. And so the software infrastructure layer tends to accrue a lot of value and then on a more selective basis at the application layer. Where the models fit into that is very, very difficult to say because we never had a model layer before. So that would be interesting to see. And you see the model companies have become application companies and the interesting semi-recent trend has been to see the application companies become model companies again, starting to do a lot more work, whether that's a cognition or a cursor or...

1:01:07So I'm not sure those two layers will stay separate over time. Perhaps they merge back into one. In the old stack, we had only four layers and the new stack perhaps that merges as well. I'm curious about how you think about the place of NVIDIA in the ecosystem. The name has come up already a bunch of times during this conversation and it's so central. They're an investor, they're a partner. How does one work with such a dominant player in the space in a way that preserves optionality and preserves your independence and self-reliance as a business long term? So interestingly, other than as an investor, there's no legal document between us and NVIDIA.

1:01:59They don't resell our solution. We don't actually do anything together from a legal perspective. Having said that, they are our best partner by far. And at any given point in time, we have a high teens number of projects that we're collaborating with them on. We have a triple digit number of developers working with them. They have the same on their side with us. Everything ranging from new networking gear to new inference microservices to this agentic confidential computing project to the next generation of GPUs and DPUs and how we take advantage of them. all of that we love because as we build this together then the solution ends up being better for our joint customers and so we really enjoy working with nvidia also from a cultural perspective they move fast they believe that anything that is physically possible is possible they believe that it can be done when others believe that it cannot.

1:03:14And so it's been a lot of fun working with them in parallel to it being also very lucrative because they are the ones that are creating these new environments and putting names on all of these new concepts. And so they're leading the charge and we're trying to, as much as we can, ride their coattails. Having said that, there's no type of exclusivity. We work with AMD, we work with other companies in parallel to working with NVIDIA. In practice, they do have the majority of the market and so the majority of the deployments that we have are with them. One thing that strikes me in a big part of this conversation is that you guys seem to be just relentlessly launching new products, expanding new customers, having key relationships.

1:04:09We mentioned NVIDIA, but XAI has been a very important customer for you guys. What have you learned in terms of moving at the speed of AI? Well, you can't move faster than Elon. I was fortunate to see some of the ways in which he pushes his team to move faster. and it's super simple. You find the limiting factor and you get rid of it and then you find the new limiting factor and you get rid of it and that cycle keeps accelerating you faster and faster and faster. And that's what we like. The reason we keep innovating and keep accelerating and keep demanding more from ourselves is A, because we're paranoid and we're afraid that somebody from somewhere will pop up and try to catch up to us but B, because we love it.

1:05:02We enjoy it. It's the joy of building new things and exploring and figuring stuff out that you didn't know how to do last week or last month. And our customers expect that from us. Our customers are the most demanding in the world. And they're all in a race. And they're racing each other in this quest for better AI. And they need us to never slow them down. We can never be the limiting factor in their progress and in achieving their success. And practically for the founders or leaders listening to this, how does one actually do that? Meaning finding bottlenecks and because the bigger the organization, the more bottlenecks they are.

1:05:49And they may be at the top of the organization or at the very bottom. As a CEO, how do you do that? You have to talk to people and ask them that question. they know as CEO you don't know but the people that are on the ground know and so you need to build an organization that's as flat as you can possibly build it such that you have direct access to everybody and you need to build an organization where people are not afraid of raising those problems there cannot be a chain of command because that slows things down I think I heard this from Elon, bad things should be stated loudly and often and good things once and softly.

1:06:35So we can't get into a mode where we're congratulating ourselves too much. We always have to be in this mode of somebody's going to kill us. We don't know who it is and we have to keep fixing everything so that they don't have a crink in the armor to get in through. All right. So maybe to close some kind of forward thinking, projecting ourselves in the future a little bit as possible as it can be today. But 10 years in from now, whatever, many years, looking back, what do you think that the industry believes today that may turn out to be wrong? Like just one thing that may come to mind. I think 10 years from now, again, assuming AI, and I think we're starting to see proof points, will not plateau at the level of human intelligence, but will continue beyond the level of human intelligence.

1:07:31And assuming it has physical access to all of the things that we have physical access to on this planet and in space, I think everything is different. Nothing is the same 10 years from now. We will not be needed for all of these tasks. And I think the way government is structured will be different. And I think the way money is transferred and ownership of things will be different. And the way we think about how we live our lives will be very different. And so it's impossible for me to imagine that world, except to say that it's probably going to be a lot more different to what we have today versus where we are now and where we were a thousand years ago.

1:08:28So I think in the next 10 years, we'll see more difference than we did in the last thousand years if this plays out in the way that it seems to be playing out. And in that world, and maybe that's not 10 years, that's just five years, in your wildest dreams or perhaps your very pragmatic vision, where does VAST sit? Becoming the operating system for AI, what does that actually mean if everything plays out? We want to, A, enable this so that it actually can happen and that we don't see obstacles in the form of we don't have the required infrastructure for it to happen because we can only build this fast and we want to build a thousand times faster or a thousand times bigger.

1:09:13We need to make sure that it doesn't kill us and so that it's safe and that along the path, we put safeguards in place and enable us as people to understand what's happening at the very least, if not guide what's happening. I tell my team all the time, if all the data is managed by us, we can't ask for anything more than that. And we're still a ways away from that. But that's the ideal that we're marching towards. All the data in the world. All right. Well, that feels like a wonderful place to live it, Renan. Thank you so much. This was a terrific conversation. Thank you. Hi, it's Matt Turk again.

1:09:56Thanks for listening to this episode of the Matt Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.