Inside the Race for AI Compute, Why AI Labs Must Slow Down & Ex-OpenAI Researcher on RSI

27:57Jerry Tworek:I think that's kind of what the role of the Neolabs is. We have incumbents that are very successful and have established techniques that are used for training models. There needs to be someone who is exploring alternatives and hopefully one of us, maybe us, will find something that actually works. So, you know, you think so deeply about what agents can do and about research, which brings us to the conversation around RSI, which is something that you likely spend a lot of time thinking about. What do you think people get wrong about the conversation about RSI right now? I think the most thing people get wrong is that RSI is a spectrum and it's not something binary that you have.

28:44You think it is a spectrum? It is. It is. Okay. So we all never really know.

28:49Jerry Tworek:I think I'm one of the people in the market that maybe thought of it longer. There are probably definitely some that have been working on it as long as or longer than I do, but there are very, very few people in the world that did so. I did very first work at OpenAI on coding models. I did work on coding and applying coding to doing AI research for majority of my career. I did start the project of building AI scientists internally. And now I'm trying to bring it even further with trying to think, how do we build a whole company about the concept of the AI scientist? And in some ways, what is RSI?

29:33Jerry Tworek:RSI is applying AI to developing better AI. And in some ways, we've been doing that for years already. It's just the fact is that through the recent developments in coding agents, it's going now faster than before, faster than a year ago. Because even a year ago, we were applying. When you are in a lab, you have AI, and you use it for whatever you can. AI, in many ways, is an automation technology, and you automate as much work as you can. But the story is the better AI gets, the more you can use. We don't have what I consider like fully autonomous or strong RSI where the moles can just improve themselves without human supervision.

30:10Jerry Tworek:That is just not happening. And honestly, the moles' research tastes and research ideas are not very good yet in any way. So do you think it will get to the, I mean, I hear you, it's a spectrum. So getting to RSI is maybe the wrong question to ask. But do you have confidence that the labs will get to, you know, what they think to be RSI here? Sure, surely it will happen. The question is when. I don't think there's anything that we can say, hey, I will never do this. AI will do everything eventually. The question is, is it this year? Is it next year? Is it in five years? Yeah. Those are very practical questions.

30:47Jerry Tworek:And in reality, what I think is that more things are algorithmic bottlenecks than people realize. And with one or two good new algorithmic ideas, you can probably get there pretty quickly, which is in some ways, this scary realization that there can be someone in the world somewhere in the lab just having a good idea and running one good experiment, and that good experiment could suddenly create a very, very smart system that we didn't foresee before and we don't know. In many ways, scaling laws are nice because they give you a predictable way of improving capabilities. But what me and my team did at OpenAI with introducing reinforcement learning and scaling and reinforcement learning bring completely new abilities that no scaling law would be able to predict for you and no scaling law could factor in.

31:43Jerry Tworek:Right. If there is any next algorithmic improvement, if there is any new step change in how we do it, then the further you get, the scarier those get. Because those capabilities are something that we don't really know how to protect and control. I want to end with something that you wrote. You said the road to singularity will be paved with BlackBerrys of our generation. and I presume you're talking about the BlackBerry the the phone I do I do yes yes what are the BlackBerrys of this moment right now that we are seeing I don't I don't think I know and I don't think I can I can call those out in any way but you know you take a guess I'm not trying to I mean here's one like you know so open claw but you know let's look it came it it took Silicon valley by storm it inspired many of these agents right now that people are getting very excited about but people are not as excited right now about using open exactly and so i would go out on a limb and say i think open claw itself is a blackberry very likely of our generation so yes you know i mean think about what what are the tools right now that you think uh we might not be using?

33:00Maybe the use cases, the agents, what are the BlackBerrys?

33:04Jerry Tworek:In some way, very likely, majority of the things we are doing today and using today's will eventually get sunset. I think we are still pretty early in the deployment of AI. In every wave, there are early winners and early stories of success, but the longer-term rollout and the longer term success is something very, very different. So it was very hard for me to say, but I was generally trying to imply that there were a lot of things that are good at the moment and are valuable at the moment. But at the same time, the more time passes, the more longer term economic trends and longer term efficiency matters and the competition becomes more about fundamentals rather than, for example, about who was first or who did something.

33:54Jerry Tworek:new. It is very easy to do something new right now, something that no one did before, and that is really, really great. And you can get, if you are the first person in a new market, you usually get a lot of it, but then others show up. Others show up sometimes, sometimes quicker, sometimes later, and we are getting more and more capable players in this market and where things will end up, I think it may look very different than where it is today. Great. Well, Jerry, I want to thank you for joining us. That is Jerry Twork from Core Automation here on TI TV.

34:42I'm here with Gokul Rajaram, the founding partner of Marathon Management Partners. Gokul, welcome to the show. It's great to have you here. Thanks. Great to be here. So you are, you do all sorts of great writing and consumer is one area that I know you've been close to throughout your career. And we have this agent out, Muse, which is taking the airwaves by storm right now. Everyone is very fascinated by it. It seems to be impressing a lot of people. What have you made of the rollout of Muse? And do you think agents are ready for prime time? How are you thinking about this? Look, I think Muse is an extremely well-designed product.

35:20the people who built it are friends of mine and I think they've done an excellent job of going behind and exploring some of the dogma and making it a truly useful product. I use it a lot myself. I also use instinct. What do you use it for? I use it for as a personal assistant, essentially to remind me of things, to essentially make sure my travel, my restaurant reservations, and my chores and errands are all on track. Okay. But I don't give it access to my email yet. And so I think the number one thing... I don't know. I would not give my assistant access to my email. So I think of it as a personal assistant.

35:54I give assistant access to my calendar, but not to my email. This must really frustrate your assistant, I have to tell you. You know, I think email has personal stuff in it. And I think the number one thing that assistants need to transcend to really be used by, and even more useful, be used by everyone, is one trust, which is people need to trust that you'll do the right thing, that with your email, email will not, there was an ex post yesterday with Instinct where it accidentally leaked apparently according to the user, someone else's messages into this person's chat with the agent. Wow. And so that kind of stuff, you just can't have happen.

36:32So I think trust is the number one thing, the psychology of human beings and trust. The second thing is, I think, bringing it to the workplace because half of our job, I use it for personal stuff, But how do I get connected to my work email, to my work calendar? I have an IT team that runs this. They don't want agents in there. It was super hard to get almost any external service. It was hard to get AI agents like Claude and Chad GPD connected to my work account. Now, getting this set of agents connected to my work account is another barrier. And particularly in regulated industries, in finance, legal, healthcare, these agents getting access to data that might be accidentally leaked or just comfort with that, I think that's the barrier to entry, really trust.

37:15So if trust is the issue, then, I mean, on one hand, we have a conversation about the models approaching RSI and the models being dangerously good. And then we have these incidents where they just seem like, you know, mistakes that any model should be able to avoid. So what's the solution here to generating trust? Is it that the models still are not good enough? because we have people saying they're dangerously good. What's going on? I think it's the reason fundamental root cause is the lack of guardrails. And I think that models assume trust versus earning it. And I think a good example is, I think on the medical scribe section, we've never heard, I think there have been issues, but there are companies like Abridge and others who are now used by doctors in very sensitive cases in sensitive situations to basically scribe the voice memos.

38:08Right. They all started by first getting the doctors to approve every single thing they transcribed. So 90 % of things are approved. Slowly, they move to only having 20, 10 % of things being approved. So you assume that it's untrustworthy and you slowly get every single thing reviewed by humans, but you move to a situation where very, very few things are reviewed by humans. I think right now, Muse, Instinct, all of them need to earn my trust. Is there a historical parallel, do you think, for a product that has required this much trust in technology? You've been through so many waves. I mean, mobile, social media, crypto even.

38:50It strikes me that there's not really a parallel for this. What do you think? I don't think any product has wanted access to a third-party source of content. It doesn't control this level of depth. Google is a good example, though. We do trust Google. We search, use it for email. We do all of these things. but they host it. It doesn't do it for you. It's not sending the email for you. Exactly, exactly. But we do trust it with data and what we look up and it won't leak into other people's searches. I think transparency and control are the two things. I have actually given both Claude and ChatGPT access to my email.

39:21Why? They're very clear connectors and they very clearly tell me that when I do emails, I will only put it in draft mode. I will not send the email. And so you have read-only access and you have write access separated out. And so I think all these agents need to very clearly say what controls they offer and be very transparent about what they can do when you give them the control. So then when enterprises get nervous about data privacy, zero data retention, the policies that these labs have, you have people like Alex Karp coming out and saying, we should not trust the labs with our data. Do you think there's validity to that?

39:58I think as an enterprise, you cannot be dependent on a single model anymore. You have to rent the model and own your data is what I believe. In other words, we know, unlike a year ago, we know this is going to be a multi-model world. We're going to know there are going to be at least 10 models between open-weight and close-weight models. And so there's no reason to go all in on a single model. You need to build a routing capability so you can easily switch between different models, the right model for the right task. But you need to own your data and keep it to yourself. There's no reason to give your data to the model anymore.

40:28You can rent the model and switch between models. Right. I want to ask you about some of the writing that you've done. You do a lot of great writing on the current state of tech, on your perspectives, on analysis. One of the pieces that you wrote was talking about the customers that AI companies need to attract with their products. And look, customers translates to revenue. But I think the point that you were making in this piece of the new kingmakers is that it's not necessarily about how much revenue, it's about who that revenue is coming from. Talk to me a little bit about what message you were trying to send with that piece.

41:02Yeah, I've seen, I've had literally a few companies that have been selling to enterprises, CVS, Walmart, Pfizer. They've got to millions in revenue, but it's been very hard for them to get venture interest. And I think while I have companies in my portfolio that I've sold to, say, two customers, which are AI-native companies, very young companies, but they've literally been able to raise round after round. And I was trying to understand why. And this is why I wrote the article. And I think the reason is that just because you're selling to these Fortune 500 companies doesn't mean your product is good.

41:33Because these companies are not yet sophisticated enough to truly put your product to its paces, to truly evaluate your product vis-a-vis comparables, and to truly, when I as an investor talk to them, tell me and give me confidence that they are a sophisticated buyer that knows the landscape out there and have picked your product after rigorous evaluation. It could just be a pilot. It could just be a pilot. It could just be because they met this company. They've never met any other company. no one else was able to get to them. However, I know that an AI native company in Silicon Valley, they're literally any product trying to sell to them faces incredible competition from their own engineers who are trying to build it from every other company that's trying to sell to them.

42:07And any company that has been adopted fully by this company has had to run the gauntlet. And so validation of one or two of those top folks, I mean, of course, the labs would be amazing. But even the next set, which is Notion, Granola, Decagon, Sierra, those kinds of companies using a product is a huge validation of the product's quality. of the ability to be better and being able to go through a competitive gauntlet with these companies. So what you're saying, what I hear you're saying is that the Fortune 500 is no longer the customer that companies should be after. They should, if I think the sales cycles are far too long and the validation you get, it should not be, yes, it is no longer the quality of customer.

42:49The highest quality customers are first the labs and second AI native companies, foreign AI companies. So is there, you know, an equivalent index that you want to put? I mean, you should do that. You should put together your own index of, you know, I don't know, the Silicon, yeah, the Kingmaker 50 or something, you know, get the notions together. Because, I mean, that's what I hear you suggesting is that you need to get the AI native companies to adopt it. There is a secondary question, though, that this leads to, which is customer concentration. I hear you on that you would want the AI labs to be major adopters and you talk about distribution you know them telling their friends about it being your ambassadors but what do you think then about customer concentration risk and the fact that and this is a slightly different topic but you know these chip companies and infrastructure companies neolabs I mean 70 % of your revenue coming from one customer what do we do about that I think you've got to diversify I think this is at the very start when I think this is really was applicable to seed and CDJ companies.

43:52The first two customers you want are sophisticated customers who are really putting your product to its basis. So I think those are the first companies. But then as you grow beyond that, you have to go outside. You have to go outside, not just the AI native eco chamber of Silicon Valley. You've got to go outside tech. You've got to get non-tech companies and so on. But to start with, I think the best companies you can get is these companies. They're the best companies that are going to really push your road back. I agree the AI Kingmaker Index sounds good. When you say best companies, how are you valuing companies today?

44:22What are the right multiples that you are thinking about? Is it still just growth, profitability? Are there other metrics you're using? How do you assess the trade-offs between these two? I think in the Valley, there has been a very strong focus on growth at all costs. I think you've got to look at efficiency. Efficiency is two things. One is, what is the sales and marketing cost to get a single dollar of revenue? So this is the burn multiple. You basically look at how much new revenue did you get by spending a dollar of sales and marketing. If you can't even get, if you're spending a dollar of sales and marketing costs and you're getting less than a dollar of revenue, that's a poor burn multiple.

44:59So second thing is, after you get that customer, do they expand? Do they stay at a dollar or are you going to able to retain and grow them with time. So those are the two things we clearly look at. I actually think the number one metric for business quality is net revenue retention and net revenue retention. But I mean, talk to me about the AIR. What's different about, I mean, you've, because you've talked about the ARR multiple not being the multiple to use anymore. So why? It's because the ARR multiple in isolation without looking at the burn is basically irrelevant because capital market, right now ARR multiple simply is a number that is divorced from the capital market's reality.

45:39When the market is expansionary, all things are good. But if the market turns, if you have a very high burn multiple, you're dead because you're relying on markets to finance every leg of growth. What I want is a company that when the markets turn, they can still keep growing efficiently and they can raise around even in a bad market. Right now, 90 % of companies are growing because they're burning so much. The only way they can raise is to raise one in a good market, in a great market. So you're saying that we need to get out of the growth at all costs mentality of evaluating the AI companies, the rule of 40, we need to start applying to these series C, D stage AI companies.

46:16At some point in growth, exactly. I think if you are at, say, a billion dollars in revenue or even 100 million, 200 million dollars in revenue, the fact that you still have to raise another billion dollars to grow and you're still not able to show operating leverage is challenging. I mean, it works. It works till it doesn't. It works till it doesn't. When will it stop working? I wish I had the crystal ball. I wish I had the crystal ball. Great. Well, Gokul, I want to thank you for coming on. That is Gokul Rajaram from Marathon Management Partners here on TI TV. Thank you.

47:01I'm here with Finn Barnes, co-founder of the General Partnership. Finn, welcome back to the show. It's great to have you here. Good to see you. I'm glad I saw you in the audience. So we just finished a session here at our AI Agenda Live conference talking about the data center buildout that is going on. and we had some great guests on the panel. You and I caught up afterwards. And I mean, I wanted to bring you on here because we had a conversation about your vision for distributed compute, which is slightly different or maybe complementary to the data center build-out that we're seeing right now.

47:33Walk me through what your thesis here is on what this distributed build-out would look like. Yeah, so I've been just very interested in, first, how AI gets built, and we have to build massive investments in infrastructure in order to see the capacity sort of increases that we need to see AI continue to march forward and sort of all the value that it can create. At the same time, if you have compute and AI, you have intelligence but no distribution, then you're in a lot of trouble. The value can't get to the edges to the people who need it. And so I've been very interested in the systems, software and physical systems that will deliver sort of the AI to the end user and then figure out how to do that in the ways that are optimized for very specific workloads and needs that each individual user has.

48:20So what does that mean, tactically speaking? I mean, we've got these data centers. They're getting built. I hear you. How does it get to the end user? How does it get to the end user? So I think part of it will be cloud access and telco and the infrastructure for the Internet network. You think about what that network will have to support as we think about physical AI, robotics, and the criticality of that support. So right now, if the internet goes down, you and I can't have a Zoom call. But if you're in the middle of a heart surgery, and AI is running that surgery, and the internet goes down, it's a much bigger problem.

48:54And so I think there's a bunch of interest in sort of how the telecom network has to be built out and what that needs to look like, how it has to be managed, probably managed by AI. And then also interest in smaller and smaller models that can live either on device or closer to the edge. Like edge computing. Edge computing. And then figuring out the software that orchestrates when does a workload require the frontier models and going to these massive data centers that we were just learning about in Abilene, Texas and other places. And when can that workload be executed at the quality required, you know, maybe on the chip on your phone at some point.

49:31So when you say smaller models, you know, and we saw this week, I mean, the models are getting smaller. But when you say small models, so are you suggesting, is this at odds with the data center build out then? Or what are you suggesting? I think big models and small models could run in the data centers. The question is, if a small model can run locally, is it economically beneficial? Is it functionally beneficial to have it run there rather than in the data center? And so I don't think this will necessarily have impact on how much compute people think we need to build in these centralized data centers, but I also think there'll be workloads that are pushed out to the edge and that a lot of that compute that exists already on devices either in your home or in your pocket, that those will be able to handle some amount of the demand that we're seeing, that we're forecasting.

50:22Why not, though? I mean, why wouldn't that jeopardize the centralized compute build-out? Because if it's happening on your device, Aren't we just one innovation away from all these data center buildouts? Oh, no, it's a good point. I think it will offload some of the demand. So it could be one reason that we could end up overbuilt. Like in every previous infrastructure buildout, at some point we were overbuilt. And we saw that in the 2000s with all the dark fiber. And that could be the case here. Although I think in this case, the technology creates so much value across such a broad swath of the economy That it feels like the demand is, you're better off to estimate it as infinite rather than to try to guess, like, at what point would local compute mean that we have too much centralized infrastructure?

51:07But I think when you think about where, in another panel we saw, you know, Blackstone was talking about sort of their portfolio and the average use of AI by employees inside that portfolio. And they said it was well below the median cost of something like$12 a year, basically non-usage inside. And so all of those folks are going to learn how to run their life on leveraging AI over time. And as they do that, I think the complexity of those jobs, the vast majority of them is likely to be able to be handled by non-frontier models. Right. I want to ask you about the interest rate environment that we're in, because you are a macro guy yourself.

51:50I mean, all these data center build outs in a regime where interest rates are going up. What do you think is the fate of these large scale build outs that are happening? I think it puts a lot of pressure on time to build, which we heard. Because you've not just got the overcapacity issue now, you've also got the funding risk as well. I mean, do you see these projects stalling, getting canceled, companies merging, consulting? What do you see end up happening? Yeah, I think you could see where the folks who can support the build-out, they have to continue to be this massive scale and continue consolidation.

52:26I think NVIDIA is doing a great job of positioning themselves sort of as the central bank of the space, and I think doing a great job making sure that the vast majority of their partners in these bidirectional relationships are smaller than them, and therefore, they continue to maintain sort of the control and can set the direction of the entire industry. Right. What about customer concentration risk for these big build-outs? And not just the build-outs, I mean, the chip companies too. you know eventually my sense is this is going to come back to bite them right I mean how long do you think this this can sustain itself where you have one customer that accounts for 70 % of your revenue or five I mean in the case of Broadcom they've got like five labs that are that's all Hawk Tan talks about on the call how long does this go on for yes I think I think there's I think there's a few ways to look at that but one would be today Nvidia is clearly the dominant platform, I think you will see sort of heterogeneous data center start to be an interesting thing.

53:25I think you'll see increasing numbers of new chip designs that are sort of more and more workload specific, therefore forcing heterogeneous data center design. So you're going to need to have multiple types of chips in a given centralized data center. But I think the other thing, the other side of that is if it's true that the larger the cluster is, the more powerful the models can be that are trained on that cluster, the faster they can run, the more efficient that entire sort of economic chain is. Then when you think about sort of customer concentration, I think the consolidated companies will be able to have, you know, one gigawatt data centers, but they'll have 10 or 15 or 20 of them.

54:05And they'll be hosting multiple different customers across those different campuses. And so they'll sort of get away, I think, from customer concentration by just rolling up the market in this way where you're going to say, yeah, I serve. So you have a facility dedicated to a customer. That's the future. Yeah, I think you will. Because to get to that scale, I don't think you're going to see the data centers themselves go from a gigawatt to 10. But are there enough customers that can afford to have an entire facility constructed or allocated to them? Because there's not that many customers to decide.

54:42I think there's a very small number, but I think that they are so dependent on this as their core currency, effectively. The token is the currency. And you're providing that, right? I mean, that's where you sit. Right. Let me ask you one more question about another segment of the AI story. So these expert data labeling platforms, the Mercores of the world, Handshake, Turing, Scale, Surge. I mean, look, there's a conversation that I've been having with a lot of people here on the ground, which is, so these companies are invaluable insofar as the data that they provide. What is the sustainable strategy for these companies down the line?

55:24And I think the consensus is they're going to have to move into enterprises. Not so dissimilar to the strategy that the NeoClouds are playing as well. It's enterprises that are going to be the future. Do you think that the enterprise software companies are going to have demand for these expert data labeling platforms and also the neoclads too? I mean, I guess this is a question about how vertically integrated you see enterprise software companies being down the line. Is this a sustainable bet or is it kind of a long shot? Because I think the data space is fascinating for a few reasons. The first is privacy.

56:05And so I think to advance AI, you need data that is effectively labeled. You need to know that person A took these actions across a given workflow. But that typically requires you to reveal PII about that person. And so you have to obfuscate that to be in compliance in the United States. Certain other countries don't have that problem, and so there's no assumption of privacy. And so they can train on all kinds of data that we can't. There are companies, we invested in one called Integral, that is working to make sure that the data inside a company that then is used for training, that those labels do not reveal PI, that that data is compliant.

56:45And they sort of make it possible for someone to purchase that data and to train on it with high degree of accuracy and being able to track a given human going across these workflows, but not know who it is. And so they obfuscate that in an appropriate way to make sure that data is compliant. So I think data compliance and privacy is one area of the labeling landscape. That's interesting. But then for these big data, and the data labeling companies are not doing a great job around this compliance. And you see them selling data. They're not. Some of them selling data to China, selling data to Chinese companies, Chinese labs.

57:18Some of them, you know, there's names being revealed. Do we know that? Yeah, I think these are known instances of sort of fuzzy with the rules around compliance on some big data labeling companies. And you see that in the process that they go through in order to label it and then ship it off. And a lot of the companies now, they're trying to buy corporate data. You know, we want your Slack, you know, cash, and then we're going to look at that, and we're going to help train models on it. Asking about the process of de-identifying that data is pretty critical, and I don't think there's been enough attention paid there.

57:47So that's one piece. So what do you think? I mean, let's talk about that. What, you know, what do you see as the solution to that? if that's a problem. I mean, is this some type of regulation? I mean, there are regulations, and I think it's just a question of enforcement or detection. Oh, interesting. And there are solutions to this, and I think you can be compliant. Right. This is like export control. Yeah, exactly. Okay. And then I think the other area that you're pushing on in terms of why would an enterprise want data labeling companies to be in their business labeling their data, well, that sort of validates what I've been saying about sort of customized smaller models.

58:26is that they're doing that in order to post-train or fine-tune models to be more accurate for their specific workflows and use cases and style and taste, rather than using the out-of-the-box foundation model. So you think enterprise software companies will post-train their own models? I think if the economics warrant it, then they absolutely will. Because right now it's expensive. It is. It's expensive. And also I think there's a bunch of concerns about accuracy, ease of use. I think there is something to be said for you pick your lab of choice, you're engaged in their application software, and then their software can help you route to the appropriate model.

59:02It's a little hard because their incentive is for you to burn the tokens and your incentive is to save the tokens. So you have to be a little bit careful about who makes that choice. But I do think that living inside a piece of software that then figures out the most appropriate model to use with latency in mind, with accuracy in mind, and with cost in mind is the future of us using AI. And when these companies want their data labeled and then used for post-training, they're effectively trying to work down the curve of less and less expensive models performing the same level of intelligence, which I think is a frontier.

59:37What about the business model for these labeling companies in the sense that it feels like batch revenue right now? It's not exactly recurring. It is that. And so that's a very different business than what tech investors, I think, have gotten used to. Is there a way to get away from that? They still call it ARR, though. Right. Right. But it's, you know, we can call anything AR in this day and age. Yes, we can. But do you see a path to actually making it ARR? Or even when the enterprise software companies come on board, even when they start post-training their stuff, is it still kind of lumpy revenue?

1:00:21I think you end up with sort of a batch process problem because you're going to label, you're going to have data from a certain workflow. You're going to decide that you want that workflow to be incorporated into it. You want AI incorporated into that workflow. You're going to label that data post-train files. And that model will then be able to perform to the level that you want it to. And then you're going to find the next thing. And so I think you're going to move down that curve. And eventually you're going to have AI integrated into everything. I think the other force that sort of cuts against the business model of the data labeling companies is the ability for individuals to create deterministic coding application, coded application.

1:01:01So models are very good at absorbing variability, right? And so they're probabilistic. And so if you have a workflow that's constantly changing, a model can adjust to that. And a human can use a single model and have it in that workflow and be able to absorb that variability. over time, if you can write code that figures out the repeated processes in that workflow, even the JEV model is just a classifier, right? That sort of says... You're talking about like a personal eval? Well, so I get to that. But this is more just sort of saying you have a given workflow and there's some amount of variability in that workflow.

1:01:39And the more you can eliminate variability by having deterministic software handle the job, the less expensive it is and the faster it can go. And so if you think about the end result of an internal user at an enterprise taking their workflow, creating an app, their vibe coding an app, and then using that app as a daily driver to do their job, right? What that effectively means is the inputs and outputs of that app, this deterministic app, those are fantastic data labels, right? So it would say, this was the ticket that came in. This is how I handle it. I press a button. I put it into this category.

1:02:17And I do that over and over and over again. Seeing the model actually in practice. And so that's a deterministic thing. It's labeling the data. So now the employees are labeling the data. Right. And so you don't need to... Like what Meta... What Meta's doing. Exactly. As you sort of have this application that you've built to let you do your job faster. And then there'll be pieces of your workflow that are always a little bit variable. And so you need a model that could be a classifier, like Jeff. So a small model that's inexpensive and runs really quickly. and then there'll be pieces of your work where it's net new to you as you show up and so you really need the most powerful model to help you do the planning, figure out how to solve the problem and go all the way down to execution.

1:02:51Right, great. Well, it was a great conversation. Lots more that we could talk about, I'm sure. I want to thank you for joining us. Happy to see you. That is Finn Barnes from the General Partnership here on TITB.

1:03:19I'm here with Mark Boroditsky, the Chief Revenue Officer of Nebius. Mark, welcome to the show. It's great to have you back. Great to be here, Akash. Thank you for having me. So I want to talk to you about pricing, okay, because pricing, clouds, compute, I mean, this is the topic that everyone loves to debate. It's a question that I always wonder. How do you decide on a price? Because the conversation is that anyone who has any control over any compute right now has control over pricing. How do you think about that? Well, it's actually a really important element of the overall offer that we make in the market.

1:03:56And by the way, that goes for all products and services, whether we're talking about the jacket you're wearing or the price of gasoline, which seems to be a sensitive topic these days. It also applies to compute. And the reality is that we as a vendor obviously want to be able to price our products at a fair, but as high as possible price. And the fair but is the challenge. The reality is that you can look at things on a cost plus basis. You can look at things on a value derived basis. We want to be more on that side of the spectrum. And we're in a market that is probably like the gasoline market, supply constrained.

1:04:37And the reality is that for every GPU and every cycle of compute, we have many customers, four, five, six customers lined up for that compute. So figuring out the right and fair price is a priority for us. And you guys have been doing some innovative stuff here. I mean, auctions you've been running, preemptive pricing, how does all this work? So what we have been looking for, let's take a step back. Arkadi, our CEO, pushes all of us to be innovative. We are innovators at our core. Big privilege to be a part of an extraordinary engineering organization that is building amazing products and services, all very innovation-driven.

1:05:18And he wants, Arkadi wants all of us to look for the innovative opportunity. And that comes to pricing as well. So we're looking for ways to actually change the equation from us picking customers to try and get back to customers picking vendors. We know we have a compelling offer. We do know that we have a tough time pricing. So we've been looking for other mechanisms. And we've implemented two that give us a window into what the market thinks. We've implemented an auction where we made available a cluster of B200s to customers that had previously shown interest. And we ran the cycle and discovered that we were pricing too low.

1:06:04Actually, 15 % less than what people were willing to pay in the auction. And likewise, when we looked at our pipeline, our pipeline was 20 % less than what people were willing to pay. So we know we're off. The second mechanism, which we just announced this week, is we've put in place for our preemptible. This is like the lowest end of compute. This is available for people that have short-term focused projects that are okay with only having a minimum amount of compute available for a limited amount of time. We're now pricing that dynamically based on an algorithm them that takes into consideration historical use plus existing demand and the price range that people are willing to set in order to determine if you're going to be able to operate on the compute that's available.

1:06:53And so how often does this price change? Is this something that changes week by week, day by day? Every 15 minutes. Every 15 minutes. So you've created your own marketplace. Spot market. That's correct. The nebbiest spot marketplace for compute. That's correct. So it's going to give us all, not just Nebius, it's going to give the users and the broader market, and we think the auction does the same thing, an independent indication of what price the market will bear for that available compute. So this is all working very well right now because of the supply and demand dynamics. Like you said, I mean, you discovered that you were pricing things too low and people were willing to pay more.

1:07:39Let's fast forward three years from now. Let's say there's more capacity. Is the plan to keep these auctions in place? Because, you know, you could imagine a scenario where maybe the price is not what you want. And then what does that mean for your bottom line? Well, the reality is there will be a point where the market inflexes and it returns to more of a normal market. So long before what you're talking about, we want to see that inflection. And this gives us the ability to see the changing dynamic, the behavior in the market. So we're not overcharging. That's just, that's as much of a problem as undercharging.

1:08:15So we're going to hopefully have a forward indicator that helps us to, again, align to customer needs. Now, as it declines even greater, we're going to need to look for other ways to be able to derive or protect or justify value in our offerings. And that's the software stack that we're providing. And that's increasing every quarter with new offerings. And that's where we're going to have to see where we get the premium value. Until then, I think we have a very powerful way to keep our finger on the pulse of what the market's willing to pay. I want to ask you about the macro environment that we're in right now.

1:08:48So, I mean, rates are coming up. This is something that hasn't happened in a while. It certainly hasn't happened. I mean, it was the first hike in three years. ChatGPT came out, I guess, what was it, November 22, 23? So anyway, we haven't been in this scenario before. What do you say to people who argue that, well, the AI build-out is only possible in a low-rate environment? What do you say to that? Well, I think it's a little reductionist, if you're being completely honest, because today there is a significant premium being paid, as we just described. So that leaves a lot of room for potential movement in COGS, the cost of goods sold, which the cost to fund your AI build-out is part of the cost of goods.

1:09:41so I don't know that it's purely dependent on low interest rates. I do think that interest rates play a very important role in what we're experiencing right now. Every time interest rates go up 100 basis points, it is going to affect the cost of the offering and it's going to potentially affect the price that's being charged. Also, behind the scenes, as interest rates go up, private equity changes its priorities. And it's possible that private equity returns to, let's say, less risky, less growth-oriented opportunities in the market. Less private equity means less funding for some of these build-outs.

1:10:24It also means possibly less funding going into venture capital-oriented opportunities with AI startups. So as interest rates go up, the capital available to actually support the growth that we're experiencing may become more constrained. So how does that affect your business then, given that Demius has been very active in the debt markets? Well, the thing is that we need to be very careful in terms of the kinds of customers, the kinds of terms that we extend. And today we are already applying a pretty rigorous approach, working closely with venture capitalists, working closely with NVIDIA to make sure that we're finding the best AI startups, Neo Labs and what have you, so that we are confident that their prospects are great and that the likelihood of funding and continued growth is in the cards.

1:11:14So far, we've been very fortunate to work with some of the best and most innovative startups that have seen their way to great success. Right. Speaking of NVIDIA, I want to ask you about Jensen's view, but also just everyone's view on the AI safety conversation right now. I mean, this whole idea of pacing the frontier, we know that Jensen has taken a pretty hard stance on this. And I'm just curious how you feel about it because, I mean, you work with loads of customers too, and you have suppliers, of course. What's your view on this discussion? Well, I think every industrial category needs to make sure that they're delivering capabilities and services that are safe and reliable.

1:12:00That goes without saying. This recent pacing the frontier, that's AI, probability of doom. You know, the reality is there's a little bit of hyperbole in this discussion. And I need to actually call it out. We service everybody. We don't make models, but we service a lot of model makers. We don't service the frontier models, the open AIs and anthropics today. But the reality is that those are the two that are the most exposed and the most at risk. And I do think it's interesting that if they believe that there's a need to do something, Maybe they should be doing something on their own. There's nothing that impedes them from doing that.

1:12:46We as a ready supplier that's interested in supporting all the best companies out there will implement any controls or guides or restrictions that the market determines are appropriate. In the meantime, we've built into our platform the security mechanisms, the observability capabilities, the controls that give our customers, some of the most interesting and innovative players, as well as now enterprise players, the confidence that we have a reliable offering that can give them the controls they need to prevent bad things happening. What are the tactical things? You talk about the security measures that you can take.

1:13:25I mean, we hear about these agents going rogue, the OpenAI hugging face incident. I mean, I'm wondering on a hardware level, on a physical build out, are there things that you are doing or things that data center builders can do to prevent against this somehow? I mean, you know, look, I'm not old enough to remember exactly when firewalls were physical firewalls, but that was a time that existed, right? When these things were physical security measures. How do we deal with this? Well, the funny thing is those physical firewalls are a proxy. That software is all over the Internet. today. And that's still a mechanism that protects all kinds of very important boundaries that we all cross on a regular basis.

1:14:08And the reality is that's the kind of stuff that we have in place today. There are physical as well as soft boundaries. There's observability tools that are paying attention to what's going on. The real need though, and this is probably a more appropriate answer to your question, is we as vendors, all of us, not just Nebius, need to step up our game and be paying attention to what's going on. We need to be actually doing what we've done in the past as we started to see attack vectors take off. That is, we need to be paying attention to the most recent events and then looking for ways to identify that behavior on the platform.

1:14:43Some of that we'll do ourselves with tools we build. Some of that we'll do with partners and vendors that have tools that can detect it. But as we actually observe these things happening, we need to be implementing tools to be able to detect them, control them, and prevent them. This is not something that's new. This is the same that's happened with historical hacking experiences, DDoS attacks and what have you. Same thing has to be applied here in the way that we're thinking about delivering a safe and reliable platform that people can trust. Great. Well, Mark, I want to thank you for joining us.

1:15:18That is Mark Boroditsky from Nebius here on TITV.

1:15:35That does it for today's show. A reminder, we are on this stream Monday through Friday at 10 a.m. Pacific, 1 p.m. Eastern. If you can't make it then, episodes are available on theinformation.com, on our YouTube channel, or wherever you get your podcasts. Make sure to subscribe to us on Instagram, TikTok, YouTube, and LinkedIn. I am already excited for our next show tomorrow. Have a great rest of your Thursday. Bye-bye for now.