Ep 94: Applied Compute CEO on the Limits of RL, the New AI Hyperscaler & Why Post-Training Wins Inference

0:00How do you kind of characterize where we are today? What we have with RL is we essentially have a hill climbing machine. The hardest part is actually defining the hill to climb, which is why evals, I think, is like a really important thing for companies to, A, focus on building and, B, actually like safeguard pretty carefully. Your employees are not fungible. You would not like be comfortable with them going to another company and doing work there. Same thing with these models.

0:29Owning your own intelligence and training your own models is incredibly in the AI zeitgeist these days. There's probably no better person to speak with about all these topics than Yash from Applied Compute. Applied Compute is working with a bunch of the most cutting-edge AI applications on really interesting post-training examples. Yash and I were able to talk about his perspective on a bunch of the top questions around open models today, including when it actually makes sense to post-train these models, how to think about the future, when this stuff makes sense, especially as the labs get more and more data from different domains over time.

0:58We also talked about when it makes sense to harness optimize versus actually go and post-train, as well as just get Yash's thoughts on a lot of the really interesting things happening in the ecosystem today. All in all, this is a fascinating conversation with a brilliant mind in the space. I hope folks enjoy it as much as I did. Without further ado, here's Yash. Well, thanks so much for coming on the podcast. Thank you for having me, yeah. Been looking forward to this one for a while. Yeah, yeah. No, it's been a long time coming. It's good we can finally do it in person as well. Yes, always better in person in our little closet of a studio here.

1:30Yeah, this is awesome. I mean, it's very cozy, built exactly for this setting. Yeah, definitely not a repurposed library. Purpose built for podcasts. So thanks for having me. Of course. Well, I feel like there's a bunch of places we could start. Where I thought would be most interesting for our listeners is just owning your own intelligence feels like it's in the zeitgeist. I don't know when the tipping point happened three or four months ago, but it is certainly in the conversation. And I feel like there's been a bunch of different versions articulated of the risks of depending on Frontier Labs.

2:02There's been people saying you'll never know when they'll pull the APIs away, when they decide something's a safety risk and they don't give you access. They'll build products and compete with you. If you use their models, they'll train on your data. There's all sorts of different arguments. And I'm curious when you think about like this, you know, the case for this, which of these arguments like resonate for you? Which of them do you think maybe are a little overblown? Yeah. Yeah. So, you know, I – as you know, my background is I actually worked at OPA. Yeah, you've seen it on the inside. And I think like the risks or not even risks, just the things that I think about when it comes to owning your intelligence aren't the ones that are like presumed that the labs are malicious or something like that where they're stealing your data or whatnot.

2:48I think like these companies are made up of great people. There are contractual obligations. There's actual – there's a lot of engineering that goes in protecting data and whatnot. But I do think the idea of owning your intelligence really is about flexibility and control. And so you said like, oh, I don't know when this own your intelligence narrative popped up. I think it really popped up when the open model started getting good. And you could actually go and do things with them. But the control you get, it can actually be a lot deeper than that, right? Like where these models run, how you deploy them, what you optimize them, whether it's cost or latency or specific domains.

3:26And the more control you get, that's what we're talking about when we say own your intelligence. So if you want to own your intelligence and you want to be able to train these models to do tasks on your data, you actually need to have access to the weights. You need to have the infrastructure to post-train them, to inference them, to route between them. You know, I do think there are risks about being super dependent on a single model vendor. Luckily, I think, like, the labs are doing their best to sort of, like, serve as many people as possible. but there are times when models can get pulled from different products.

3:58That's happened actually more than once, particularly in the coding space. And so I think as these labs start to move more into specific verticals, it's not clear whether they'll continue serving companies that they're directly competing against. And so the safest, most durable thing that you can go and do and the advantage you can build up, I think is investing in the full stack to be able to make models that are powering your products, that are intelligent, that are doing the things that you want, and not having like extreme vendor dependence on any one set of models. And it feels like this narrative has really won out.

4:37I mean, I think maybe one of the most interesting manifestations of this is even yesterday, OpenAI announcing like this partnership with Base 10, right, where you can kind of use OpenAI credits. So let's burn them down on open models through Base 10. So I think, first of all, we're big admirers of the Base 10 team. And I think that Tuan and Danny and all of those folks, they're really an incredible company. But yeah, I think this was a clear signal that, like, hey, it's a multi-model future. Customers want choice. They want flexibility. I think on the OpenAI side, it's a clear sort of motion towards, hey, we want to be more of a platform.

5:13We want people to be able to draw down from their commits on different products. We want to be the gateway to enterprises using AI. And I think it was a big win for open source models. A lot of people want to use this stuff. The fact that there is a gateway now that you can go and use a variety of models, proprietary or open source, I think it's a big step towards this multi-model future. I don't know if you noticed this, but the narrative a couple months ago was definitely like, hey, we are going to talk about and sort of promote and analyze the strongest, biggest, baddest model out there. And now it's like much more talking about a Pareto curve.

5:59There's some cost performance tradeoff. You don't need a mega model for every single task. And you actually get more choice and flexibility. I think open models take that to an even more extreme where it's like, hey, you can slide along the Pareto curve by picking different open models of different sizes from different families. But also what our core thesis is is you can move that Pareto curve out. You can train models to be better at particular things, and you can actually get better models for cheaper cost. And so all of these things I think are like sort of the general trend of people wanting more flexibility.

6:35They've gotten more comfortable with deploying AI and production use cases. They've done a lot of exploration. And now it's like, hey, this AI thing is really working. Let's go and optimize it. Do you think the opening iBase 10 thing will work? I mean, it's interesting. Are they in a position to be this neutral partner to enterprises of like, we'll give you whatever model is the best? It's something that we're also excited about participating in. The tricky thing is, right, like it makes a lot of sense for an Amazon, for example, to be a marketplace because you have bedrock and you can run anthropic models, you can run open AI models, you can run open models.

7:13Now, like, is open AI going to take it to the extreme where you can run anthropic models and gem-dye models? I'm not sure. You mentioned kind of this pushing out of the Pareto frontier that post-training enables, and it seems, you know, like there's been a ton of really interesting examples of this. You know, it feels like most notably, a lot of application builders have realized, wow, when I do similar workloads at massive scale, like that inference bill really runs up. or, you know, hey, this experience for my customers is so much better when it's faster. Totally. And so I feel like a lot of the, you know, initial post-training uses I heard about were, like, you know, cost and speed feel like tremendously valuable things.

7:48Totally, yeah. Capabilities, I'm curious if you could talk a little bit about, like, you know, they're certainly pushing out a parade or frontier of you could get for the same size model probably better capabilities. How do you think about, like, the current state of being able to post-train to beat frontier models, like, the best mega models at, like, you know, capabilities? So I think the interesting thing that we're seeing is, you know, there's obviously frontier models that are ahead of the open source models on base capabilities. But the fact that you're able to post-train the open weight models means if you have differentiated data, something proprietary to you that's out of distribution of the models, that's when you can actually push these models to be better on capabilities as well.

8:30Our general take has been like the more out of distribution your data is, the more likely that post training is actually going to give you that capabilities lift. But also, again, it's all about that Pareto curve, right? So if I'm able to take a cheaper model and train it to be parity with the frontier but at much better cost, you know, sometimes you're able to serve more users for that. Your data has to actually be quite differentiated, but most people are actually post-training to basically optimize price performance. On the capability side, I think it's an interesting question of what data remains out of distribution over what period of time, right?

9:09Obviously, I feel like the labs are laser-focused on coding. They're obviously doing work in other domains, but maybe not as focused as they might in a few years from now. I'm curious how you think about, I don't know, five years from now. So, you know, how many businesses that Applied Compute works with feel like, oh, are data still out of distribution from what the models do? Yeah, so I think, like, my honest answer here is that the way we're going to be training models is going to change, right? So we've done, you know, we're in this paradigm of, like, offline RL. You go and you procure high-quality data sets from data vendor companies.

9:49You go and hill climb those specifically, and you get a model that's very spiky in that domain. RL is reinforcement learning. It's learning from experience. I think a lot of the online RL methodologies are actually using production inference to go and improve your models. That is actually going to be how most enterprises codify their judgment expertise by basically the more they use the model, the better that it gets. but generally, the proprietary data that will be out of distribution is things that are produced inside of the company. The very extreme example of this is a shout out to Liam, who used to be by my manager, Liam Fettis.

10:32Extremely out of distribution, right? Because they're making that data. They're actually going in the real world, running experiments, getting reward signal, things like that. But I think in the enterprise, what that looks like is judgment traces. You know, the way one company operates can be very different than the way another company operates because, you know, they have different thresholds for risk. They have a different operating model. They have different expertise, historical data, all that sort of stuff. And that contributes to the decisions that they make. The way we think about out-of-distribution data is maybe like we're putting it in the context of offline RL where it's like there has to be this highly verifiable signal from some math problem or some rubric -based RL problem or something like that.

11:19There are times when you're going to go and optimize an AI system by configuration. So you're going to go and prompt the model or you're going to put things in context. And there are times where it makes a lot more sense to optimize the policy. So how it does judgment, how it does reasoning, and these things are actually like harder to codify. And that's where we'll see in five years from now. I think a lot of companies will basically have systems to basically gather input from experts and then turn that into better policies. So as these models get used, you kind of codify a bunch of the way that an organization makes decisions and you kind of use that to create – to improve models.

11:59Is there like enough data that's generated to do that? I guess it feels like to do a lot of these things requires a large amount of data. Hi, I'm Anne Maris and I'm the producer of Unsupervised Learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership and see you next episode. Is there like enough data that's generated to do that? But I guess it feels like to do a lot of these things requires a large amount of data. I think part of this is – this is like a greenfield space, right?

12:38There's new research happening here, and it's all about data efficiency and learning from sparse rewards. So we had pre-training. It's like not that data efficient but very generalizable. Then we had this like supervised fine-tuning. Pun on, unsupervised learning is the name of the podcast. But supervised fine-tuning, it's more data-efficient but still not very good. Then there's RL with verifiable rewards where you actually trade data quantity for quality with these replayable environments and some sort of verifier. It's still pretty bad, right? It's still not very data-efficient. The answer to your question is there will be more data-efficient methods of training.

13:19And that's what a lot of these newer methodologies in online RL or on policy self-distillation or some of these type of techniques or SAO, these types of things that people are pushing on. And I basically think what's blocking continual learning is basically extremely data-efficient training from sparse rewards, which is an unsolved problem. Yeah. How do you think about as a CEO the extent to which you should be working on that problem within the applied compute context? or it's like, hey, just understand the enterprise problems really well. And as that research progresses, you can kind of take pieces.

13:56Yeah, so we're actually doing this today with customers, this sort of online training where we're actually able to take advantage of large inference deployments where people are serving tens of thousands, hundreds of thousands of users, and then use that production data to go and improve the models. We aren't married to any specific type of training. RL and these online methodologies are the best things right now, but these will continue to evolve and get better. That's why a large portion of our team is actually a research systems team trying a lot of new methodologies in production with real customers.

14:30There's a ton of potential on updating models and training models. And also, I'm sure anyone that's played around with these tools has realized, like, you know, I don't know, if you do some really great prompt engineering or create some skills, or like, there's other ways to like take knowledge. And I think there's maybe a feeling sometimes that, you know, researchers can be a little bit of like a hammer looking for a nail. Like, oh, we can do it with the model. There's other ways to do it. How confident are you that like the long-term solution of this involves like many, many different kinds of models of an organization versus like stitching together things in other ways?

14:58Yeah, I think we are pretty confident that the future looks like a non-static set of weights. I think that is something that we deeply believe in. there is going to be some sort of feedback loop from the real world back into the model. Does that need to be done for every single task? There's been some good examples of that in the coding world. But what do you feel is above the line of where that feedback loop makes sense and maybe what's below it? Basically, things that change with temporality. You often want some sort of closed loop where new information is making its way back into the model.

15:38But also I just think there's this expanding umbrella of intelligence that you're getting from the base models out of the box. And at some point, that is going to be sufficient for a lot of these manual process-oriented tasks. The things that will require sort of iterative training are the things that are more judgment-oriented. You know, are there, like to the extent you can talk about it, examples that come to mind of like ways, you know, of kind of taking this like real world data and using it to kind of improve? Yeah, yeah. So basically, one of the examples of customer working with today is they essentially have some sort of assistant product that a lot of users use.

16:22They ask this assistant to go and accomplish different tasks and then give feedback on what the model did wrong or right. And so basically, it's pretty impractical to go and build some sort of giant skills file or context or something like that. So we're actually able to bias the model based off of the direct user feedback using some of these online training methodologies. And we're able to see real upticks in user satisfaction, completion rates, decreased tool call failures, things like that. The main question and answer is, what's the TCO of doing this? How expensive is it to post-train? And that's a lot of the stuff that we're going and building is how do we actually decrease the cost to go and RL a model?

17:08How can we actually do this automatically and online without you having to go and build hand-tuned RL environments that are fully replayable and things like that? And so what we see happening is the cost to post-train and actually adjust the way it is falling. You're still better off going and doing a bunch of context optimization and harness optimization. But once you reach a point where you've sort of squeezed a lot out of the harness that you've built and the tools that you've made, you actually can still continue to squeeze a lot out of optimizing the policy to use those tools better. So some of the companies that we work with, they have really specialized harnesses with custom tools and things like that.

17:54We're working with a company in chips. They've built, like, a bunch of software, actually, that general models don't really know how to use well to go and do, like, you know, ETL verification and things like that. So it turns out when you apply this, like, you know, Goldilocks algorithm of RL, you can actually push the models to, like, a lot better performance. You talked about kind of, like, this effort to, like, lower the TCO and, you know, enable, like, more continuous training. What are the hardest problems to solve there? And, like, you know, where are we on that journey? Yeah, the hardest problems to solve with RL and infrastructure.

18:29Yeah, to be able to bring the TCO down to the point where – and then productize it even more. My co-founder, Lyndon, who's our chief architect, he is like the wizard with this. He's one of the smartest people I've met. And you'd be surprised how underutilized GPUs are. Like if you were to also just aggregate things globally. Basically, I would say there is a lot more that can be squeezed out of the available hardware to actually go and lower these costs quite drastically. You can invest a lot in algorithms that are a lot more data efficient, so you need to train for less long and use less compute.

19:13And so I would say the system side and the algorithm side are the places where you can squeeze the most right now. In terms of the data available and things like that, there's synthetic data generation techniques to be like, hey, can we go and take some amount of tokens and explode it into something much larger and pull learning signal out of that? But I would honestly say that most of the cost optimization and cutting down costs there is going to happen on the algo side and the system side. Satya came out and said that there should be as many models in the world as there are firms in the world.

19:53Yeah. Which obviously was a good selling point for you all. Yeah. Is that a vision that resonates with you? I mean how much do you believe in this vision that like every single company is like its own special snowflake with its own unique judgment that requires like its own model? Yeah, I think in principle, whether this means that every company needs to pre-train a model or mid-train a model of their own, I don't necessarily think that's the case. But I do think there are closed-loop feedback systems inside of each company that is worth capturing. Right now, there's just a lot of low-hanging fruit.

20:31So people are starting there with harness optimization and context optimization. But the idea that different companies are unique and they operate differently and they have different judgment structures and are not the same, that we fundamentally agree with. And that's kind of the thesis of our whole company is that if everybody is using the same model, like great models sort of set the floor for everybody. But then how you go and optimize those models and build really amazing AI systems, that's what's going to set the ceiling. So there actually is a difference in the products that people can build both on the harness and context engineering that they do but also the models that they put behind those products.

21:16Certainly, you take one extreme like a pharma company. Pharma companies have such different data from each other. It makes so much sense that they focus on one disease area, generate a lot of experimental data. Of course, they should have their own model. If you take banks, it's like how different is one bank from another bank? And, like, certainly maybe there's little judgment, you know, calls that are different. But I wonder, like, you know, if a frontier lab is post-training on a bunch of, like, really good, you know, general finance data. How much uniquely unique things does a frontier lab?

21:43Yeah, I think there are certainly things in the market that will benefit from consolidation. I mean, this is why you have roll-ups, right? Like, you basically, you know, to take, like, accounting firms or something like that. Like, they're fragmented just by the nature of the business, I think. It's like you have an accountant and you have a client and they're a one-to-one match or something like that. So it makes sense. Hey, let's go and take the menial sort of labor out of doing the work of an accountant and put that into a model. Those types of things I think you will see sort of consolidation.

22:17When it comes to how different companies operate like banks, though, I think there is a whole litany of decisions and sort of like trade-offs that each company makes that is different than the other. So they're less of like, I guess, like a commodity where you can use one or the other. and that's kind of why you have like such a wide distribution of banks that are doing well and banks that are doing poorly, I think. There actually is like room to compete. It would be fascinating to see, right? I mean like, you know, essentially we'll kind of learn the extent to which a lot of these industries, the companies are actually particularly different or have different ways of doing things.

23:02Yeah. By the extent to which like having your own model is actually beneficial relative to using whatever the off-the-shelf average, you know, Mercure data labelers can do. And I think the way, you know, one of the ways that this actually manifests is not always capabilities, right? One of the reasons why we feel very excited about what we're building right now is we are in a supply crunch, right? Like there is like limited compute. Everyone's kind of feeling this. So the more you can squeeze out of your models and the more you can squeeze out of the compute that you actually have available to you, the more stuff you can do, right?

23:37So basically, that's why we think having the infrastructure to go and optimize models and make them more efficient, even if you are pushing the Pareto curve out and you're not necessarily pushing the frontier on capabilities, that delta is some sort of opportunity or advantage. If I can do something 10x cheaper than my competitor, that is differentiation, and that is something that I can actually win against them out on the competitive landscape. I guess maybe zooming out and talking about people that are at the precipice of deciding whether they should embark on their own models. I mean obviously you've talked about you should try a lot of the harness optimization stuff first and a lot of other things.

24:18But I think a lot of folks are trying to figure out how well does RL work in these nonverifiable domains. How do you kind of characterize where we are today? Yeah, so what I would say is like, you know, RL started or like the way these projects started and the internal models that were being trained were on highly verifiable domains like math and coding. Partially because they're highly verifiable, partially because everyone at the labs like loves math. But it was more of a former. I think it is surprisingly easy to hill climb on non-verifiable domains if you turn them into some proxy verifiable domain.

24:57So rubric-based RL and this sort of stuff actually works quite well. So I would say the state where we're at right now is if it's not something that has one definitive answer, basically giving it an expert answer and grading it against that serves as quite a good proxy. So that's actually why different companies have different experts. And so optimizing to two different groups and two different companies can actually give you very different models. So to keep all your ex-employees off of the data labeling companies. Exactly. Yeah, yeah. No, I think it's actually what we have with RL is we essentially have a hill climbing machine.

25:36Um, the hardest part is actually defining the hill to climb, which is why evals, I think, is like a really important thing for companies to, A, focus on building and, B, actually like safeguard pretty, pretty carefully. If you are thinking about competition and stuff like that, if you look at every public benchmark, it's going to get benchmarked, right? So, um, if you have things that are very particular to how your business operates, and you want to make your models really good at that, it's actually to your advantage not to tell everybody else what good and bad looks like so people can go and make their things really good at that.

26:15So that's why I think, yeah, it is really important to actually keep your evals and employees and all that sort of stuff. Your employees are not fungible. You would not be comfortable with them going to another company and doing work there. Same thing with these models. It's like there's this tension if you're a leading app company where obviously you want to just expand the set of capabilities you can deliver to your customers. And so going to labs and telling them your evals is helpful. And then the other side, it's like a lot of these hard-earned insights. Totally. And it's a tough position, which is why I think investing in open model infrastructure, investing in a multi-model future is something absolutely every company should be doing.

26:56because, yeah, you're kind of caught between a rock and a hard place. Yeah. Yeah. I guess, you know, I would love to shift to like just a black and pew like the company. Yeah. And I'm wondering, I'm sure that like there is no shortage of enterprises that would be like, it would be pretty sweet to work with you guys and like, you know, get you rolling up your sleeves and helping whatever the set of problems is. How do you like qualify business and figure out, you know, who you work with and who you don't at this stage? Great question. So we basically see post-training and actually training these custom models as the differentiation and the part that makes the business sticky and actually provides more value on top of just the off-the-shelf models.

27:39But really inference is actually where most of the value capture happens. If you look at the dollar spent on models, the vast majority goes to actually serving and using the models than training the models by themselves. So when we work with customers, we typically go after very high value use cases. And that's either because the capability increase in the model that we're going to provide is extremely valuable. So some of these companies that you were describing before, like a pharma company or a cybersecurity company or chip company or something like that. The other is if the inference workload is really big.

28:16So basically some sort of efficiency or optimization gain or quality improvement distributed over a very large user base or billions or trillions of tokens, that's actually super valuable as well. So we actually look for, like, large inference workloads. These are typically the best customers for us or, you know, very out of distribution data where people are wanting to train on things that are kind of only unique to them. You kind of mentioned the inference side of things. Obviously, it feels like the temptation for, like, any infrastructure company these days is, like, you know, having an inference business is pretty nice.

28:54Yeah. I feel like there's different schools of thought on this. Like, how difficult of a product is that to build, like a really good inference engine? So I think like, you know, maybe taking a step back, like why we think – so first of all, we think inference is going to be like one of the biggest markets in the world, if not the biggest. Tons of people have said this. It's not like a very unique take. I mean it's kind of obvious if you believe in AI doing a lot of, you know, knowledge. Not going to get a lot of training dollars if there's not – Exactly. Exactly. And so what our view is is that the most post-training actually wins inference.

29:31So if you are – basically what the trend we've been seeing is like companies will start off with frontier models. They'll do a lot of experimentation. Then their workload will become mature. They'll be scaling that workload. Then they'll think about open models and how to sort of cost-optimize, basically push the Pareto curve out. And the interesting thing is that the most at-scale workload, so the largest workloads where you're spending the most money, is actually where you're going to want to go and post-train first, right? Because that's actually where the most gains are to be had, and it's kind of power law between all the other use cases.

30:12And the interesting thing you can do is you can both co-optimize your inference and your training. So basically the way we train models actually influences how we go and set up inference deployments for customers. If it's a heavy tool-using model and we train it to do a bunch of tool call parallelization and that kind of stuff. Can you say more about that? What does that actually mean in practice? If it was designed that way, how does that influence the inference choices you make? Yeah, yeah. So it's a great question. So basically, if we are doing a long-horizon tool-using agent, the way we set up the inference deployment or the inference engine, is different than if we were doing like a decode-heavy sort of workload.

30:58And you can sort of set up these disaggregated setups where you're using multiple different types of chips for pre-fill and decode. And so having influence on how the models are trained actually can influence how we want to go and set up the inference deployments for very large-scale workloads. It's interesting. I'm really curious to dig into this because I feel like there's been this question for a while of how much does creating the model help you run better inference? And it originally showed up in, like, a lot of the original open source model pitches, right, where the idea was, hey, we train these models so we're able to run, like, you know, inference way more effectively on them.

31:31And then, like, the way it played out, at least in the early days, was actually, you know, someone like Fireworks would be like, thank you for training a great open source model. Now we'll be the ones to run that, like, as efficiently as possible. Yeah, so I think what's happened is basically the kernels that you use for training and inference, you obviously want matches between them. We do a lot of work on the training side to go and optimize how fast we're able to train these things, how efficiently we're able to train these things. And a lot of that work translates over to the inference side.

32:06I think also one other really interesting thing to point out, this is kind of unrelated to what we were just talking about. One thing I think is really often overlooked is, you know, you can do a bunch of inference optimization and things like that if you want to make your inference, say you want to cut your inference bill by 10%. You could do a bunch of inference optimization costs by 10%. Or I think these models are actually quite token inefficient for a lot of things. So what we've been doing with customers is say, hey, we can actually cut your token bill by making your model 10 % more token efficient, preserving the same performance against some sort of evil that you care about.

32:44And so I think one of the things that we've been able to tell customers is like, hey, these things are two sides of the same coin. You can actually influence both of them if you do it with us, which is why we're trying to be the centralized provider for both training and inference. It'll be interesting to see the extent to which doing the training yourself gives you, like, even more insight into the inference workloads. And I think, like, you know, back to your original question about, like, inference offerings and how easy they are to spin up. I think, like, what we've seen, right, is there's – actually the main thing to have is capacity, right?

33:24So, you know, you need the actual GPUs to be able to go and serve the models and things like that. And the open source projects like VLM and SGLing are great starting points. In fact, I think almost all of the inference clouds have sunset a lot of their custom inference engines in preference for taking one of these inference engines and doing the workload tuning and performance tuning and maybe having a couple custom kernels in there. So I think Dylan Patel was on DoorCash's podcast, and he was actually talking about how easy it is to basically, if you have some GPUs, you put a VLM on it and you throw it up on an open router, you kind of have a production inference offering.

34:09Now, I don't want to trivialize it too much. I think there is a lot of optimization and hard work that goes into making these inference deployments a lot more efficient, faster token speeds. And that matters, especially as you hit scale, right? because 1 % of savings across tons and tons of chips actually adds up. But in terms of actually getting production quality model serving going, it's definitely the easier thing to do out of training and inference. It's interesting because I feel like as an infrastructure CEO, it feels like there's a bunch of companies starting in different places and there's this desire or this belief that over time, a lot of them will go pretty horizontal and intrude on each other in all sorts of ways.

34:55And I imagine you both have to figure out the sequencing that makes sense to do here as well as figuring out the red line. Do you want to draw? Okay, we want to do lots of things, but we don't want to do that. How has that evolved over the course of the business, and how do you think about it today? Yeah, it's a great question. Our large-scale vision, what we're actually building is we think there's a new AI hyperscaler to be built. In the cloud era, you have AWS, GCP, and Azure. They took CPU compute. They built really awesome flexible compute products on top of it for networking and storage and databases and data lakes and stuff for building applications, all that.

35:39Now you have this new compute structure, GPUs. You have this weird thing, this collection of weights and parameters called models. It turns out you can serve them with an inference engine, which is the first piece of software that has achieved real stickiness inside of the AI software stack on top, and you can serve these valuable intelligent tokens. Our view is that there's a ton of stuff in that software layer, actually, besides just inference. Our big bet was, hey, post-training is a massively underlooked part of the stack. How do I take GPU compute and turn that into more intelligent tokens?

36:17That's training. How do I serve those tokens? That's inference. How do I pick the right tokens to serve? That's routing. But there's also a ton of stuff like security monitoring and identity, agent sandboxing, all this sort of stuff. So when we think about what we're building, we actually want to be very horizontal. But we actually want to start, we kind of think about things as an inverted pyramid. There's things that very few people can do at the bottom of the pyramid and things a lot of people can do at the top of the pyramid. So at the very bottom of the pyramid, that's where we think model training is.

36:51And then above that is inference. Above that is routing. Above that is context and harness and all that sort of stuff. And so our strategy has really been focused on the things that are extremely high leverage and extremely high ROI that are very hard to do, that you need to build super robust infrastructure. You need a GPU fleet that you actually own and operate and all this sort of stuff. And so our goal is to, as an infrastructure company, widen the offering as we mature. And we're 16 months old right now. So the first step was going from training into inference. But we have a lot of exciting things on the roadmap and things that we want to go and build.

37:33What's so interesting is that I imagine when like 80 % of your pipeline approaches you, you're like, this could probably be solved by like a good harness and like some data work or something. And so what do you end up doing on the harness optimization side? I imagine a lot of folks are thinking through this right now. It's a great question. I think we are really going for focus right now. So we actually, the customers we work with have mature AI products. They have already built amazing product teams, and they are investing in creating really great harnesses. And so our current operating model has actually been we're not focused on the harness layer, the context layer.

38:12We're focused on the model there, and we don't necessarily do the zero-to-one agent builds. We optimize the models behind mature products. Makes sense. Yeah. And I guess in terms of like the building the RL environments today, I'm sure our listeners would love to hear like what is actually – you can even just take an illustrative example. Like what is actually involved in like building one of these? And how do you think about like – I'm sure there's 500 annoying parts of it, like the parts that need to get better to – or that you hope get better. The nice thing is, like I said before, RL is just like hill climbing machine, right?

38:44So it actually is the same way you're going to go and build your training data is actually the same way you're building your eval. So hopefully that can help folks kind of conceptualize what goes into building great eval is like, hey, you have to have the right task and you have to have the right environment and you have to have the right verifier, right? Those are kind of the three main things when you're going to do RL. What is the problem? What is the scope of tools and services that the model has access to? And then how am I going to judge whether it's good or bad? So take, for example, we did this Harvey Review Table custom model that we trained.

39:28We put out a case study about it. But for that, we basically had worked with the Harvey team. We're able to collect a bunch of expert rubrics and answers, also supplemented those with synthetic data, and gave the model access to some tools. We're able to say, hey, here's the task that you're supposed to go extract all this information out, graded it against the expert answers with rubric-based RL. and that's how we were able to create the task. The environment tier was the tools that the model had access to in the document base and then the verifier was the grading against the expert answers. Is the hardest part of that just like actually getting people to create these rubrics or like, you know, create, you know, what are realistic documents that would live in these things?

40:14Yeah, so we work really closely with all of our customers because they are the experts in the domain, right? We really come in with the infrastructure and the training team and the inference production serving and stuff. So this was pretty collaborative. We oftentimes can do a lot of work on the synthetic side. If it's like some, say you can take some sort of document and back translate it into question and answer pairs, for example. That's a very concrete example. We can do that stuff synthetically, but when it involves real experts and real lawyers, we actually want to work with the teams directly.

40:53Yeah, and I'm sure like the entire thesis of this seems to be like that type of work is going to be way higher quality than like whatever lawyers a data labeling company gets on and tries to like create. I think probably distant memory at this point, but like OpenAI shutting off cursor. Do you expect that kind of stuff? And Windsurf before, right? Of course, yeah. Anthropic with Windsurf, yeah. Do you expect that kind of stuff to happen more or like? I mean it happened twice. I think competitive dynamics are real and like their incentives are real. So especially as companies start to move more into the application layer and build their own competitive products with their largest consumers of their models, I wouldn't be surprised if it happened more.

41:33I always like to have a quick fire round where I basically shove in like everything that I – in an unstructured way that I haven't gotten to before. And so maybe to start, is RL going to generalize? In some ways, it is generalizing, like models' abilities to do error recovery or use tools and subagents and things like that. But I think if you actually empirically, you can run these experiments, you can train on a math data set and see how much that translates to something in the legal domain or something like that. the transfer is not really there, certainly not to the extent of the generalization that pre-training has.

42:20So I think it will bleed into adjacent areas, like intelligence is some sort of sphere or circle. It's got jagged points on it. The points that are closer together, you'll see more generalization there, but certainly does not have the same generalization as pre-training or something like that. Yeah. Yeah. Um, what have you changed your mind on an AI in the last year? Yeah. I mean, I guess one thing that I didn't, maybe I didn't change my mind on it, but I didn't fully appreciate is how real, uh, Jevon's paradox is. Like ways that are like a particular thing you saw that. Yeah. Like, like, you know, I think, um, one, one interesting thing is, uh, And we are in, like, basically, like, if you look at some of the open router statistics, whenever there's a price cut for a model, usage just spikes.

43:17So I really think we're moving to a world where things are, like, extremely cost optimized. We're feeling the pressure, like, everyone's feeling the pressure right now to, like, make things as efficient as possible, lower the AI bill. I don't think I fully appreciated how much of a driving factor that would be over something like capabilities. We are reaching a point where I would say most of the open source models can kind of do everything that you might want them to do. There are obviously a lot of frontier things, and I don't want that to be a blanket statement. Frontier models are amazing, but I didn't expect how much people would care about really cost-optimizing and using the right model for the right task.

44:08Yeah. Which is interesting because obviously for some of these tasks, people were using frontier models because you got a lot of value. Yeah. But if there is a way to do it much cheaper and the use case... Yeah, there's some marginal return of value for like, and at some point, people are going to do the experimentation, try a bunch of things, and then consolidate on the right model. When something like JEV comes out, what's that like at Applied Computing? How have you guys thought about that? I think it's a super cool model. I basically think there are probably going to be a bunch of things like that coming out.

44:40It enables a whole set of new use cases. I mean, it's in the name. That's why they called it JEV, right? Things that are cheap, fast, designed for a particular set of tasks. That's how we've been thinking about a lot of things that we're building in Applied Compute. So models like Jev, we're working on, hey, what does this look like to integrate into the training process as sort of like massive billion token scale classification on rollouts or things that we couldn't do before that were cost prohibitive. So yeah, when things like that come out, I think it's super exciting because it basically just expands the amount of things that we can go and build.

45:21and it's some sort of matrix, right? So it's highly multiplicative. Like these things actually end up stacking. Jev is like much, I would say it's like much less of a replacement for stuff rather than an enabler for things that we weren't able to do on both training and on like inference for, say for observability or something like that. Yeah, just because like at that kind of like cost level. Yeah, at that cost or the price point, It's basically like enables like a whole new set of things that we can actually do that we couldn't have done before even with like the smallest, cheapest Quen model or something like that.

46:00Yeah. Is there like a visceral example for you of that that you've seen so far? So we actually put this out in a blog post. We basically used Jev to do essentially like massive trace analysis for online training. Because I imagine you're on this like exponential slope of growth. So how in the world do you budget for the amount of compute you want? Because you're probably entering into some of these. So I think the way we think about it is we're trying to actually make a very sticky, differentiated business. So we aren't going after every single opportunity that we could that, oh, someone needs a reference or something like that.

46:42We are very sort of convinced of this thesis that people will use custom models. They will want to invest in the infrastructure that we use to post-train. So we actually are, instead of looking at the total demand for everything, we actually are like, what are the most strategic companies that we can go and work with that we can grow with them? and they're also the right folks to basically go and prove out this thesis that custom models are much stickier and valuable and the ROI is persistent besides just some sort of off-the-shelf model. Two things I wanted to hit on that I've heard you say. One is you said something that I thought was really interesting is that the best engineers you know all learn to code before AI existed.

47:30I want you to expand on that a little bit. and kind of like there's all these 16-year-olds now that are coding and kind of like native to these – I've only learned to code with these AI tools. How do you think about like advising folks? Yeah, no, it's a great question. So to be clear, I'm not against AI coding. I'd imagine you use a fair amount of AI coding tools. Yeah, exactly. And also when I was at OpenAI, the main thing I worked on was Codex. That was actually like the thing I was spending all my time on. So I'm like very excited about AI coding. I think there's a difference between delegating your thinking to coding agents versus using them to go and execute on some sort of plan and design that you've thought of.

48:13So what I would say is I actually think the best thing for people to focus on is system design. Like, funny coming from like an infra company or something like that, but the types of problems you face in like building scalable systems really force you to think about the things that you're doing and why you're designing and structuring things that way. I think the models are getting really good, so you don't need to know the exact syntax and all this sort of stuff. It's much more important to get the frameworks and design right than any of the actual coding. So yeah, I definitely am a big fan of AI coding.

48:58It's just you can't delegate your thinking away and basically rely on it as a crutch, because then you get these massive slop code bases that you can't actually explain. So probably the best thing is learn how to explain your code. That's what we interview for is we'll let someone go ham when they're on their onsite with us, but then we'll ask a ton of questions about why did you build this this way, what were the tradeoffs you considered. And if they said, oh, Claude did it or Codex did it, that's not something that we're pretty excited about. Interesting. Obviously, I feel like everyone's trying to learn from the most AI-native companies on how they do things.

49:38How have you thought about, like, so obviously that's interviewing on the engineering side. You know, I'm sure you hire other roles, and I'm sure you want people to be relatively AI native and fluent. Like, how have you thought about, you know, interviewing or changing processes around that for other types of roles outside of engineering? Oh, great question. Yeah, I think, like, you know, when we interview folks, it's kind of the same principles apply. So for finance roles or biz dev roles or things like that, people will often use AI tools to go and build a financial model or something like that.

50:16And if they can't explain why they built the model that way, that's basically what we're looking for. Really, it's like our view is people should use whatever tools that are available to them, but they should have a complete understanding of everything those tools went and did for them. So we do much more of like our interview process is unconstrained, but then very particular in like how we audit or essentially assess understanding of the outputs that were produced. That makes sense. Makes a ton of sense. I guess you gave this kind of lecture at Stanford where you said, I think they made you say something you were short, and you said you were short the general data vendors, right?

50:56Like Mercor and Surge. Curious why that is and where you might be wrong. Yeah. No, I guess I'm not short the companies themselves. I actually think they're great companies. We serve a lot of them as customers. They're amazing companies. Basically, my main point there is basically a lot of data is becoming more and more synthetically generated. So you can use these smart models to go and create the types of RL. In fact, actually, a lot of RL environment companies end up using the models to go and create data that actually feeds back into the model. So it's a little bit of a circular motion there.

51:38sort of the big point was kind of like paradigms change all the time. So for data companies, I think the main thing to get really good at is pivoting, like going from each paradigm to the next. So we used to be in SFT land, preference tuning, then like RLHF, human feedback, then RLVR. Now the big thing is like robotics data. So I think my point there actually was less like, hey, these aren't good companies. I think they're great companies. It's just that the particular RL and data creation company is going to move around a lot. It's going to be quite fluid and the paradigm will change. The research paradigm will change.

52:27Among the RL environment companies today, what makes an RL environment company better than another? I think it really is the human part. People are able to get to it. Yeah, exactly. The expert networks, so like Mercore and Handshake and these types of companies, they have incredible expert networks. Actually, when I was at OpenAI, we worked with Mercore. I think they obviously still do, but we worked with them very closely for some of these first agent products like Deep Research and Codex. and that is where you are getting the best of the best in the world for coding or math or stuff like that.

53:09There's still signal that needs to flow into these models so models are a function of data. The higher quality of the data, the better your model. That's what I would say probably differentiates is the ability to very quickly run a campaign across a wide network and basically narrow that down to the highest quality data points because when you're going in training, you're not actually using that much data, right? You're using a few hundred data points, all of those each, or a few hundred to a couple thousand per task or something like that. And so if your quality assurance is pretty low, that has a material effect on the models, like the final.

53:50It feels like a lot of this, like the power companies specific models will be determined by this delta between the, like, ex-bankers that Mercor can recruit and, like, figure out what the tasks are versus, like, the, you know, bankers within J.P. Morgan and, like, the stuff, the tasks they have and, like, the evals they create. And, like, I don't know what the gap ends up being between those two. Yeah. Well, I think, like, when banker, or, like, take, for example, your example. When bankers leave, they don't get access to, like, a lot of the proprietary data that's been built up over time. Like they're obviously doing lots of, you know, reasoning and judgment and stuff like that.

54:27But it's also very heavily informed by the data that they look at and that they sort of do their work on. But yeah, I think it's like a real source of like alpha permeating the boundary of an enterprise. I mean it's always been the case, right? Like that's why people have like non-competes and things like that, right? It almost seems like, yeah, like, you know, obviously you've seen this trend of, you know, a company goes under and people buy it for the data. It's like if I was a bank and a bank went under, I would not want like one of the labs to go acquire like whatever. I don't know what data they can actually train on.

54:59No, exactly. Right. I think it's like – and that is almost a sort of argument for why these companies are valuable. Right? Like that data is valuable. So that company is valuable. And therefore, the more you can keep things sort of like proprietary to you, the better. Yeah. And it'll be so interesting to see like what that boundary ends up being. It's like clearly chimps, you know, biology, like that feels like it's hard to get out there. Whether like the labs eventually figure out kind of the basics of running a bank or something else will be – Yeah, well, I think there's just a lot of – I mean running a bank is like – it's like a thousand-liter cake, right?

55:37I think, you know, unclear whether a single model or an ASI model or something like that is just going to be able to do that out of the box. I think people are important. They've built up years of expertise doing these jobs. Codifying those into models is actually quite difficult. But yeah, I think there will be large portions of roles at different companies that will get automated away for sure. Yeah, super interesting. Well, I was thinking about where to end it. One thing I thought would be interesting is just like because you're doing RL all day, a lot of things like models do some crazy reward hacking sometimes when you set these things up.

56:18I'm curious, like, what's the best example you've seen? You know, you don't need to mention the specific customer, but, like, you know, in some deployment where maybe some run didn't go quite as you expected. Yeah, yeah, yeah. No, no. I mean, like, there's, even from OpenAI days, there's, like, all these interesting ways that these models are really clever. They figure out, it's like water. They find the path of least resistance. And so I think for us, with some of the companies we work with around coding, for example, we've been able to find some exploits in packages that the model just used to get easy reward on the tasks.

57:05I think there are much worse things that are public out there. I think that's honestly why we're seeing there's so much interest from cyber companies in training RL models and things like that because it's like almost the perfect problem to go and just apply optimization pressure to because these models are really great coding priors. And yeah, basically the offensive cyber capabilities are quite good. Well, it's been a fascinating conversation. I really appreciate you coming on the podcast and taking the time. Thanks for having me. This is great. It's been awesome. I'm Jacob Efron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world.

57:50As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project, in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.