Why RL Won — Kyle Corbitt, OpenPipe (acq. CoreWeave)

25:03You also need to simulate the user and i mean you can do the naive thing and just say oh we're going to have a separate llm that you know with a system prompt that is like the user simulator and we do that but it's like okay but like the breadth of ways a user might respond there's like a lot more diversity in that than the actual diversity you'll get in practice when you have this like simulated user and so then it's like okay well is this environment close enough to how a real user would interact that like you know if a user says something different that it's going to know what to do and the answer in many cases is no if you're just purely training on kind of like an llm user simulator, it's going to have its own idea of like what the correct way to answer is.

25:39And the breadth of like a way a human might respond in this situation is wider and your agent just may not be able to deal with that. Do you feel like it's hard to build the simulations as a company that needs to build the product that lets everybody do it? Or do you feel like even for the individual companies that own the code base that are like domain experts in their own product, it's still just like a very hard infrastructure problem? I think it's still very hard. You know, Like ideally all companies should have this anyway, because they're getting, you know, if you're doing end-to-end testing, like theoretically, if you're following best practices, you would have one of those set up.

26:10When we talk to enterprises almost universally, that's like not something that really exists. So there are some startups, like there's some companies we've talked to that do have it and we can just like use that. But it's a very, very small number that actually have an environment like that. And I think it's hard to do. And like there's lots of like weird bugs that don't show up in an environment like that. And even if they do have a testing environment, they don't have it populated with full realistic data, which is also important so that it understands how to interact. So I think in practice, it's hard in both cases.

26:42Maybe it's easier for the company, but at the same time, depending on the quality of the company's engineers, it might not be easy for them either. Yeah. How do you classify the types of environments? So you have formal environments like a compiler. You can put in there. It's like you don't need to do any work. They just work. then you have this kind of like rl environment startups in a way that are building a bank environment they're building these things that are not digital twins or whatever term of like the actual environments but they're like close to it and then on top of it you have helping people trying to build the exact replica of their thing there's obviously value in like the formally verified ones we verified that do you think there's value in this like rl environment startups that are building like somewhat generic but test specific environments and then if none of those work then what do we do instead of grpo i guess the question yeah i suspect there is value in that you know i think the you know the folks buying those environments and training on them in the big labs would have the best knowledge on how well they work i think they probably work okay i think they probably also are like you know and we'll see maybe with the next generation of models release like how well they transfer i would say so far um it seems like they don't train well enough like if you if you use um you know open ai's agent interface it's like okay or if you use the computer use products that everybody's putting out they're like okay but like not reliable enough to like actually like let go do something interesting unsupervised in the world and i think if the eight you know if the environments they were training in were high enough fidelity then they would be good enough in the same way that like coding agents can go much further because i think that in that case we do have environments that are much higher fidelity because it's a much simpler environment a lot of ways it's like it's a code base it's like maybe running a web browser like it's it's it's much easier to capture the full realistic environment in that context for those who are interested when you make a reference to our environment startups selling to the big labs they're selling it for a lot of money yeah like at least seven figures right i i don't know understanding yeah i don't know i'm not a buyer please please like drop data points because like people who are not in silicon valley don't know this and like it's like probably the current thing in vc which is just our environment startups um anyway a lot of them there's like 20 of them apparently yeah but it's like a small number i know that yeah all the labs are buying ad hoc but in a way it's almost like they don't even care it's not a product it's like They're basically paying the company to build an environment head off for them.

29:18It's a very services business. Services business. But I mean, if you're spending like a billion dollars in a training run. Yeah, but you can specialize in like, we are the one that does e-commerce. We are the e-commerce experts. So come to us for e-commerce. Go to the other guys for social media. Go to the other guys for like, I don't know. But I'm curious, your take is like, how do you need to get the data out to make it fit in your training run? Especially when you get to like these larger labs. I think they're like very sophisticated post-training pipelines. And I don't know if there's like a way to just build a company where it's like you just send them a CSV of like data.

29:51It needs to be very integrated in it. But I'm curious what you've seen working with customers, too. So for RL, like the whole way this works is, you know, it has to sort of be getting feedback from the real environment. So I don't I don't see a world where it's as simple as like, hey, you can you know, there's like a CSV type approach. I guess you could code anything as a CSV. But if you try hard enough for our old to work, you have to be looking at real runs, ideally of your actual agent in its current state across within an environment as real as possible. So you have to look at actually and like the data form, it's like actually super simple.

30:28Like it's just like basically a list of, you know, like chat completion messages. It's effectively whatever. Two calls. Yeah, exactly. Yeah. It's whatever your agent will be seeing and doing when it's running. So getting the data is not hard, but what's hard is like when you're doing one of these runs and your agent makes a tool call, okay, now that tool call has to connect, you know, somehow it's got to get data back from something and that data has to look like it will look in real usage. So setting up that whole part of the system is the challenge. and then for just a reference job for more people web arena is my first instance of this kind of thing where you literally have a docker container that has like a clone of reddit a clone of wikipedia clone of gitlab clone of cms and a clone of a e-commerce place and i think since then there's like mind to web maybe i don't know if there's any other large well-known academic environments where people are basically using these as benchmarks but probably also it's pretty useful for trading.

31:22So if you want to check out those things, you can definitely check there. I think the question for you is, as someone who bet on SFT, then you bet on RLFT, and then now you see these guys making a lot of money. Why didn't you go there? It seems to me like that definitely is a services-heavy business at the moment, as it's presently constituted. I'm sure that these companies are all developing different kinds of secret sauce on how to do this more quickly. So that's part of it. I don't particularly enjoy services businesses. But I also kind of feel like we will move towards a world where either the big labs like it's one of those businesses where like the only customers right now are like whatever four big maybe maybe maybe six big labs that like you know are training these models on environments and i don't think i'm a little right what's the tab yeah um but you know like look you can say the same about scale ai and and all of their competitors that are like you know many billion dollar companies that have basically the exact same customer set.

32:18So, yeah. It may work out. Yeah. I don't know if you want to do a small shameless plug for Varys. Oh, yeah. I mean, so Varys, one of our portfolio companies, they work with the people building the agents, not with the model, on their internal tool call loop. So they can observe all the internal traces and build the data to then have an open pipe do the RFT on the thing. I think in the enterprise, we've seen a lot of that, especially for chatbots. it's like the less sexy use case but like their work with a lot of financial services company where their customers go in there and say what's my balance like when did i do this transaction and those are all tool calls you know and they need a way to test and improve that behavior and the models haven't got in that much better because these tools are like badly documented they're like badly named i think that's kind of like the the problem with a lot of the agent builders that are not ai native companies it's like they just put this like very generic tools in the thing and And then they expect it to work like magic.

33:15And the simulations kind of help them also have the usual compliance things. It's like before shipping this, we tested that it doesn't give financial advice. We tested that, you know, there's all these different things. So I'm curious to see how much the companies generalize. You know, I think like various has a lot of success in like highly regulated environments because of different requirements. But I'm curious if you have a different way to segment the market of like when you think about RL, There's like environments that are like low stakes. There's like environment that are like high stakes.

33:47There's environment that have implicit rules that are made by the SEC or other government agencies. How do you think about it? Yeah, I don't know that that segmentation is necessarily the most relevant. I'd have to think more about that segmentation, whether it's, you know, there's like a strong difference in how useful RL is across those sectors. where I see the segmentation is something basically just like capabilities based where it's like, hey, if I'm trying to do something that's like much more advanced and, you know, maybe like long horizon, then RL can probably give me a much better behavior.

34:22And I might almost think that like, yeah, those sort of like more compliance, like I feel like in those kind of environments, you probably don't want your agent doing very much because then it's like you can't make any guarantees about what it might do. And so you're probably not doing these long horizon things. And maybe RL is not going to get you what you want. But I don't know. Yeah, I haven't thought about it too much. I think a lot of the customers don't necessarily end up doing RL anyway. It's almost like the simulation and the environment. It's like a way for them to understand the paths that the agent can take.

34:54And less about we need to then use that data to do fine tuning. But I think it's going to be a spectrum. What replaces your PO? go yeah it's a good question we need the alpha yeah i mean i don't know is is the short answer i do think this this is like a fairly high salience question in the research community i think there's a lot of folks like trying to figure that out every paper has a variant um yeah but a lot but i think you know the the big question is like are we doing you know normalization based on grouping or in some other way right that's that's like i would say like i would claim we're just going to keep calling it grpo as long as the normalization is done within like a group, even though, yeah, there's a lot of things that like probably should get their own names.

35:34A lot of things that have tried to get their own names and have failed on the marketing side. Yeah. I think something that like doesn't require group level normalization, which a lot of, you know, older things didn't probably works, but I think that the older things also are really finicky. So there's, there may be other kinds of simplification and I don't know exactly what, what those will be. Where do you put the prompt optimization thing? We did a DevDay episode and we mentioned JEPA and then everybody came out of the woodwork on Twitter. DS5 bros. Yeah, exactly. Okay. Tell me, have you or people you talked to tried JEPA?

36:05I want to know like what? I read the paper. I'm just like, look, like the prompt layer of updates are not the same as weights updates, which they're just comparing apples and oranges. And I talked with a few people I respect on this, on the RL side, and they kind of validated like the way that these grad students market their papers is their thing beats the current hot thing and the current hot thing is grpo but like i they're just not that comparable i disagree with that like i actually think they are comparable in the sense that like but it depends on for what purpose right but like if i'm a company and trying to like get the best performance out of my agent like i don't care if you're changing my prompt or if you're changing my way so if you get better performance on my agent you know i'm i'm happy on that front i do think they're comparable and we've evaluated i mean we evaluated like like the so their answer was you were going to do both if you really want max performance you're going to do both yeah we valued everything from dispute and we we evaluated jeff as well and it's like it just doesn't work okay like okay that's going to be the fighting words yeah jeff doesn't work it didn't work on the problems we tried it on it just didn't it got like a minor boost over the sort of like more naive prompt we had and was just like it was like okay, just kind of like our naive prompt with our model gets maybe like 50 % on this benchmark, and like JEPA got to 56, and we do RL and we get to like 96.

37:24I mean, it was just like not even comparable. And so maybe we were holding it wrong. So both sides are claiming skill issue, right? So what they would say is you probably used it wrong. And then RL people are saying that probably JEPA guys, when they set up the GRPO benchmark, it wasn't a very fair comparison, which is exactly what my source said. It's hard to tell. You know, everyone has, everyone has, is trying to get to some version of the truth. Yeah. But what I will say is like, we, we want it. I mean, I don't know if I would say it goes so far as to say we want it to work, but we certainly want to know if it works.

37:54Like that's like actually very relevant to like. Yeah. And if it's more efficient to get there. Yeah. And then you should probably do it. That's yeah. It's actually kind of more credible now that you're like, you know, you're part of a larger core weave that you're not. obviously because i think jeppa maybe is uh makes uh open pipe like less relevant i i totally would disagree with that okay because like the level we see ourselves operating at is actually we're not like rl bros trying to figure out like the use case for all rl we're like hey we're working with all these enterprises we all these big companies we're talking to and we're trying to figure out like how we make their stuff work better and so like i personally i'm very motivated like if something like jeppa works like okay let's let's build a product around that that's how that's how I think about OpenPipe at least.

38:37No, I mean, that's a good clarification to make. Even more so, you actually took a sincere look at it and you concluded that there was nothing to do, nothing to build. Well, you know, maybe we were holding it wrong. So we had Shen Yu on the podcast a while ago and like, I think he's been a proponent of automatic prompt optimization and this idea that like you can do a lot more in the prompts than you can do in the weights. And in principle, I'm biased, inclined to believe that something like a DSP, something like a JEPA works. uh so i'm very surprised to hear this yeah like we keep trying it you know we tried the mipro v2 stuff that was hyped before that also okay i should not bury the lead on the best argument for this which is it basically jeppa models how the big labs do their system prompts it's uh genetic evolution you know and they and they sort of incrementally uh evolve based on like the overall evals that they have it's slow because it's done by humans but jeppa theoretically improves it automates this okay hold on is the kind of the big labs have something uh this is this is no no no this is philosophically the same i'm not saying like oh sure but like you're injecting a whole lot of human intuition and kind of like potentially out of band we have the best model in the world which is humanity yeah or like smart humans uh and now we're doing jeppa using dumb lms right but they're also like the humans can bring in out of band information that like maybe is not captured in the actual, like, you know, the eval, like they can be like, oh yes, technically this did well on the eval, but it's like not really, you know, like I would suspect that a lot of that ends up getting injected through that human being in the loop.

40:11Yeah. I've always been very surprised at how these guys work on their system prompts, which are tens of thousands of words long. And there's no ablations. They just kind of pick what seems to work and then chuck it in there. And that is the Claude system prompt. Can't argue with success. Is GPT-5 the first model that had a prompt optimizer by one of the large labs? I believe so, but I don't remember. Cloud Workbench had this like a year and a half ago, if you see it that way. It just wasn't like fully automated, but it was extremely good for its time. I kept telling people about it. Nobody believed me.

40:47Do we know if they used it internally? Cloud Workbench? Yeah. Okay. Why not? Oh, I don't know. My experience, knowing a lot of people at these labs, is they launch a lot of products because some team is super excited about this product. I put that, I wouldn't put that much weight on it just because they launched it. For some measure of use internally, I'm sure the people I talk to are biased. I don't know if you fully explored that thread. Yeah, no, I think that's, it's just interesting that now it's been a knowledge that like the LLM can improve your prompt. And so I think like Japan always also writing this way of like, okay, maybe we can do this programmatically.

41:22But I also think the long tail of people just prompts really badly. And so I think there's some value there versus once you go into RL, you already have like a more sophisticated audience, you know, like who gets to do GRPO? People that are really smart, who gets to do prompt optimization, like everybody's trying to do it. Yeah, that's fair. Maybe our baseline was... I know, your naive prompt is probably like, you know, top 10 percentile of prompts that people put in these LMS. Sure, I'll take it. Yeah. And then the other thing that comes to mind as you were talking about things, injecting things out of band and all that, I think it's a broader trend that I'm tracking for WorldSquare 26, which is the move to online evals.

42:00The way that we do evals today is probably too locked down. You're kind of fighting the war that you already know should be fought, and you're not fighting the wars that you don't know about because you didn't plan for it, whatever. How can we sort of move more online evals into our JEPA process? Maybe that's what it is. That part I'm much more bullish on. And we can make the analogy, like we can pull in kind of like RL intuition here, which is if you're doing JEPA on a sort of static data set, of like oh this is the input this is like what makes it good or bad output then like as you're updating your prompt like your information the data you're training on becomes less useful right because it's generated by you know because it's based on kind of like the problems you're running into before and that's the same problem you have with with rl where you have this concept of being off policy where it's like as you're doing training you really want to be training on rollouts that came from the latest version of your model because if you train on some that came from further back then it's like it's sort of still data and it's like not it's no longer representing the current issues with your model and so if you try and correct for the issues that existed back then it may not actually be helping you that much and i think you know for either rl or prompt optimization that's definitely true i think that like one way to apply that in practice is exactly what you're saying where you're using the actual data from your your real evals you have some way of saying like hey either people flagging these or no i'm flagging these or some way of saying like this was a good or bad output.

43:21I totally agree with you. If you're bringing that into your process, I'm much more optimistic that you're going to get good results. Yeah. And the pipelines are not set up. This is analytics and UX people being drawn into the ML process, which they've never been done before. If I had to make a bet as a big theme for next year, this is going to be it. I agree. I think that all of the observability platforms see that and are trying to figure out what the right shape is. I haven't seen the right shape yet but yes it seems like a like a theme for next year stat sig maybe yeah i haven't i haven't used them but opening eye seems to like them yeah i mean like i do think like buying you know an experimentation platform makes sense and like you know i think it's sort of like i've said before on the podcast i think that i'm very bullish on model routing as a feature but less bullish on model routing companies because of exactly stuff like this where like it is just going to get absorbed into the model.

44:21It's a very big part of building the process. You probably don't want to, and it's not that hard. It's not rocket science. You're just connecting pipes and making sure things are set up so that it's easy to use that data. I have a question for you, a general question. So what fraction of tokens generated by say, like the end of 2026, do you think are going to come from open source models versus proprietary Oh, that's a fun question. So we have an answer from Ankur, from Franchus, where he was like, it's 5 % and going down. I think it's going to go up because of the amount of enterprise adoption of open models that I'm seeing.

45:01And also... Because there's a lot of demand. The enterprises would much rather be on open models if they actually could get the performance they're looking for. Yeah. For cost, for privacy, all that stuff. And I think like basically, honestly, it's just literally like we may have hit quote unquote AGI in a sense of like it is the average LLM is capable of the work of the average human, not the best human, but the average human. Sure. Like it's actually pretty decent at customer service. And it's actually pretty decent. And like, I don't know, transcribing things from PDFs, whatever. so like yeah i mean totally i think i think that should rise but people who believe that it should rise to like 50 are out of their minds and i think it's a true question we should take coding out i think once you take coding out i think yeah it can be like 15 20 but i think with coding it's still going to be very low because like these max plans are like so subsidized and so many tokens are being generated like anthropic is like you know so 50 percent of the revenue is your claim that it'll mostly be, you know, that coding will mostly be closed models because the tokens are subsidized or because the models are just so much better that people use anyway?

46:10I mean, I'm paying 200 bucks a month and it's like, I'm spending thousands of dollars. Like by accident, by accident, I pay with like my credit card and I spend like a hundred bucks in like an hour. And it's like, what do you think about? This is like the thing that Noe wants to talk about for Anthropic. Like Anthropic went from like 1 billion in revenue to 5 billion. And it was like, ooh, ooh, yay. And then like, what's the margins? You have this like goose meat going like what's the margin um they say it's like six percent there you are part of the six percent that is abusing everything so everyone i'm not abusing you're the last leader it's not like i'm rotating accounts i'm just using the product for you know it's like yeah yeah but like through you people like hear about cloud code they pay the 200 a month and then they don't use it and they they pay for your influence thank you thank you everyone keep doing it right so i don't have to go away but i think like i don't really see it's hard to see a world in which quant coder or whatever model replaces that because between quality and cost it's like to make to generate this amount of tokens for 200 bucks a month i don't know how anybody can like offer like together fireworks they cannot really offer it at that price and the quality is not as good but the reason they can't offer that price is because of the subsidies right which which is not like the long-term like sustainable i mean it's interesting because so both anthropic and open AI are building their own infra, right?

47:30And like, they're going to get to a place where they're going to have idle GPUs that they own. And so they will also be incentivized to have a hundred percent utilization. And so, you know, they will subsidize some of it just the same way, you know, if you go on SF compute, like you pay a buck 40 for like an H100 instead of like the 220 listed price on AWS. So I think it will continue, but again, it depends on whether or they actually have the 500 billion like they were saying which i think they do you know just to be clear i think stargate will go online but once it goes online then it's like well they figure out how to pay for 500 billion dollars worth of compute then they probably can subsidize for a while i think they have the 500b they're going bigger isn't it obvious what do we mean by have i like at the start of this year when they announced stargate people were like oh you don't even have 10 elon was like you don't even have 10 whatever and then satya's like i'm good for my 80 but like now now we're seeing all the money start coming in and like probably it's in the order of like 200 300 billion like that you could probably get raised and committed and they're gonna get the rest like it's it's fine like i think that the plan is actually a lot i just can i just say i love this industry it's like yeah they've got like two or three hundred billion and like what's another couple hundred billion there's no other industry in the history of the world where you yeah yeah it is it is stupid but like also like do you doubt it like i i don't i like yeah that's fair no like i i literally like after last week i think maybe two weeks ago with the whole oracle nvidia and then even amd deal i'm like oh like these guys not only they've locked down stargate one they're working on stargate two wherever that uh that is and and like the sheer ambition is like freaking crazy there is still one more shoe to drop which is the non-sovereign wealth funding that open ai needs to get which they've promised to drop by the end of this year and my money is on they have to do a coin like it's i'm not a crypto guy at all but like you know this is like an open ai coin this is the one ai founder that has his own coin already yeah and like he needs more money and he said that they will come up with new innovative financing methods what else is they're already in a token selling business but you gotta that's a great line buy an open air token, it translates to a GPT-5 token, like, you sure?

49:49it's a stable coin hmm you'd have to get a lot of political buy-in I think, to take that level of risk the White House that is most crypto-friendly since the dawn of time Elon's out of there now, so maybe they can get the make the friends I think it's doable. We'll see. Who knows? For what it's worth, this is a me theory. I don't have any inside information. Yeah. Should we go back to Ruler? Yeah. Sorry. Right. Open fire. Anyways, we were saying. I think this story takes us to July 25 when you release Ruler, which you call Easy Mode for RL Rewards. And then, I mean, shortly after you get acquired in September.

50:31So maybe you just want to talk through the summer, you know, what was the vision, then maybe how the acquisition came together. Yeah, absolutely. So, you know, I mentioned my initial, like, opinion of, like, how likely this direction was to work was maybe 25%. We're up to, you know, 55 % or so. And Ruler is actually a big update on that got me from the 25 to the 50. So let me, you know, I guess just for context there. So basically, there are several problems you have to solve if you want to use RL successfully. the problems you have to solve i mean some of them are just like really dumb basic like hey you got to get the infra and like the libraries have all really sucked and been built by you know phd students who don't know like how to build reliable software so it's really like there's there's like all these like practical issues that that we're working through that's one thing and that's that's kind of what we're trying to solve with art but even after you've got that solved you've got like major issues which is like you you got to know if you're if your agent is actually or you know whatever system you're using on rl is doing a good job right that's that's fundamental you have to have a reward you have to know what's doing well or poorly sometimes that's easy to do if you're solving like a math problem or something you can come up with a data set of math problems and the known solution and check if it's the same the on the coding side there's been a lot of like innovative work around i mean there's first of all like a lot of open data and a lot of you know there's like i think the approach a lot of companies take is you you find existing test cases and then you break them but there's sort of like a way to figure out if you know you you could run the test case right and see if if your code fixes it or not in a lot of other domains it's like much more murky.

51:56It's like, what is a good job versus a bad job? How do I know if I did a good job? And you really need that information. So we've tried a bunch of different things. Ruler is a library that we released. Which let me, relative universal LLM elicited rewards. Thank you, yes. And the way it works is basically this depends on the sort of GRPO insight, which was mentioning earlier, that you actually don't, with GRPO, it has this nice property where you don't have to have like an absolute judge of the truth. you just have to judge relatively and so simplifying a lot it's basically just lms judge on a whole group so you say okay this is the task i'm trying to achieve here's four different runs of an agent trying to achieve it which of these did best and it stack ranks them and it turns out that works phenomenally well with grpo like way better than i expected way better than you know anyone who kind of like i talked to before we actually tried this expected because it's sort of in in in the the LMU is in his judge, it can sort of like self-ground because it's, it's just getting these relative ranks.

52:55Right. So it doesn't have to like have like an omniscient view of like what good or bad looks like. So that has worked at basically everything we threw it at. We've done it with a bunch of client projects. We've done a bunch of our own customers. It basically just works. Like it's basically like, I honestly kind of feel like the reward assignment problem is like fairly solved. Yeah. It's fantastic. Just any LMU is judged off the hook. Like we've tried it with so many things. So one of the results we published was we used Quen 2.5 14B as the model we're training. And as the judge, we used Quen 2.5 32B, which is like not I mean, it's fine, but it's like not it's much worse than any frontier model.

53:31Right. And even with that combination, we were able to get our agent doing like state of the art better than any frontier model on the task we tried it on, even with like an extremely weak judge model. So it really doesn't depend on having like a really great judge model in practice. so yeah it's it's just like it's just not something we've had to worry about since then at all so that's kind of like checked off so that's sort of like got me like a significant increase and like okay this is actually something people can apply this is now something that's packaged up people can just use our it's a we open sourced everything you can use it off the shelf if you stick in your train to run it will probably just work so that leaves the remaining problem which we were i guess we were talking about the amount of order but like that remains leaves the the environment problem right that's like the one big remaining piece that like we don't know yet how to automate or remove and requires a lot of manual work for every single task.

54:18For listeners, you know, this is why I kind of refer to it as self-supervised because it is like removes more and more of the human judgment and like the history of machine learning all the way from like, I guess, the start of like ImageNet and everything is really like that insight of like, you should just take humans increasingly out of it and scale up the data you can just throw in there with no supervision. Yeah, totally. Yeah, it's really awesome. Are you bullish on dedicated LM as judge models? Have you looked at those? Bespoke Labs, we did an episode with them, and they're really trying to carve our niche in there.

54:54We've looked into it. We've trained some ourselves. We've also used some off the shelf. There's an evaluation benchmark that the AI2 people put together. A reward bench. So reward bench is kind of like trying to benchmark models on serving as LM as judge. in your mind is the same thing. Yeah, yeah, yeah. Mildly different. Depends on the task. Like, LMS judged is usually more sort of product-facing and reward modeling is much more specific within, like, a chat task. Which is, that used to be the old meaning of reward model. I don't know. Maybe terminologies change. Like, I think they're pretty equivalent.

55:31I understand that. Yeah, I can see you aside. Anyway, so, yeah, reward bench is kind of like, and so we've tried a bunch off that. The thing is, I guess my maybe meta take on this is any task that is extremely common is going to end up as a specific part of the training data for the Frontier Labs. And LMS Judge is just something everybody is doing in so many different contexts that you have to assume that all the Frontier Labs have a bunch of LMS Judge-style tasks that they're training their models on. And I do believe that if something does kind of like make it in a like more than minor way into their training data, that like they're going to do at least as good a job as a dedicated model.

56:12So I don't think there's probably a lot of alpha in dedicated elements judges just because it's something that like the let me caveat that and say, like, if you've got like a very, very specific task that's like weird and has weird requirements and you have a lot of data on what's good or bad, then like training a reward model for your specific task, I think, could still work. or, you know, fine-tuning an LMS judge on your specific task could work. I'm pretty bearish on like, hey, this is a model that is trained as an LMS judge, but it's a generic LMS judge that can be used to judge anything. I just don't think you're going to beat the Frontier Labs on that.

56:44Yeah. One other version of this that is not quite an LLM, but some people are thinking about it, is something that we're working on for a future episode, which is world models. Sexy. Yeah, very sexy. First applied in video, as far as I can tell, for Genie 3, Genie 1, 2, 3. And now with code and potentially with virtual cells for AI bio. Any exploration there that's interesting to you? Yeah. So we've been playing around with it a little bit. It's one of the directions that I'm fairly optimistic on for solving the environment problem specifically. Because if you think about it like a world model, it's a simulated environment.

57:23That's like its whole purpose, right? But in an LLM-like thing, not like a Docker. Yes. Yeah, yeah, yeah. So it's like, you know, whatever, hallucinating, generating, imagining the responses you'll get from the world. So you can imagine, right, if you had like a really, really great world model that you're training on. Yeah, it's like your agent that you're using, it would go and make some tool call. And then this world model model would generate, hey, this is like probably what the tool call. And if you have a smart enough, strong enough one, then it could keep its own, you know, effective internal state of like the changes you made so far and how that affects.

57:55So we've played around with it some. You know, I think if we can get it to work really well, then that could be a solution for the environment problem where you just take a bunch of production traces and use those to condition your world model. So it understands your specific system and what its failure modes are and then train against that world model. And, you know, and the resultant, you know, agent that you train with that would then be able to perform in your real environment. So I do think it's like a really interesting area of research. Yeah. And did you see the meta cold world model work?

58:26I don't think I saw that one. Okay, yeah, it was like two weeks ago. We just confirmed that the guy for AIU code in November. And it's really interesting. Like the world model is... Oh, sorry, you're talking about the meta one? Yeah. Okay, I missed it. Yes, I did. I saw that one. I said a lot of syllables, so it may not have parsed. But like, yeah, it's literally like having a debugger as the environment, as the world model, and opening up the execution trace to the model to see what's going on and see the state and track the state as the code executes seems to be smart and exploits the unique situation of code environments where we can actually do these things.

59:02Yeah, I think the way they envision that model being used is a little different. Like, I think they're trying... Actually, I'm curious. I'll have to see the talk. But my understanding from that paper is, like, the goal they're imagining is this is almost sort of like a pre-training step. And then now that this model understands code really, really well, we can then use it as basically like a code generation or a coding agent of some kind. Okay, yeah. Which I think makes sense. That's almost more like a different kind of pre-training, I would say. The way I'm interested in applying world models is basically as its own end, right?

59:34Where it's like, actually, the goal is to come out of this with something that simulates the world, which is not something you really need in code at all because it's so easy to run code and you don't need to model what will happen if you execute this code typically because you can just execute the code and see what happens for training purposes. But it closely models how we think about code when we code. is we kind of mentally execute the model as we type and we go like, is that what we really want? Yeah, I don't know. Anyway, it's the first model that Meta's released since the MSL reorganization.

1:00:01We know, you know, just based on our context, they're very, very interested in code models as a path to AGI, which I'm also, of course, very interested in. I know we kept in here for a while. Let's wrap up on the acquisition. So a lot of people say, you know, companies are not sold, they're bought. What was that process like for you? Did it just happen? Like, what was the behind the scenes? Yeah. So that was driven by actually mostly the Weights and Bias' founding team. Yeah. So, yeah, Lucas and Sean, particularly. So they, you know, had recently been acquired by CoreWeave. And CoreWeave was looking to, you know, continue growing up the stack.

1:00:39And so, yeah, they approached me and were like, hey, you know, like, no pressure. But, like, this is like an area that we think is really promising. And, you know, would you like to work here? And so that's how the conversation started. It was like long. It was pretty painful. There were there were points as late as, you know, like the week before we actually signed where it was like unclear if it was actually going to happen. So that part was super painful. However, we've been there a month now. We just shipped a product yesterday, which I'm super excited about. It's been fantastic working there so far.

1:01:07Like I was like very concerned. I was like, OK, yes, this is great. We make a lot of money by selling our company. But like, is the work environment going to like really, really suck? And I was like, well, I guess that's just a risk I'll have to take. It's been fantastic. Like, it's honestly been way, way better than I could have imagined. Do you go down to the office, the one down here? I was there today. We work for it, so I'm based in Seattle. And they have a small office up there that we work for. Ways and Biases office in San Francisco is fantastic. If you have the chance, go visit. They do all the hackathons and co-working things.

1:01:35Yeah, there's a hackathon going on in a month or so. Every week there's a hackathon. But yeah, I mean, so do you consider yourself working for Ways and Biases or Core Reef? or both and open pipe two no no yeah is that so we so we i i report to the weights and biases like yeah founders so we're within that organization in the org chart we're there i don't know like branding wise they're trying to say everything kind of fate that that's not being sold to like big labs is kind of weights and biases so like our stuff we're launching is weights and biases branded yeah it's not um yeah not not core we've branded as much i don't know it's still like they're still figuring it out and what's the product you launched we launched serverless reinforcement learning basically it lets you offload all of the gpu management you don't like you don't have to worry about crashes and out of memories and like you know scaling up and down we handle all that for you and you just like define your environment you define your reward function and then you just like every time you run a step you kind of like ship back to our back end hey these are the trajectories these are the rewards now update my model and we just like make it work for you it makes it way easier yeah okay very thinky like it is very thinky like i love the thinking machines launch i i think they have a really good idea it's also very validating how did this take so long to appear like it seems i don't know yeah but that's i felt this way about everything like there's so many things that should exist like clearly i just think there's like still not enough people like smart people working in this space like honestly we need like i realized that there's like you know like a lot of people just feels like there's still a lot of low-hanging fruit nobody's doing okay one thing i saw from your your post was uh your north star as the rlt mac core weave is to build an old world where every agent learns continually from his real world experience so you're touching on the hot topic of the moment continual learning what else do we need to get there i super believe that and like that's basically the vision where i'm like you know i keep talking about these percentages 25 like if we get to the world where we build that then I think it's just like the advantages are huge.

1:03:37They're clear. Everyone should just deploy their agents that way. We want to be like the team that builds the software that makes that easy to do. So I talked to a lot of engineers at our customers and they're trying to deploy agents and it's so easy to get the initial prototype and like something that like kind of works well. It is so hard to get from that to something that like you are confident is reliable enough to actually deploy in production. And when you actually look at what those failure modes look like it's like oh yeah like we know if it gets in this situation or if it gets like these kind of like inputs like it behaves funnily but then it's like yeah you can update your problem to to address that but like that's not scalable because at a certain point it's like going to start breaking other things you know you don't know what it's breaking you really want some way to just like say okay look this thing you did there that was the wrong thing just like adjust this behavior when you get in this and then you know otherwise carry on right and that's what we can do with rl and that's what we can do with with continual learning is like we don't have to like have this concept of like, oh, up front, I'm like trying to make the perfect model that solves everything.

1:04:33It's like, I'm trying to make a model that's good enough. I can deploy it in production. And then when these errors come in, I'm going to say, oh, you know, exactly that. I mean, very analogous to how you train a human employee, like be like, oh no, actually that's not what you should do in that situation. All right, fix that and carry on. And that's just going to make this whole process so much easier. And I think that, you know, like, I think that there is today, like 10 times as much AI inference that could exist than is existing right now, just purely with projects that are like sitting in the proof of concept stage and have not been deployed because there's like huge bucket of those.

1:05:06And it's all about this kind of like reliability issue where it's like, OK, like it works in controlled circumstances. There's areas where it doesn't work. And so we can solve this problem. There's that like 90 percent of the like inference market, like addressable market today. That's just going to like come online because we've solved that problem. So that's what we want to do. I'm super excited about it. And I think we have very concrete ideas on the specific pieces we need to make that work. And we just have to execute against them. Do you feel like the online RL is more susceptible to the reward hacking, especially as you're shorting this loop and you don't spend as much time looking at the different checkpoints?

1:05:39I'm not that worried about it. And the reason why is because it is reward hacking is quite easy to detect once it starts happening, because once the models found some hack, it just starts like doing it all the time. It's like, oh, yes, this worked great. I'm just going to keep doing it. And so you like notice very quickly, whoa, it's doing this thing. And assuming you're using at least in part an LLM as judge to like determine which ones are good and bad. It's so easy to just throw in an extra term and be like, hey, that like weird thing that you keep doing. Like if it does that, like that's bad.

1:06:07Give it a low reward. So we've done this with a bunch of customers and like reward hacking does happen, but like you just see it and you'd like adjust your reward prompt and it just goes away. What's a thing from YC that guided you through your entrepreneurship journey? And what's one thing that maybe you like find that you disagree with YC on? Oh, that's a good question. One thing that I really identify with and I've tried to do a good job is kind of like, you know, sort of, I think they say like, hold your problem tight and your solution loosely, right? Where it's like, that's what you did. Yeah.

1:06:40Spend a lot of time thinking about what is the problem people are trying to solve. And then it's like, don't be too bought into like the way you're solving it today. Um, I think that's super important. Everyone, you know, it's, it's very easy to, to, to get that balance wrong. If you're not thinking about it very consciously, but something I disagree with, that's a good question. I think there's, there's lots of things I disagree with, but I don't have it like cached in that direction in my brain. Um, I don't know. Like I, like I definitely have disagreed with lots of specific pieces of advice, But yeah, I don't have like a great answer right now.

1:07:11I'll bridge it for you in case something comes up. Sam Altman's like, you know, everything I said as president of YC was wrong for opening eye, right? Like do B2B, ended up doing B2C. You know, you should ship like products often, like ended up being installed for three years. Yeah. Yeah. Actually, I think that that second one does resonate with me a lot. like we have tried to ship really quickly and just kind of like sort of like follow the gradient of the market i think if i do another startup like and i don't know maybe this is just me like being beat up by the market too much if i do another startup like i would like i think at at least some points i probably would have done better to be like heads down and execute my vision for longer and like kind of like go for the more ambitious thing but that would take longer to sort of like prove value which is definitely not the yc way but i think if you have like i don't know a good vision and good taste, then like that, that can like work quite well.

1:08:04Yeah. We'll see what that is whenever that comes out. But thanks for your time. This is a great overview of everything. This has been a super fun conversation. Thanks to both of you. Awesome.