Building the GitHub for RL Environments: Prime Intellect's Will Brown & Johannes Hagemann

0:00If data is the bottleneck, if having the real expertise is the bottleneck, would you rather have the smartest person in history work at your company or someone who's been there for 30 years? Sometimes you really want the person who's been there for 30 years. There's a lot of expertise that comes from really understanding a problem deeply and interact with it over a long time. And this is really what happens in training that is almost impossible to replicate in a short prompt. You really want the ability for institutional knowledge to compound over time, for best practices to compound over time.

0:27And this is how institutions and companies grow to be really powerful and successful is they stand on the shoulders of what they've done before rather than kind of resetting every day. And we want to have this be accessible to any company that wants to do this. And I think that's how we've thought about approaching it, especially as software becomes easier for people to manipulate, as the barrier to entry for coding becomes easier. We see the same happening for AI research.

1:07Will and Johannes work at Prime Intellect, which is one of the coolest neolabs in AI right now. Your mission is to make frontier lab training accessible to everyone, which I think is a very noble mission. You have really, really strong taste and just developer feel and just understanding how to, you know, that intuition for what developers care about. And then what you all launched with the Reinforcement Learning Environments Hub was like really, really differentiated and people were very excited about. And so I'm excited to chat about many topics with you all today. Post-training, reinforcement learning, agent harnesses, your platform, the RL Hub, and then big picture questions on what's coming next and post-training in RL.

1:47Does that work? Sounds great. Absolutely. Maybe to get started, you are one of the leading research labs enabling customers to post-train their agents. Can you tell me about what is that? What does that mean? Yeah, for sure. Happy to take that one and give a bit of a higher level overview of what our platform does as well, our research at Prime Intellect. As you already mentioned at the beginning, we try to make frontier infrastructure available to any startup, enterprise and Neolab as well. And basically the infrastructure that is currently locked behind the walls of the big labs where nobody really has access to them.

2:19And, yeah, we really start from, like, the compute layer and the compute orchestration layer and go all the way up then to the entire full post-training stack. So everything from, like, the training frameworks that are needed to do large-scale reinforcement learning to, yeah, the environments with, yeah, a bit of a more community approach of our environment hub to, yeah, other pieces that actually needed to do this, like, yeah, sandboxes for secure code execution and, yeah, evaluations as part of our environment hub as well. and, yeah, to offer this, like, as an end-to-end product in a sense. And what's the intuition for, why even, why pursue that mission statement of making all that infrastructure available to everyone?

3:00Yeah, there's a lot of reasons. I think one is, and I think something that we are very passionate about is just, like, open science as a way that humanity moves forward, where, like, a lot of the big scientific discoveries historically have been things that we talk about and as a world we can kind of build on top of. But kind of more practically speaking as well, there's a lot of value in model customization, where we are like the winning applications are using AI for a specific thing, for some agent, for some workflow, where you also want to be able to deliver this at scale and with cost effective performance.

3:29And so really the way to kind of really optimize these systems end to end is to be able to have access to the model weights directly, where you can then craft the model to be the best model for your problem rather than some model off the shelf where the crafting happens just in a prompt. And so it's really just allowing a deeper layer of customization than what you can do at the prompt level. And then is your vision for the future then that every company will be pre-training their own models, post-training their own models, fine-tuning their own? What do you think the future holds? We definitely think that every company will be an AI company.

4:02And we think most AI companies will want to have an AI research lab. And research can look like many different things. It can look like pre-training, especially if you're in a domain where you maybe don't just want text in, text out, if you want something more bespoke. In a lot of cases, it will be like post-training agentic or otherwise kind of focused models for specific tasks and workflows. And I think that's getting to the point where it's productionizable and cost-effective, where you can actually make this very practical for people to do at scale for the right shape problems that people want to solve.

4:35Awesome. And then can you say at a high level what your platforms us? Yeah. So we have a full stack research platform called Lab. And Lab is about giving everybody the ability to do the things that a frontier research lab can do internally, but for anyone in the world who wants to do this kind of research. And a big focus of Lab is the notion of an environment. And so I think a lot of people have heard the term environment in the context of reinforcement learning. And that's a big focus of it for us, which is that an RL environment is encapsulating the things you need to do to do reinforcement learning, where you can have a model improve via trial and error.

5:08But it's also more general beyond just reinforcement learning. And I think if you haven't heard of an environment before, it's essentially the same thing as the evals that get reported when people talk about new model releases. So Sweebench and Amy and Terminalbench, these are all examples of environments where there's a data set of tasks, there's a harness for the model to be in, and there's something called a rubric or reward function, which is responsible for grading the quality of the outputs. And so the same thing you'd use as an eval offline as your kind of your test set. You can use this in reinforcement learning as your train set.

5:39And this is a way to improve model performance interactively. And so our platform is really enabling people to use environments in their workflows for post-training, for evaluation, for synthetic data, for reinforcement learning. And it's also very much focused on as a community platform in the same way as things like GitHub are, where this stuff is new, it's complicated, and we want people to have lots of building blocks they can draw from and lots of examples and for it to be collaborative and for people to have reasons to kind of show off ways of using models or different tools in workflows as environments to allow other people to post train models in those environments to kind of have this way of sharing the ability to improve performance across different tasks.

6:18I think the general idea is to give more companies the actual ability and advantage that's currently only the big labs have in a sense of like this product model optimization loop where they can optimize their models for their specific product in a sense. And we see it as a thing where, yeah, that's the kind of reason why like a chat GPT was created by OPMI or like a cloud code was created by Anthropic. They actually have the capabilities to optimize models for their specific scaffolds in a sense and yeah, have their models work way better in their products. And some more popular as those kind of products also become in a sense like a cloud code becoming extremely popular right now um there is a big levels have yeah naturally less enough an incentive to actually make it work better for like other coding startups in a sense right and um the idea there is to to give them the tools to have like their own like model product optimization loop and um yeah i think there are early adapters on that front um i think yeah one great example um uh always in this case that i always give is scurr for example um that yeah realized that in my opinion quite early on they built their own like composer one model where they did like large-scale post training in a sense and um yeah really optimized a model where the environment was actually cursor itself so uh yeah it gives them all the tools that like you have in cursor in a sense as well and um yeah optimize a model inside of cursor and yeah we believe there's a lot of more startups that yeah will go this direction to um yeah on the one side optimize their current products in a sense, but also, yeah, build completely new products that are really not possible right now without having this product model optimization loop.

7:58Awesome. And can we say a word on, you said something interesting. You said environments are just evals. Can we dissect that statement? In my head, an environment is a state. It's a description of world state through which, you know, you observe what actions you take, you observe how world state changes, and therefore you update your world model. That to me is distinct from an eval, which is like, you should have gotten this answer on this set of questions. And so can you help me merge those two realities? Sure. Yeah. So I think there's a version of eval that's kind of where we were maybe a year or two ago where a lot of evals were like question and answer.

8:34And it's like this big bank of questions. And then maybe there's other notions of environment that people think about when they talk about like an old school RL with like Atari that is much more about like this kind of long running state interaction loop. And I think where we are now is it's both in the same thing where especially the environments that you want to do large scale training on, they do have this complex state. They maybe are simulating a web app. It's a full fledged kind of coding platform where you have an agent doing these things. But in the original RL games, there's always a reward.

9:02There's a goal. And so the notion of there being a goal of this problem, where it's not just running through some system and a human is going to kind of vibe check it, there actually is something that can measure progress and performance. And that's kind of what I mean by it's uneval in that there is a system that it starts at some state, it interacts with the system, the environment, the harness, the agent, whatever you want to call it. But there is some goal and there's a way to measure whether it's doing well or not. Okay. Got it. And then is there a difference between, you mentioned kind of cursor as a great example of somebody doing really great frontier work and reinforcement learning.

9:39Is there a good way to think about when should you be kind of constructing RL environments versus when should your actual application and your application states be the environment, so to speak? Yeah, I think there's definitely reasons where you want both, especially like let's say you're training a model to be a really good Rust coding model. Here you might want it to be good for lots of different applications where you'd have different environments that are focused on some like domain task. Maybe you're a company that wants a model that's going to be really good at calling your tools or using your specific domain language that then you can provide as a service to people who are building around that.

10:15And there's also ones where the product very much is like a user interface for an end user where the user is interacting with an agent, in which case it might make the most sense for the product to very directly become the environment where I think the companies who might want to be doing this are the same who care about whether they're using Cloud or GBT5 or Gemini or the ability to choose models and have internal systems to evaluate. whether a certain system prompt is good or whether changing out a model endpoint is good or whether using the mini version of a model is a better cost performance trade-off.

10:44The infrastructure to do that is that a lot of people have already been building at these kind of advanced agent companies is the same infrastructure you use to do reinforcement learning. And so I think that's kind of this kind of convenient world we ended up in where the training paradigm that makes the most sense for improving model capabilities is the same sort of thing that a lot of people have been building up the muscle for just without doing the training piece. And so that's kind of where we see RL being a very useful tool for people to then have this as an option in their toolkit for system optimization.

11:13Got it. Okay. I want to talk about agent harnesses as they apply RL. I feel like harnesses is like the theme of the moment, especially with you mentioned Cloud Code getting so much love. I think one of the things that they do exceptional engineering around is the harness. Harness and RL, are those things orthogonal, mutually exclusive? Like how do they relate? they definitely relate um i think the way i think of a harness is like a piece of the environment and so um there's some for any like eval or environment task there's some input a bunch of stuff is going to happen then there's some output state which is then going to be graded and so this whole intermediate piece whether it's uh interacting with some simulator or interacting with uh some another agent or physical world sim this is the environment and the harness is very much a piece of that, where it couples how the model interacts with any other pieces of the system.

12:05And I think depending on the application area, like I think for coding agents, we have a pretty clear like definition of what's the, you could say the harness is the CLI coding agent and the terminal is the environment. But this isn't necessarily going to be universal across like all different types of agents. In some cases, it's a system prompt and some tools is the harness. In some cases, it's something that is going to be spawning sub agents and those sub agents also have their own harnesses. And so there's a lot of complexity. And I think the way we've thought about it is like harnesses are going to keep evolving.

12:34There's going to be this Camperian explosion of ways people want to use models. And we want to take a pretty general approach in defining what you could do with a harness. And so what we are really thinking about and why we use the term environment as the abstraction is like agent is too narrow, harness is like kind of too narrow. And you can do all of these things within an environment, but the environment as an abstraction on the whole allows this, any sort of system model interaction is in scope. And then do you think then, do you think all companies should be post-training their models with environments?

13:09Are there specific kind of, you know, where the bullseye, like you absolutely need to be using environments versus like you could be post-training with a different method versus you should just be prompting your thing? I mean, I think environments are tools you can use to do all of these things. And so I think That's when I talk about kind of environments beyond RL. I think part of the reason why it's a useful abstraction is because it doesn't tie you to RL. Let's say I want to have a small model and I want it to be distilled from a big model. The way you can do this is you take the big model, you plug it in your environment, you let it run a bunch of times.

13:42Now you have all this data that comes from the same interaction protocol. You can use the same greater at the end to filter for the best examples and then do SFT fine tuning on that. You could do prompt optimization with an environment. You could A-B test different models with an environment just as you would with an eval. And so we really, I think the idea is that every AI company should be optimizing their AI systems. I think that's a less like controversial take. Okay. And maybe there's another way to frame it. Like the eval is almost how your agent performs in the set of environments that you expect your customers to face.

14:13Yeah. I think it depends on what you want to call, like, I mean, in kind of traditional machine learning terms, you don't want to overfit to the test set. And so in some ways, we kind of are already accepting that we're going to be like using the test set or the eval to measure, to have that kind of filter back into the model. And so I think it's a little tricky to distinguish even like what is the eval? What is the environment? We kind of think of them as one and the same. And we use the environment term very generically, where an eval is a type of environment that's used for measuring performance but not training on.

14:45and that's kind of how we see a lot of people thinking about when they're doing evals what they really mean they're doing is they have some way of measuring current performance and they're iterating on it and in some ways rl is this iteration applied at scale where you're automatizing the process of changing the model a little bit changing the prompt a little bit and having this be the way that you can kind of hill climb on some goal maybe another question Do you see reinforcement learning as synonymous functionally with post-training? Meaning like if you're post-training models, are you doing reinforcement learning or are you doing other things?

15:18There's definitely a lot of things. I think reinforcement learning is like the big thing now where it's like in many cases, practically speaking, if you're doing a large scale model, RL will be where you spend the most of your time and focus and compute. But it's not the only thing you want to do. There's a lot of things involved in the pull process of going from some initial model to the system you want to deploy. This can be prompt tuning. This can be SFT. This can be online distillation. There's a number of algorithms that all kind of are under this umbrella where RL is like the big one in the middle.

15:46But there's a lot of stuff around it. And really, I think exposing this toolkit to people and letting them have all these knobs they can play with is the way to unlock. And are you finding that it's the, you know, really smart researchers that actually know how to make this stuff happen in practice? Is it, you know, towards your goal of democratizing AI development for all, you know, does your average Fortune 500 know how to use the platform and get value out of it? I think most companies have people who can. I think any Fortune 500 company will likely have a team of AI engineers who are capable at following the latest tools, who are good at using cloud code, who have a lot of opinions about models and prompting.

16:28And those people certainly can do this. That's kind of the audience where we see as the target customer for this. Especially if you give them the right tools to actually do it. Like maybe some of those large companies don't really have anybody in there who can like debug your GPU cluster in a sense to actually kick off such a run or other components that are needed in there to like actually just make it easier in a sense to do large scale like agentic reinforcement learning with tool use, with code execution and pieces like this to just abstract away the entire infrastructure for them. Yeah, got it.

17:02Awesome. Actually, on that note, do you guys have any favorite customer stories that you want to share? Yeah, one of our favorite customers that I would like to point out in a sense is RCI and Neolab working on like Frontier Open Models have been working with us on the entire stack in a sense, right? We've been talking a lot about reinforcement learning, right? But yeah, we've been also doing a large pre-training in the past. that's basically where our history is coming from in a sense as well right so yeah I've been training with them some of the largest mixture of expert models and actually able to yeah open source them as well and yes well on the on the post training side with them as well maybe well do you want to share some more sure yeah so they're a very close collaborate of ours that we've I think I'll be all been friends for a long time but also like we've been I think they've been a way where we've had the they've had a lot of things that they want infra for.

17:55And that's been a way that we've been able to kind of force us to build out a lot of the pieces, both from compute orchestration to post-training to pre-training to inference around kind of just making everything that is needed to kind of be a frontier lab. And I think they're very aligned with us in kind of the openness mission. But I think their kind of focus, they are more targeted at like enterprises and the end user where they are kind of going to work more directly with customers in terms of like end to end kind of delivery of a certain artifact where I think where we come in is we are we are really focused on the developer experience at the infralayer and the ways that we can make put these tools into more people's hands where the process of going from idea to deployable model can become as seamless and kind of quick as possible and efficient as possible.

18:45Awesome. And then any other favorite customer stories, maybe on people that are using the environment's product specifically? Yeah. So we are definitely really focused on like the research community. And like some of this is like a lot of grad students use it. A lot of like students and people who are getting early in like their career learning how to do this stuff or using it. But also a lot of labs who are focused on a very specific domain where let's say you're starting a medical AI lab. We work closely with a number of groups in the medical space where they want to create more both benchmarks to understand how good our models at medical capabilities, both in terms of diagnosis or patient interactions or question answer about medical literature or agentic search over certain medical tasks.

19:27And so like SoFont, OpenMed being two that we work closely with, where the focus there is to they really care about like earning trust from the medical professionals. And so for them, they really want to focus on the domain and not as much on kind of the generic medical, the LM infra that is kind of like a headache for a lot of people. And so for them, like being able to kind of have this platform for creating evaluations, for showing them off, for being able to use them to then improve model capabilities that could then be deployed locally in a hospital or deployed locally for some end user where the ability to have this customization and have this kind of end to end trust and understanding of the data providence for tasks at hand or the ability to kind of customize models very directly is very key.

20:12Do you have any customers that are using you for more of like what you call the old school kind of Atari style, you know, learning from your environment type? Do you mean like, so when I think of Atari, I think of like non-LLMs. And so we definitely are focused on LLMs and foundation models that look like LLMs. There's definitely a lot of researchers who use our platform for these things that are more like, there's some examples we have on the platform that are much more like games. So, I mean, one fun one that I use is like a demo a lot is the game Wordle from the New York Times, where this kind of ended up just being the like a great hello world environment for people because it's the infras really simple, but it's very expressive where you can kind of get a feel for it.

20:52And you get this aha moment of seeing a model learn to think about the game as you give it rewards for doing better. and you can do it with a really tiny model too. Like it's in this sweet spot of difficulty where you can do these runs on like a couple GPUs in an hour and see a model actually learn how to get better at the game. Yeah, I would say like the more toyish game examples. Yeah, usually the ones that people are going for for actually learning how to build those reinforcement learning environments. And yeah, that's also what people are heavily using the environment hub for in a sense, just because we have all this infrastructure built around it to be able to actually test and your environments and yeah that's usually how they start out and then yeah go to actually building more complex environments later on another group I would love to give a shout out to in a sense that I've been building some of the more complex environments on the hub are people part of our reinforcement learning residency where yeah we have like a group we initially started out with like eight to ten people I think 14 to 16 are now in the group you have people grad students as well as yeah people working full-time that yeah part-time like building reinforcement learning environments, as well as doing novel research on top of the Environment Hub.

22:01And yeah, those folks have built amazing environments in all kinds of verticals, in a sense, everything from verifiable software engineering and lean to medical physics environment to some cybersecurity environments. And then also we give them the tools, obviously, now to actually do the training in those as well. Yeah. Awesome. Can you help demystify for me, I've heard that some of the foundation model companies are spending millions of dollars each on some of these environments. And so you mentioned cybersecurity at the end. What goes into constructing a cybersecurity environment? Yeah. So there's someone in our residency program who actually does a lot of this stuff.

22:43And so there actually is a lot of tooling in the cybersecurity world around these capture-the-flag games where there's some system that has some hidden bug in it. And this is like a challenge that originally these are built for programmers. or programmers will have like a little hackathon where they try to go find the bug in some system. But you can adopt these challenges to LLMs as well, where it then is a full software environment, where it's a terminal, where the agent lives in the terminal and has access to tools for doing bash commands. Maybe it's using Cloud Code or some other wrapper for an agent harness, but it's in a terminal full of files and can interact with these files.

23:14And then at some point, it marks that it's finished with the rollout. And then you can grade the state of the environment using other pieces of software, other code that executes. But we do like, we actually have a lot of people we work with who are in the data and environment space where we've found that there's a lot of interest in using reinforcement learning as a way to evaluate data quality, where the ability to measure what happens when you train a model on a set of tasks, set of rubrics, set of environments, allows you to understand bugs in the environments. Because there are issues that come up in reinforcement learning where maybe if your environment has a backdoor, a model can exploit this and kind of game the system.

23:55And so I think there's a lot of interest in people using RL to like in the pipeline from like idea to environment that ends up in some frontier labs, like foundational training run for the next GPT or cloud model. There's a lot of vetting that goes into this and doing these like smaller or like medium scale runs in like, let's say one environment allows you to really poke and see where the problems might be. Okay, super interesting. And if we take the cybersecurity environment example one step further, by the way, I have no particular affection for cybersecurity, but it's something that like I love having a specific example in my head.

24:27I could see how you could construct a toy example, right? These Capture the Fly examples, I would imagine they're toy examples. I would imagine they look nothing like the actual kind of corporate network environment of a, you know, real company with all of its cybersecurity products. And so I guess how do you construct an actual, you know, does what people can construct, does it actually scale up towards imitating and reflecting the complexities of the real world? Almost like the, you know, the robotics terms of real gap. Yeah. Is there a sims real gap in crafting these environments? And is it a solution just to like train on like real kind of corporate security environments?

24:59Yeah. So I would say there's less of a barrier in terms of the actual complexity. Like the, in principle, these can be as complex as you want. It's anything that could be on a computer. So just think of anything that's on a computer or network of computers as potentially an environment. What really kind of becomes the bottleneck in many ways is like cost of the simulator, where I think there's a lot of focus on identifying clever ways to kind of mock the right piece of the system. Where let's say you have an agent that you want to like, let's say the Internet, like let's say you want to have an agent that like does a lot of like web search.