

9 Nov 2026 · 35m · 17 chapters

The episode explains how an AI agent system called Hermes can automate “life admin” tasks and how RL (reinforcement learning) and continual learning could let agents improve by learning from their own failures. It also covers token-efficiency, steering, and “trace-to-environment” ideas for recursive self-improvement.
Sam (Noose Research) works on RL and has worked on Hermes-related training. The host describes building/using Hermes and earlier RL work, including hot-swapping newly trained weights into a running agent.
Hermes started as an internal productivity tool; it can write skills and run end-to-end workflows. A continual loop could: collect agent traces → synthesize RL environments → train nightly → update. Models may not need excessive reasoning; steering/inference-time control could improve behavior without retraining.
Hermes finding a government-auction three-wheel ticketing car; “Hermes mobile” purchase; finding clothes matching pottery via image similarity; smart-home analytics (dog door counts); drafting utility-bill checks; RL “endless terminals”; trace-to-environment from agent failures; steering ideas for pragmatic math solving.
Written by AI. May contain mistakes.
An introduction to the Hermes AI agent and its capabilities.
“In SF and New York, they drive around these little three-wheel ticketing cars.”
Sam shares personal experiences and uses of the Hermes agent for productivity.
“And this time I got an old friend with me, Sam, who's working at Noose Research.”
Discussion on creative uses of Hermes for art and smart home tasks.
“because I think that is a good gauge of, you know, like who's using it, how they're using it.”
Exploring how Hermes assists in daily life tasks, including finding a car.
“That is, and it's like, what vibes with this?”
Highlighting various innovative use cases and workflows created with Hermes.
“to kind of be on your computer a lot or like be.”
Discussion on the evolution of Hermes and its RL training frameworks.
“And, like, of course, a lot of people use Hermes for things like, you know, automated crypto trading workflows and market analytics.”
Exploring the concept of recursive self-improvement in AI and its applications.
“workflow, let's codify it and let's make it a little bit more deterministic where we can.”
Explore the challenges and advancements in long-horizon reinforcement learning models.
“I think all of the open weight Chinese models that have come out have since, you know, really benefited from focusing on this like longer context stuff.”
Discuss Xiaomi's early adoption and contributions to agentic reinforcement learning models.
“They were some of the earliest to jump on this agentic RL stuff and they had also picked up focusing on training specifically for Hermes agent pretty early on.”
Understand the process of creating synthetic RL environments based on agent traces.
“And this could work in a continual learning loop like every night, every task that you've had your agent do while you're sleeping, it goes and creates RL environments, trains itself on them and updates.”
Examine how failure analysis helps improve AI models by creating training environments.
“But if you're, oh, maybe you want to start serving a bunch of traffic and you want those traces too.”
Delve into the trade-offs between token efficiency and reasoning depth in AI models.
“but I think that that's, like, the last, like...”
Discuss innovations in steering AI models towards desired behaviors dynamically during inference.
“Like, I think there's a lot of cases in which models sort of over-reason because they've been sort of trained into this basin where, like, this longer reasoning answer involves a lot of backtracking and second-guessing.”
Learn about the intricacies of model dynamics and inference mechanisms.
Discover how controlling model behavior relates to interpretability.
“I think most people don't want to touch it.”
Utilize visual analogies to comprehend the complexities of AI models.
“In my head, I almost envision that animation that you see when you look inside of how a key clicks all of the different pieces of a lock before you can open it.”
Explore the potential for continuous learning and interaction with AI models.
“But you kind of blew my mind on a few of these pieces.”

Get the gist, explore an idea or ask about something said.
On Agentic Conversations (formally mlops.community), what are the key points of this episode?
Automatic transcript. May contain errors.
0:00In SF and New York, they drive around these little three-wheel ticketing cars. They started to do some research and found out they're sold on government auction. In May, a listing came up that I bought. You have one? Yes, I drive it.
0:12Sam Herring: You took the bit too far? I have one. I run loops on my own hardware of like, I have a Quinn model running as Hermes that it designs an RL environment, kicks it off, downloads the weights, and then hot swaps into this. Dude, literally Neo, learning country. Yeah. They call it trace to environment. And the idea is that you take a bunch of agent traces and synthesize RL environments based on those traces and then train on them. And this could work in a continual learning loop like every night, every task that you've had your agent do while you're sleeping, it goes and creates RL environments, trains itself on them and updates.
0:52We've only trained models to be like this because we expect that they sort of make a wrong mistake and then correct themselves. I actually think they could reason less and still get to the right answer.
1:14Sam Herring: We're back. And this time I got an old friend with me, Sam, who's working at Noose Research. You all probably know that company because of Hermes, the agent that has taken the world by storm. Sam's working on RL, and it's been great to catch up with him. I just want to talk through what you're most excited about, some use cases you have for Hermes, RL in general, like what you were doing a year ago versus now, how it's changed. And then maybe we can touch on the hottest topic in the world, the continual learning and potential recursive, what is it, self-improvement? Recursive self-improvement.
1:58Sam Herring: Yes, the RSI, dude. It is so hot right now. So I laid it all out for you. Where do you want to begin? Thanks for having me here, first of all. Super, super stoked to be sitting down and chatting. I'll start off with kind of my personal uses of Hermes. because that's a good intro to those viewers that might not know, you know, what Hermes is or the full power of what you can do with it. And I'll just give a little bit of the context and the background about how Hermes started. And if you know what Hermes is or if you're using Hermes, tell us in the chat because I think that is a good gauge of, you know, like who's using it, how they're using it.
2:33Sam Herring: Yeah. So Hermes was started internally as sort of a productivity tool. It was an idea to be like, we have Open Claw, and it kind of had had its moment sort of about a year ago and changed. Maybe, you know, nine months ago, it was really, really starting to take off. And some people internally at Noose were like, wanted something different out of it. And they wanted to design their own version of kind of a long-running agent harness that would just improve productivity internally. Like there was no ever semblance of like, oh, we're going to make this a product and people are going to use it. and so we did and they kind of started like telling us like hey you should you should start testing out hermes and i remember like telling some of my friends like oh yeah we're like testing out this new thing internally and i didn't talk about it much um i was still using cloud code at the time for all my work uh and i really didn't like have anything that was jumping out at me to use hermes until one day uh i kind of either was bored or i think i was on i was having a conversation with a friend on Twitter who was posting all these pictures of like ancient art pieces like these old ceramic pieces and like Native American art and they were saying like how do I find clothes that match this art nice and that was the first idea clicked in my head of like you know what I could probably task Hermes with doing this and just see how far it could I knew the pipeline in my head of like pull a bunch of images from men's clothing say you know do some sort of image embedding similarity to the target image of whatever art you're looking for and then rank them by like price or by you know whatever there's there's a million ways to filter it once you have this like similarity distance so tasked hermes with doing it and an hour later i came back to him with a program that you know it was like code is on github in an hour no way and it could find any clothes currently listed that matched either clothing that you could send it or a picture of something random.
4:26Damn. Yeah. So I still use that. A random item, just like pottery.
4:31Sam Herring: Yeah. That is, and it's like, what vibes with this? I really am inspired by this pottery. I need that. Yeah. All right. I'm going to go look up that. Yeah, yeah, yeah. Do it. It was a fun use, and I still use it when I want to find clothes. That's great. The other thing is in SF and New York, I guess other cities, they drive around these little three-wheel ticketing cars. They're these tiny triangular cars. Yeah. I have seen one in my neighborhood, one or two, and I started to do some research and found out they're sold on government auction. So in February, the first live demo of Hermes Agent that we did on Discord, I got on and showed everyone the end-to-end process of me tasking Hermes to write a script to search government auction sites every day and find one of these cars.
5:18In May, a listing came up that I bought. You have one? Yes, I drive it.
5:23Sam Herring: the bit too far. I have one and I drive it around San Francisco all the time. People probably are like, no, don't wait. I'm coming back. I paid for my parking. Well, it's spray painted. It's interesting. It's painted so... It's got all this graffiti on it and stuff because the guy who had had it previously was an artist. So it's like people think, oh, I'm a ticket person and then they realize it's not. I'd love to repaint it. It's kind of blue and black right now. It almost is like Hermes colors. It'd be cool to have some yellow streaks in there or like I'm jokingly calling it the Hermes mobile to coworkers.
6:01Sam Herring: I mean, it got you the car. It did. It did. And that's what I tell people often. Like if you're looking for an apartment that fits some specifications or you're looking for a car and like instead of checking Craigslist every day, you know, that's a great use of Hermes. It's like life admin tasks that like would require you to kind of be on your computer a lot or like be. And follow up. Yeah. And, you know, outside of that, we have seen wild uses of Hermes. Tell me about this. That's what I want to know. A lot of people use it for creative pipelines. Like our launch videos on Twitter, super hype.
6:35They're beautiful. The music behind them, people love, all done through Hermes, through this Hermes workflow that some of our designers have laid out that you basically end-to-end get this beautiful ASCII art launch video where it composes the music itself. And the whole thing is just this cohesive creative pipeline. And so our designers use it a lot for that. We've seen a lot of uses of people using Hermes for smart home automation. So like having Hermes do analytics or control on like your lights or like one person, they had like a dog door that was attached to their smart home and like they could pull the numbers of how many times their dog had been in and out, you know, within a week and like start tracking the behavior that way.
7:17That has a beautiful data visualization. Yeah, yeah, yeah. It makes this pretty, yeah. I mean, anything's possible, man. And, like, of course, a lot of people use Hermes for things like, you know, automated crypto trading workflows and market analytics. You know, I think, like, we created an internal and sort of external benchmark to evaluate model performance inside of Hermes. And there's already a ton of coding agent benchmarks. Like everything out there is targets like the majority of what people use that for, which kind of comes down to writing and executing code. But we wanted to target the things that were outside of that.
7:54So a lot of it was life admin stuff. A lot of it was, you know, managing your utility bills, having Hermes go through your email, compile all your utility bills and like even draft a check to send out. Like you're the last step in the approval workflow for that.
8:09Sam Herring: and that's what i that's probably what i love is how far it can get without me oh yeah and because you don't necessarily need to and also you don't need to sit there and explain it step by step it's like hey here's this outcome that i'm looking for you go figure it out once it's figured it out then you can optimize on it and you can decide like okay now that i have this workflow, let's codify it and let's make it a little bit more deterministic where we can. You don't need the biggest model to go and do this every time that we have it because we've already explored the space. We found what the optimal way of doing it is.
8:50Sam Herring: And now we can optimize. And that's the cool part about Hermes too. I mean, you're teeing me up right there. It's like when we designed it, we started to add in the ability for it to write its own skills. And normally with the agentic workflow stuff as you were starting to do more you would tell your agent like oh right this is a skill hermi is just like if you have it do something especially multiple times it's all converging to the same skill i think that was a big unlock man that was the thing that made like that magical experience where you're like oh wow it just does it and it it does it like automatically i don't need to think about it i don't need to tell it to do it and it knows how to do it yeah it's pretty wild so yeah we've had a lot of fun and growth with Hermes and I think this whole landscape is a lot of learning about you know what the right paradigms are to use and how to we talk a lot in news about you know time to magic we want to understand like from someone's first uses of Hermes how long does it take them to really click and understand that this you know this agent grows with you and has this Like me as that user, how long until I can offload a task and trust that, wow, that can get done and I don't have to think about it.
10:02Sam Herring: Should we talk? RL? Yeah. Yeah. This is perfect segue into the rest of what I wanted to talk about. So you asked what I was doing a year ago and how the landscape has changed. A year ago, I was working on, we had our own internal RL framework. I mean, it was public. It was open source. But, like, we used it for all of our RL training internally. and Thinking Machines had just released their Tinker library, which was basically cloud RL. You could offload your training loops to their compute, and you could pay them per use, but for a while in their dev beta phase it was free. So we basically were getting this super great deal on running RL experiments.
10:43Take advantage of it while you can. That only after a while started to cost us a little money, but still worth it. And I was an intern at News for about a year, and then I joined full-time in October of last year. So I just passed your anniversary there. Thanks. So my first project was to integrate our RL training library into their online RL, their cloud RL thing. So I built out this whole library. It was a lot of the RL stuff that we do now is very different than what it was back then because the whole idea of a genic RL like was just starting to form like a lot of times we were like the stuff i was proof proofing out for this was like arling on like certain benchmarks like math benchmarks and like different types of problems like it wasn't until about march when i was using this framework to build out um it was a project called endless terminals and it was basically procedurally generated terminal use tasks that you could just keep generating them with varying levels of difficulty and then training the model on it, and it would just get better and better at this complex terminal management.
11:56You're talking about Hill Club. I know, which is like ground zero for agent performance. You optimize that. Your agent gets so much better out of the box. That's awesome. So that was a really cool one. And what I'll say about this is that the main contribution I made to Hermes, which we ended up moving off of our RL framework, but in the early days, this was the biggest pieces of code that I ported into Hermes Agent. we built that idea of using our RL framework with this on you know this cloud compute into the earliest version of Hermes and the idea being that if you're running Hermes with a local model and you have an ability within Hermes to design an RL environment kick it off on some cloud compute and just let it and then download those weights and replace them with whatever's running in Hermes.
12:45Sam Herring: It's like hot swap. Yes. It never, it can teach itself a skill. Like if you task Hermes with like, I want you to create an RL environment to learn how to use the terminal better. Yeah. Right. I'm having a horrible time with this ERP system. Yeah. Like here's a bunch of data of me using it. Go make it easier for you to use it. Yeah. And so that was the earliest like embers of what we would see RSI becoming. And there's a lot of other ways that this has all kind of come into the fold. But that was a big contribution I made. And I felt, you know, I run loops on my own hardware of, like, I have a Quen model running as Hermes that it designs an RL environment, kicks it off, downloads the weights, and then hot swaps into this.
13:36Sam Herring: Dude, literally Neo learning Kung Su. I know Kung Fu in the moment and you're there like oh sweet my model is now much better yeah and it can do that but then it changed yeah so we ended up like our training framework was good but a lot of priorities in the company have shifted since we did all that namely Hermes agent took off way more than any of us expected so we shifted a lot of resources towards that um model training paradigms got a lot harder since then like the the deep seek rl explosion was sometime you know yeah like it was recently enough that people are just now figuring out how to really harness like not only rl on specific tasks or specific task types but this like long context long horizon rl and like really hammering it for agenic loops and like agenic you know models and I think we've noticed this a lot in like the latest, you know, anthropic series of models.
14:37I think all of the open weight Chinese models that have come out have since, you know, really benefited from focusing on this like longer context stuff. The first one that I'm forever will show this model because I personally love it. So Xiaomi, the telecommunications company. And CarMaker.
14:58Sam Herring: Yes, and CarMaker, right? Yeah, the massive. incredible car makers. I'm really thinking about getting around because I live in Germany and they can ship them over there. You should. I know. See them a lot. They seem really good. Anyway, I digress. What do they do? They were some of the earliest to jump on this agentic RL stuff and they had also picked up focusing on training specifically for Hermes agent pretty early on. And they're doing it in the open, like writing papers about it? Yeah. Really? I haven't seen this at all. Xiaomi is like, I was a fan of their model. It performs really well in Hermes for its weight class or whatnot.
15:39And so we were really inspired with what they did to sort of, we've been amassing all of these different learnings and basically saying like, okay, if we can take all of this Agenic open training and stuff, and we're just kind of reshaping the post-training team to be able to do this kind of thing well. Like that's, I think, our goal as the RL post-training team at News is we got our start training models. And we have a really strong talent density in terms of the, you know, there's eight of us and we can train a really good model given the right time and resources. That is so wild. So we've been sort of biding our time and making sure we can do it right.
16:17Yeah. And so definitely more to come on that soon.
16:20Sam Herring: But how does it go to the continual learning? Like explain that now where you're at with all of it. I'm glad you asked. All right. I'm glad you asked. So, Xiaomi, the latest model is Mimo V2.6 Pro. In that paper, they talk about this brilliant thing that has come out lately. And it's sort of obvious when you think more about it, but still the idea struck me. They call it trace to environment. And the idea is that you take a bunch of agent traces and synthesize RL environments based on those traces and then train on them. And this could work in a continual learning loop like every night, every task that you've had your agent do while you're sleeping, it goes and creates RL environments, trains itself on them and updates.
17:05Or you could do, you know, amassing tens of thousands of different traces on data sources that you have. Why not just spin up a bunch? Yeah, if you have a lot of data of Hermes agent uses, what we do first is what's called failure analysis. So we go through and find the exact points where the agent makes a mistake. then we cluster those by the mistake type and once you have a bunch of different piles of oh the agent you know entered the wrong directory here or oh the agent you know copied the wrong file here like you just have a bunch of different types then you can sort of build an environment that captures those mistakes or build a bunch of are you human labeling those or is that all all automated pipeline also awesome yes and again that helps with the recursiveness yeah huge okay that was one of of the last things I've built at Noose was like our internal, we call it the insights and observations pipeline.
17:58It's actually based on some research, Anthropic Publish, where they do the same thing. And they didn't publish the code for it, obviously, but they did publish their methodology. So we sort of co-opted that between Claude itself and their Claude code variant of doing it and built our own internal pipeline that does this insights and observations, like failure analysis, clustering. And then we have someone working on lately, which I'm going to start collaborating on too um you know environment synthesis and then we can just loop that what's the environment synthesis it that's the generation of like you you take a bunch of failed traces and and you have a bunch of examples of like here's where the model messed up and then you build some kind of like fake environment that that's supposed to match pretty closely to like those like the world that those environments were in like the state of that computer at the time and the files but it's not the real thing obviously um it's just synthetic and then you can add sort
18:55Sam Herring: of levels of difficulty based on how well models do how often are you trying to train on like gigantic cloud or so not train on the cloud but like train for cloud environments because if i've got a bunch of different aws services i'm using and maybe i've got some google cloud services I'm using and I want to recreate a little bit of that can you bring that into the environment yeah yeah we've done a fair bit of like Google Cloud you know simulated like cloud environment training too and and I think that that's a lot of this is very green like there's so many areas of exploring ways in which you can put an agent in the simulated scenario that we like probably haven't touched yet and I think exploring like cloud deployments and and managing cloud infrastructure is one of them because a lot of times the traces that we see or the traces that we get from ourselves are all Hermes running on our own local hardware or local machines.
19:55That's what I was thinking.
19:56Sam Herring: But if you're, oh, maybe you want to start serving a bunch of traffic and you want those traces too. And you want to see like, oh, that now the agent is trying to fix something on production traffic or simulated production traffic, right? But it's messing up and we've got a 7-1 incident and then we can see that cool it's messing up here let's spin up that environment kick off another training run and have that whole loop happen again yeah so i'm very excited about that that is one of the most the things i'm the most excited about and i think is a very healthy place that the rsi conversation has turned to uh is less of like just having the agent decide like to teach itself some new skill because we kind of do this already with like the tiers of memory in Hermes and this like automated skill writing like the agent can kind of already bolt on things that it's learned to itself in context but learning from its own failures you can only do so much without sort of this this idea of like like just collecting all the failures and building some learning environment for it and yeah you know that might honestly be what's left in post-training.
21:06Like, everything else has kind of been... Really? Yeah. I mean, there's a lot to do, but I think that that's, like, the last, like... Yeah. I mean, only thing we can do right now is improve models for agentic use because that is by far, in a way, like, what we care about models doing, how long they can act in a harness, how long they can go without making mistakes. Yeah. Improving token efficiency is one. Like, that's one I explored early on in my research. Tell me more about that. You want a model to be able to finish a task as efficiently as possible. And then there's this like exploration versus exploitation balance of like you want to produce.
21:43Yeah. You want it to spend some tokens exploring other options. But if it knows that it's going down a wrong path, you don't want to waste a lot of turns or tokens doing that.
21:53Sam Herring: Yeah. You know what messes you up there, though, is like the models will a lot of times lean on their pre-training. Yeah. And so you say like you want something done and it goes, oh, yeah, I know how Vercel works. Meanwhile, Vercel came out with a new version. And guess what? You don't know how Vercel works. So it doesn't go and like read documentation or try and figure out how it works because it says I know how it works. And then later you're like, ah, it doesn't work like that, does it? You get these, I don't know how to explain it, but there's those moments where the model is magical because it knows things, but it also is almost like it's Achilles heel.
22:31Yeah, 100%. And that used to happen a lot worse with knowledge cutoff dates. And now less so, but I think a lot of labs are trying to train models to act with the, oh, I should look this up. instead of, you know, oh, let me pull from my memory. But then it's not as token efficient, right? Exactly, because then it has to read through the documentation. Yeah, so that's what I was thinking.
22:57Sam Herring: Like, if you're doing research on token efficiency, like, you want that, but you don't. It's a huge trade-off. And for me, token efficiency comes more in, like, long reasoning traces. Like, I think there's a lot of cases in which models sort of over-reason because they've been sort of trained into this basin where, like, this longer reasoning answer involves a lot of backtracking and second-guessing. I think a lot of that is, I mean, this is, like, my personal hot take because I think that, like, a lot of that is somewhat unnecessary and that, like, we've only trained models to be like this because we expect that they sort of make a wrong mistake and then correct themselves.
23:35And, like, most of these models are good enough. Like, I actually think they could reason less and still get to the right answer.
23:42Sam Herring: Yeah, we also just kind of assume that the longer it reasons, the more correct the answer will be or the more accuracy we're going to have at the end of the day. And that's true that potentially, like, a lot of that reasoning is wasted reasoning. Yeah, it's just tokens. And, you know, on the contrary, like, I did do research on this. I published a research blog about this. It did turn out that the longer you have a model reason, the better it does in a generally constrained setting. So intuition is right, but... But I also think that a lot of people liken this to a Turing tape analogy. If you're computing a program on a Turing tape, the longer the Turing tape, the more you're going to approximate the program you're trying to compute on it, like this classical old computer science analogy.
24:32And that's roughly the case here. I wonder if there's a point of diminishing returns. Like I was looking at, you know, up to maybe 2 ,000 reasoning tokens. But like when you're reasoning for 10 ,000 tokens or more, like there might be a diminishing returns point. And I think it's worth looking into more. But that's also one that comes into training. Like a lot of training regimens these days involve like reasoning length or, you know, token efficiency and reasoning. Like sometimes you want the model to reason less. It's usually there's less room for mistakes. the newest quen series has three reasoning settings and default is extra high but you could do you know medium and low reasoning if there's some cases where you might want that dude i'll
25:13Sam Herring: tell you what i never know how to for whatever task i'm giving it's always high and then every once in a while it's like ah i think this is this could be medium or low we'll see well first of all like the i'm pretty sure the open ai models they don't even have low now starts at me yeah yeah yeah so like uh and then it just goes medium high extra high ultra high like i don't understand when i should be using these i have no idea it's so complicated and please can it just figure it out for me yeah like that is the fact that we have to decide what we want is a little bit for me it's probably like an artifact of today yeah but i get the feeling it's not gonna no i also i mean i never think about it and i don't change the options i'm like i'm always using the highest reasoning because i just don't think about it that's it but then you end up wasting a lot yeah it's i think it's just like a hypothesis that i'll put out there is that that's likely the case is that there's maybe more of a dynamic way you could control reasoning length um and then you would also ask like things I'm excited about and right now the areas that I'm most excited about with all of this actually or even outside of the RL post-training regime inference time improvements you know I think between there's there's a lot of work that comes out in steering and like using these external vectors and activation math to like guide a model in some behavioral direction yeah And then also sampling like the order you pick the tokens.
26:52My earliest research for the token efficiency thing was in sampling. And latest research that I've published was in steering. And it was specifically focused on like refusal steering like safety related work. But I am more interested in steering towards behaviors that could make an agentic model better, especially smaller models.
27:12Sam Herring: Yes. So like an example is like if you're solving a math problem, if you had some way to steer the model towards being more pragmatic, like if you could multiply some of the activations at inference time and be like, hey, you're solving a math problem. I want you to go hard. Yeah. Be more pragmatic. Or if it's doing creative writing, I want you to do more. At inference time. Yeah. It's almost like you have latent lauras or something. Yeah. In essence, it's kind of the same thing. it's like a trained adapter that you can basically choose to add in at inference time and lauras are an extension or sort of a parallel you know adjacent thing to that that yeah you can train specific lauras and hot swap them and apple was one of the first people to publish research yeah i remember that's why it was like oh yeah i think somebody was doing that yeah and you can have that come in and especially when it's on device it's very useful yeah super but then you're talking about it a little bit different and you're saying not necessarily these Lora's that are hot swapping in it's just inside of the actual transformer it's like at the the mechanism that would be multiplying the vectors in is external to the transformer but there's no really swapping it's sort of just multiplying a vector into the last layer of the model or whatever as it's sampling to change the behavior and you would create that when you're rling the model i would we would create that as sort of like an external thing that builds into the inference engine so that's like my vision is that if we are able to serve inference so into like vll yes yeah into like a vllm you know sidecar that basically does this like dynamic whoa and and that i think is like a really underexplored area of improving small models because right now we're kind of trying to rl them to death to get performance gains or distill the shit out of it make them super small so you can have them on edge and then you're it's a different story when you're saying you can still get the value of the small model or of a larger model but with that smaller model maybe you need it on device yeah and i think i don't know i just think again there's some areas of this that are really underexplored yeah uh and so this is what i'm personally very excited about I think most people don't want to touch it.
29:28It ties a lot into like the interpretability field and understanding. Oh, really? Yeah, because if you can control a model's behavior, likely you're getting closer to understanding the components that you're touching, where those behavioral things come from. It works a little bit in the other way where if you do understand a behavior, then you can easily control it. And sometimes when we do the steering, we control something and think that we can explain backwards why it actually exists in the model. Normally not the case. But that's why they're like this Venn diagram overlap of like understanding the model interpretability, steering, like controlling behavior.
30:08And like the middle is when you've hit both on the head and you understand both the components that exist and how to maneuver them. I'm normally focused on the latter. Like I only care kind of about steering behavior, at least in this agenic context. Like I'd love to understand how a model works, but for us it would be really beneficial to be able to just kind of guide it in these ways.
30:32Sam Herring: Yeah, because you can get such a better outcome. And it's just two sides of the coin. It's like one, you're training this behavior into it and sort of designing these RL environments and these post-training environments to like converge into the behavioral basin that you want. And the other, you're sort of sprinkling it in at inference time and choosing to like, you know, and I just have a hunch. I like that analogy. Yeah, I have a hunch that there's these ways that you can sort of do it a little bit more deliberately and a little bit more fine grained at inference time. In my head, I almost envision that animation that you see when you look inside of how a key clicks all of the different pieces of a lock before you can open it.
31:18Sam Herring: It's like you need certain things to be lit up. And so then you get that key and it sticks in and then boom, we now have this model. It's the same model, but because we put the key in it this way, it hits all of these different pieces, and then you can open it, and it gives you that. Yeah, yeah, and every lock is a different, you know, you insert the key for one behavior, and all of a sudden now you've, yeah, that's a great analogy. Or like swapping, you know how like gears kind of sometimes? Yeah. So you have like, oh, the gears fold together like this, that kind of thing. I'm seeing it very visually because it is so it is pretty abstract in a way.
31:57Sam Herring: So I'm like only visualizing it. And it's a little bit harder to figure out the things on it. But, dude. OK, so the last thing I wanted to mention on this was, well, I mean, there's so much. There's so much. And I just got word that we got one minute left. So I'm going to try and like close it up. But you kind of blew my mind on a few of these pieces. so that, damn, as I went into the visual analogy, I started just thinking about all different kinds. But what I wanted to say was my friend Linus, one of my favorite talks that I've ever seen was from Linus Lee three years ago when LLM's first came out and he was doing a bunch of interpretability stuff.
32:42Sam Herring: And he was talking about how we can visualize the inner workings of how the model is coming to its conclusion. And hopefully we can get different ways of interacting with it. So in this talk, he talked about how when we interact with computers, sometimes we will use histograms. Or sometimes when you're like with music, you can have a high pass or a low pass. And you see the ways that you're interacting with the generation or the computer generation in a different way than just like, oh, we're looking at output, like just a bunch of text logs and whatever. And it feels to me like some of this stuff that you're talking about, it would be so cool to see a histogram of the inner workings of how it's going.
33:35Sam Herring: And can we steer it differently? Not necessarily steering it in how we want to type and say it needs to go this way or that way, but can we have this RSI loop training and we are deciding how we want to steer things because of a histogram. I know it's way out there, and I may have not. No, I mean, you're touching on, like, all of this does connect. If you can understand components of it, and the visualizing as a whole, like, everyone's trying to figure out the best way to visualize this kind of stuff. But if you can understand different components of a model and sort of, you know, shuttle RSI learning updates into them, like eventually we will get to what we envision is like true continual learning like a model that can just update itself constantly like we humans learn like that's the goal is we want to be able to understand like how to update one track in the model's mind and not do a full wave update of like every little neuron gets changed and we don't know why like we want to know why and like if i encode some factual information about agent con in san jose 2026 like i want the one pathway in the models, you know, neurons to encode that information.
34:45So I think that talk sounds interesting.
34:47Sam Herring: I'll send it to you. All right. So we're going to shoot over to the next talk, speaking of which. And let us know in the chat. I'm going to go look at it now, what you think. If you're using Hermes, give a shout out to Sam. Thank you for doing this, dude. Thanks for having me here. This was awesome, man. And I'm like, I'm really glad we got to nerd out on it. Yeah, me too. This is great. We'll see you all after this next talk. And I'll see you next time.
Sam Herring’s first real test of Hermes Agent was a shopping tool that matched clothes to photos of ancient pottery. A few months later, the same agent was checking government auction sites every day until it found him a retired three-wheel parking ticket car, which he now drives around San Francisco. In this episode, recorded at AGNTCon + MCPCon North America in San Jose, Sam, who works on the RL post-training team at Nous Research, goes from those everyday wins to the question everyone in AI is asking: can agents improve themselves?
He explains how Hermes grew from an internal productivity tool into one of the most talked-about open-source agents, and why letting it write its own skills was the moment it clicked for users. He also describes an early version that could design its own RL environment, train on Thinking Machines’ Tinker, and hot-swap the new weights into a model running locally. Sam then breaks down Trace to Environment, an idea he credits to Xiaomi’s MiMo-V2.6-Pro work, and the loop Nous is building around it: find the exact point where an agent fails, cluster those failures by type, synthesize environments that recreate them, and train on them again and again.
The second half moves from training to inference. We get into whether reasoning models overthink, why leaning on pretraining knowledge can be a model’s Achilles heel, and why choosing a reasoning effort level may be a temporary quirk. Sam also makes his case for activation steering as an underexplored way to get more out of small models, explains where steering meets interpretability, and describes what true continual learning could look like.
Demetrios Brinkmann: https://www.linkedin.com/in/dpbrinkmSam Herring: https://www.linkedin.com/in/sam-herring-67225819a
Timestamps:[0:00] Cold open[1:21] Meet Sam from Nous Research[2:39] How Hermes Agent started[3:38] From pottery to a parking ticket car[7:40] Benchmarking life admin tasks[8:58] Agents that write their own skills[10:10] RL at Nous a year ago[12:24] Hot swapping self-trained weights[15:00] Xiaomi and Trace to Environment[17:22] Turning agent failures into training[20:27] Where the RSI conversation is heading[21:34] Token efficiency and overthinking[26:30] Steering models at inference time[32:35] Toward true continual learning[34:54] Wrap up