Hermes Agent: Agents that grow with you

0:01Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show.

0:41Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a principal AI and autonomy research engineer. How are you doing, Chris? Hey, doing great today. How's it going, Dan? It's going really good. I've been excited for this one, I guess, ever since a conversation that we had back in 2024 on the show with one of the co-founders of Noose Research. Today, we've got another one of those co-founders with us, Jeffrey Connell, who is co-founder and CTO at Noose Research. Welcome. Hi there.

1:22Thanks for having me on. Yeah, it's great to catch up. Of course, big admirers of a lot of the things you all are doing and really enjoyed the last conversation. That was, though, in 2024. So it's been a couple of years and that's like eternity. 20 years in AI time. Yeah, yeah. 20 years in AI time. So I believe if I remember right, kind of at that time, News Research, which for listeners, I would encourage you to go back and listen to that episode, which was at a certain point, primarily a kind of distributed community of folks on Discord and elsewhere. but I believe that was kind of morphing into more of an actual company or organization at that time.

2:05Could you kind of help us understand how that's evolved over the last couple of years, maybe some highlights from the news history books over the past couple of years? Yeah. Like you said, it was two years ago, so 20 years in AI time. And we did sort of start out, like you said, as sort of this loose collaboration of the homies on the Discord, as I always like to say. And that sort of grew organically just through Twitter and other places like that. People who are fundamentally interested in open source AI and making sure that this tech that we think is the transformational technology of our generation wasn't going to just get locked behind a few closed companies.

2:49And at the time, we were sort of looking up at, you know, GBT 3.5 had just come out. the chat gbt thing had happened fully and us who cared about uh open source like how is this ever gonna how are we gonna keep this keep this alive because it seemed like the the sort of curve of capitalization was already starting if you didn't have a ton of money you were never going to be able to to catch up um but what we what we were able to find was that uh because the research space in this is actually quite like at the time especially two years ago was really quite nascent And there were a lot of like low hanging fruit sort of things that could be discovered by just like one or two people, you know, doing independent research.

3:29And we brought together some of the people who had made some of these discoveries. And, you know, it kind of gave us credence to the idea that like maybe, you know, David really can meet Goliath or at least, you know, get on the same field. And so we brought everyone together and decided to say, you know, it was really just sort of a way for us to say we have these people who are doing this for free, basically, you know, on their spare time. Is there a way we can form this into a company so that we could take people who care about this and let them devote their whole, you know, their whole time, their whole experience time to it?

4:00So we brought the company together maybe about two years ago and sort of went through a path of different experimentation. The North Star was always like, we have to keep this tech open and make sure that as many people can use it. And that was always the North Star. But the question was, what was the modality in which that was really going to express itself? We spent some time looking at different aspects of sort of like the AI pipeline as it is and figured out where are the ways we can democratize this. And we started first through what we'd originally done, which is through academic research.

4:33So we spent lots of time, we published several papers that said, like, pick something that's currently extremely expensive in the AI space or is locked behind huge capital costs. And can we do thousand X improvements on that that can bring that back down to the scope of us who, although we have some small amount of money, it's like a rounding error. It's the catering budget at some of these larger places. But it turns out that a lot of things in the A space, because everyone's scaling so fast, you can for sometimes just kind of throw dumb money at it a little bit and it does get better. But if you're a little bit smarter about it, there were big efficiency gains.

5:14So we spent around several years just doing research in that, putting out fine tunes that we that were aligned to the sort of models that respect you and aren't going to be like moralizing and stuff. That was our original Hermes series of models that were very popular. This is, of course, pre-thinking, pre-reasoning, pre-deep-seat kind of style models. We did that and had a lot of success, and people enjoyed that. The academic research, we spent a while looking at, in fact, distributed training and whether that was a viable path and put out some interesting research there about how you could train models over the internet.

5:54You know, this was also in response to some political things that were going on and kind of still are going on, to be honest with you, about like what would happen if people tried to outlaw, you know, even using open source AI and stuff. So we did research on that space, too. And all of it kind of was coalesced around this idea of, you know, needing these thousand X hacks. So one of the one one of the interesting thousand X hacks that we tried to take on, which eventually turned into this thing that is now Hermes agent was the idea of recursive self-improvement. And recursive self-improvement is where like the model is able to somehow train itself or do experiments on itself, you know, and automate that path of the of the pipeline.

6:38because you can think of, you know, high end AI research engineers, you know, the people who really know are like NFL quarterbacks now, basically, you know what I mean? Like, you got to pay them like NFL quarterbacks. And, you know, they are going to decide whether your team gets to the playoff, you know, like they're that important to like your composition of the structure and places the large corporations because the more capitalized, like have better access to this, you know, this talent, limited talent pipeline as well. GPUs are a piece of it, but human brains are a piece of it too. So we said, well, how can we also try to, you know, you know, hack around that?

7:10So we started building this tool internally for our own model research team to automate what some of the research that they were doing and figure out whether there was some, you know, and do that. And that was kind of just like, let's see if we can do this. And that tool was internally just called Hermes Agent. And we started this about six months ago. And it was about six months ago that this was our tool thing. And we used it internally for several months. And that's why if you sort of go back to the genesis of Hermes Agent, if you look at those first commits that were open source, it was basically all around model training.

7:47All of the stuff that we had built into it was all the stuff we used internally for AI research. But we had this idea. And once we had built it, we kind of were like, why are we keeping this secret? We're sort of default open source unless we can say there's a reason not to. So we released this, the repository, Hermes Agent, and it was just a floodgate of product market fit, for lack of a better word. But things were, a lot has been said about what has happened over the last several months with agents and your harnesses and stuff. And I'm happy to, we can talk all about that, but it was kind of like right time, right place.

8:26And over the last several months, it's kind of just like, you know, over every aspect of the company, which is awesome because, like I said, when we were talking a little bit earlier, is that the North Star has always been open source AI, bringing this to as many people as possible. Now, the vehicle, sort of the delivery mechanism that we were going to use to do that, kind of, you know, we were sussing out, but it's really, in the last several months, crystallized for us. So that's sort of the history of Noose in the last two years in maybe five or 10 minutes. So that was awesome. Love that. And you introduced a whole bunch of things we can dive into.

9:03So I'm going to pull you back for a moment. And I actually want to go back for a moment because I think there's something that's kind of tangential to what you were bringing us up to date on. And that's that the story of Noose can be very inspirational, I think, for a lot of people out there who are interested in getting into the space. and they're perceiving the big giants and the massive amount of money that they're throwing at it and stuff like that. And also, at the same time recently, kind of looking at the perception of open source taking a hit like with Llama, kind of coming toward the end of its life and Meta now not champion opening source within the big players and stuff like that.

9:52So could you kind of lay out for a moment, like a little bit of a combo, like how do you see open source working in the modern AI world? And how has, as you guys have formalized into this organization, you know, from that informal network and you're having to make choices and you have this North Star, which is keep everything open. But also, I mean, you have bills to pay, you have paychecks to make and all those kinds of things. Can you talk a little bit about how, you know, not only you guys see it, but in general, philosophically, what's, how do people navigate that space to, to, if you're looking at noose as, as kind of the, the paragon to pursue and say, Hey, I can go do that too, to some degree.

10:41How do you see that? What's the role of open and what's the role of being a company now? Yeah. Well, I can say there's kind of the, the inside baseball, you know, version of open source is like what's actually happening in the industry. And it has shifted several times over the last few years. So we start out with the golden age, your llama comes out, right? And you have one of the fabulous five companies or whatever saying that they're going to do open source AI. It was driven largely by Zuckerberg's personal, I wouldn't say emotional motivations, but he just decided he wanted to do it. And he's in charge of the company.

11:19So that's what was going to happen. Right. And the fact of the matter was, though, is this created an ecosystem of people who were able to research AI and, you know, a ton of the researchers now at places started out by downloading Lama and stuff like that. So it really was quite, quite important at bootstrapping the AI researcher space that we have right now. But as we saw as they released Llama 1, Llama 2, Llama 3, the price tag of that kept going up. And with Llama 4, they made some decisions that were architectural decisions within the model. Seemingly minor thing at the time, could have gone one way or the other.

12:03But they made a few mistakes and the model really didn't come out that well. And, you know, they had to look down to themselves and say, we just spent 200 million, 300 million, 400 million dollars on like essentially a paperweight. Right. And so that's kind of what I think, you know, that the question really became everything's fun and good when everything's working. Right. Like, and I think that first failure case sort of, you know, made people question, what are the motivations for having open source models? Right. So at one thing, you have one of the largest American companies sort of have a come to Jesus moment about what are we spending our money on this for?

12:40How is it aligned with our business objectives? And then you have the deep seek sort of situation, which happens, right, where you have this Chinese company drop a model that was trained and was at the time nearly soda equivalent, come out of nowhere. And now you have this now it goes from being a dollars game within the Western AI space to having a geopolitical underpinning to it as well. And always was going to get here, but like it was forced through there. And you now have open source squarely within, you know, the vectors of this geopolitical of this geopolitical thing. It wasn't just that there was a closed model provider out of China that, you know, was soda.

13:18It was that we had an open source one that, you know, that was near soda coming out of China. And then now this playbook started to get replicated with throughout China as sort of a growth hack mechanism. Right. So within China, they realized that if you could put out an open source model, you could you know, you could get your company, you know, to the top and get it in the press. So insofar as in America, it was sort of realized that this was played out and wasn't going to be profitable as that, you know, fizzled out over in China. It still was quite profitable in the sense to throw your money into open source models with the hopes that it sort of elevated you to this, you know, top tier of, you know, players in the game.

13:57And we'll figure out the money stuff later. And, you know, for different various reasons that may or may not work differently over in China than it comes over here. But the functional difference was that all of the open source was now coming exclusively, you know, the miles are coming exclusively out of China. And so that's sort of the inside baseball. And then and that's more or less where we stand right now, except for what I would call a very significant recent development, which is NVIDIA and Jensen. So NVIDIA and Jensen at his last GTC keynote came out and said, you know, we are going to be the we're going to try to carry the financial flag of Western open source models.

14:36And he committed something like 20 billion dollars over the next several years, you know, towards training open source model, Western prominence open source models. And I think this is an interesting development because if you sort of pull the threads of the sweater about who does it really make sense for someone to be training an open source model, I think NVIDIA is a company where you could actually credibly make that argument that it does make sense for them. It's because at the end of the day, they're all running on NVIDIA. Whether it happens, it all ends up running on NVIDIA chips and more things that need to run on NVIDIA chips.

15:11So for them, it is aligned with their business interests. And anything where you can't sort of eventually really see how it's aligned in the business interest, those things will always kind of follow the money, right? Like that'll tell you, show me incentive, I'll show your outcome. So I think NVIDIA doing that kind of is a potential game changer, at least on the Western side, because it's the full, they're really the only people that has it. Because it's kind of, you know, you and NVIDIA, you win a bit, you win a bit, but guess what? The house always wins. And NVIDIA gets to be like the house here.

15:37So on the inside baseball thing, I think there's some interesting developments right now. And I'm very I'm currently very, you know, enthusiastic or optimistic about where it actually will will go, at least in the short term. Right. You know, we can't predict anything out several several years out. So that's kind of like the inside baseball story. The sort of larger question about how does someone fit into the new story? Is it replicable? Who are you as a person in this space? I mean, I would say now that there's never been a time more where a single person's leverage can be multiplied or maximized.

16:12AI is a human capability multiplier. And I'll give you, like, this is a crystal example. And it is really Hermes Agent itself. So Hermes Agent was first developed internally at Noose by a guy named Technium. You guys might know who he is. He's on Twitter all the time. um tegnium is not a developer um in the sense that he didn't doesn't like really know code that much um that couldn't wouldn't have previous to this sat down and been able to write huge uh like a program that would run from scratch you know um and uh and and so you know it previously he would have been unable to make an application, right?

16:59But, you know, with the current AI tools, he was able to architect and write and build an application that is now the number one open source repository on GitHub. You know what I mean? And like, so that goes to show you that like, if you have the vision and the drive and the care, this technology can make you a thousand X, you know, on what you are. So there's never been a time more where a few people can have outsized impact on the industry and the world than it is right now. And so don't think that the opportunity in any way has passed you by. I think it's really, we're only at the very, every industry is ripe.

17:39Every industry, every vertical is ripe for someone who has laser focused vision, uses these tools to completely take over and win. So I think the area is still, the grass is still green. Wow, it seems like we need to make an update to our website all the time. Our marketing team needs a new landing page or we've got something really cool in our product that we need to create a new section of our website. All of this needs to happen fast and I can't wait days or weeks for these things to go live. That's why I'm so impressed with what our partner Framer is doing. They're an enterprise-grade no-code website builder used by teams at companies like Perplexity and Miro to move faster.

18:23Framer has a built-in robust CMS with everything that you need for great SEO or even AB testing, and changes to your website go live in seconds, not days or weeks. And so if you're launching a new website or you need a new landing page or you need to migrate your full.com, Framer might be for you. Learn how you can get more out of your.com from a Framer specialist or get started building for free today at framer.com slash practical AI for 30 % off a Framer Pro annual plan. That's framer.com slash practical AI for 30 % off. Framer.com slash practical AI. Rules and restrictions may apply. I'm wondering now we've kind of shifted into this discussion about agents and Hermes agent.

19:17I'm wondering about your perspective, Jeff, on kind of, it seems like we've seen this shift around in initial, initially with AI, everyone was like, what model are you accessing, right? What model are you using? I'm going to put in an input and get an out and output from a model. So I need to choose a model. And now everybody's talking about agents or agentic systems. Some of that is obviously kind of jargon and marketing in the market, but also there is a fundamental architectural thing that's happening, which you mentioned agent harnesses and kind of this way that these applications can operate over a distributed system of things or have all these connections that matter to tools and take action and maintain state.

20:03And we saw with, for example, when we looked into, you know, things that people were using all the time, like cloud code and that sort of thing, like the importance of that agent harness, which is also, of course, like a big part of that Hermes agent project. I'm wondering if you could kind of help me and our listeners understand, like, what is the importance of the model now in these types of agentic systems and how does that shift in this world where, you know, is the important part and a lot of the, potential value IP, however you see that in this kind of harness, and the model is just sort of something you can swap in and out?

20:49Or what is the importance of the model within that world? Yeah. So here's an analogy that's not perfect, but I'm finding it to be instructive. So the model is the brain, the harness is your body. Think about that inside yourself. So So, you know, the brain was something we previously couldn't build before. You know, like that is like the true quote unquote magic of AI is still like within the model. It's all within there. That's the zero to one thing that allows any of this to happen. We could build robot bodies before they couldn't do anything kind of deal, you know. So but that's really it. Right.

21:26So the model is your brain and the harness is the body. So let's go on this analogy a little bit more. Right. So suppose you have an IQ of 150, but you're an invalid. You're stuck in a wheelchair, right? That will succinctly limit the amount of things you can affect in the world, right? You can try and go talk to someone and tell them to do it, but there's some amount of limit that you have. And let's imagine, in fact, that most of the work you want to do needs to somehow touch the world itself. You'd be limited in that aspect in some capacities. Meanwhile, if you had someone who had an IQ of 95 but has an athlete's body, you know, there's certain more things that they're able to do just by virtue of being able to manipulate the world more.

22:07Right. So both sides are like important. And I think they exist within a sliding scale. A better harness with a worse model can beat a better model with a worse harness. And both of them are symbiotic and working with each other. And it's just that the harness itself, though, is the thing that touches ultimately what it is that we humans care about, which is the world in which we live in, this thing that we call reality around us, right? The model itself has very little ability to touch the world. All it can do is output tokens, right? And we need to somehow allow those tokens. And we don't have speech and other things, but we need to somehow allow that output to become instantiated into the world around us.

22:47And we ourselves are instantiated in a world where we have persistence, where we have causality, where time moves forward in a direction. And so all of these pieces, the harness sort of allows the model to exist within the reality that we all care about. Now, as far as like ownership of, you know, where's the value accruing or all that, those sorts of questions. I don't want to get ahead of my horse here and like make some sort of like, you know, Silicon Valley pronouncement about, oh, you know, this is where all the money goes or something like that. I would phrase it rather in a different way.

23:20I would say we are getting to the point where people will want to pay for outcomes. Right. Just as in when I hire an employee, I am paying him to do a job. And ultimately, I want that job to be done. Otherwise, I wouldn't have hired that employee. The prime motivating factor for me is in getting some sort of material, economically valuable outcome to be done or something I view as economically valuable. And so I think that is also the similar model that will eventually pull out of this whole AI system, which is that these systems will be paid to do certain things and they'll be valued insofar as they get the job done.

24:00Now, who owns the harness? Who owns the model? All of that will kind of get worked out in the end. But again, if you pull back and say, what's the prime motivating factor? It's I want to pay for something to get done in this world and it would be cheaper, it's cheaper, faster, more cost effective for me to do it this way than to do it any other way. And that's water going downhill. Follow that path that'll take you to the promised land. Well, Jeff, I love that analogy about the brain and the body and the harness and the model, I think that will be really helpful for people. I'm wondering now, in terms of Hermes agent specifically, could you kind of, I guess, walk us through some of the agentic or automation stuff that you were seeing kind of out there as you all were starting to develop Hermes agent and some of the key opinionated choices that you made about what you want Hermes agent to be uniquely in the midst of kind of all these other things out there.

25:06Yep. And really the key piece of it was the idea that the agent ought to get better the more you use it. That was kind of like the motivating factor here. And that's, you know, maybe different than how people used to design software. So we've always been trying to like break out of like the paradigm is different. We have to act like the paradigm's different. How did people use to design software? They would say, I think I know a problem. I think I can design the best solution for that problem. They're going to whiteboard it all out and have all these systems. We said, well, we don't have to be like that anymore.

25:41The ARs are smart enough to just see what we do and then figure out what they should be doing. What we said is just get out of the way of the model. Give the model the ability to do what we want it to do and then lean on it as much as possible. So a couple of the opinionated decisions was we created a very limited number of sort of bundled in hard coded features. We gave it the minimum sort of ability to touch the world, to run code jobs, to browse the web and stuff like that. And then everything after that is an emergent property that is encouraged to develop through the prompts. And that is the two main ones are sort of the memory system and the skill system.

26:27The skill system is extremely interesting to me because it is the thing that the first time it happens to you, it feels like magic, which is that you use the agent. You're trying to solve some problem. It does something for you in the way you like it. And it notices without you telling it that this is like I've done, I've learned there's something important here. There's something important here. And it will create a skill that says how to do a specific thing. And then the next time you do something that's sort of like that, it will just use what it figured out before on how to do it. So I have an example of this.