How to build a Company Operating System with Hermes and OpenClaw

0:00Agents are actually building most of our features. Any stakeholder can talk to an agent and validate any feature request they want. Our agent is an operating system because it's hooked into everything from Google Workspace to Granola, Confluence, Jira. It operates at all. It can fetch all of the transcripts from whatever meeting that happened over whichever period of time. Meet Mikhail Shaglov, the CPO at OLX Classifieds, formerly CPO at Azerbaijan's biggest e-commerce marketplace and GPM at Bolt. We've built everything, our entire stack on top of OpenClaw and Hermes agentic scaffolding. I use this knowledge graph to understand how well teams are doing with customer discovery and how well they are connected to the entire organization and the stakeholders.

0:46What's going to happen to my job? What actually will happen is that teams will be smaller, leaner, and faster. Not only it's going to change, the lines are going to blur. What's going to happen to the future? Is product management going to exist in the future? I have good news for you, I think. Before we get into today's show, please take a second to check that you're subscribed on YouTube and following on Apple and Spotify podcasts. If you want access to all of my favorite AI tools, I've gotten them to give you an entire year of their paid plans. Check out bundle.akashg.com for an entire year of Bolt.new, Airtable, Speechify, Descript, Magic Patterns, Linear, Dovetail, Arise, and Mobbin.

1:30And now, into today's show.

1:35I know you guys are overwhelmed with guides on ERMAS and OpenClaw and ClawedCode and ChatGPT. it seems like there are a million AI tools out there. And more and more of you are reporting to me that you're not even feeling more productive after using AI. So I've been searching for those PMs and product leaders who are not just getting like a 10 % boost in productivity, but are getting a 2, 3x boost in productivity. And Mikael is one of those people. What he has built at OLX Classifieds is an entire operating system. If you've been hearing about buzzwords like OpenClaw and Hermes and saying there's no value for PMs, this episode is the one that will change your mind.

2:18He has got his entire team hooked into this operating system that he has built, and they are validating features, shipping features, reviewing features. The possibilities for a PM are endless, and I can't wait for him to share this all with you. Miguel, welcome to the podcast. Glad to be here. So how do people really build a company operating system? How do they measure it? what are the keys to getting the most out of AI as a company? All right. So if you think about this, like what's the core goal of having AI native teams is to be able to automate as much knowledge as we can. Because like in previous, I call this old paradigm.

2:57Like you used to have a knowledge worker, like an employee who has built a context around a certain domain. Let's say it's a deep expertise domain, like search, recommendations, ML and stuff like that. And then all of a sudden this person decides to leave the company and takes all the knowledge with him or with her. He's a bottleneck effectively, a knowledge bottleneck. And the core thing that we're trying to prevent here is the leakage of that knowledge. Because if you could have a single storage of this entire business, customer, product, technical knowledge in one place, then it would increase the entire value of your organization.

3:41That's just one angle to it. And another angle is the better your AI knows your context, the more autonomy you can give it, the higher level tasks you can delegate to it. And let me explain how it actually works in our case. So what you see here is the entire knowledge graph of our company that has been formed over like five months. Like five months is basically nothing. And you can already see this like web of interconnected particles. And these particles, they cover everything from our contacts, who talks to whom, to which projects we're working on, to which customers we've talked about. with which businesses we've reached out, even our funnel metrics, basically everything.

4:33And you can see in the top right corner, like there is a specific metric, which is a product context coverage. And this is one of my personal KPIs, because the higher this percentage of coverage, the more, as I mentioned, you can delegate to it. Because right now, what this means is that it understands our industry, like classifieds, our market, our business, our business model specifically, the top line, the bottom line, the drivers behind it, our customers and the value drivers for them at the level of 54%. which already means that it can operate as a capable, I would say, junior to meet product manager and even make backlog level decisions.

5:23The more this coverage grows, like at numbers of 70 to maybe like 90%, I would not be surprised if this knowledge graph would enable you to do some strategy level work and help business navigate decisions. It's a question of when we're going to get there but we're definitely going to get there. Let me maybe zoom in a little bit on the elements of this knowledge graph. So you have three layers. So the first layer is the product. So all the nodes that are connected to the product side of things. And you can see like product has been very involved into everything. You have the contacts, which is basically all the people talking to each other in different parts of the organization.

6:10And there are like personal things that I share, like my thoughts, my reflections, and the personal things that some of the other people share, like, of course, fully anonymized. Then another layer is the teams, right? I don't really go at the level of actual, like say, cross-functional teams. It's more of a clusters or divisions, right? So if you take buyer and seller experience, you can immediately see how interconnected they are within the organization or like the platform teams or like the paying ship teams. And you can even drill down at the level of a specific product manager. And you can see on a team level, what's their knowledge of their existing context, which for me is a very strong predictor of how good the team is in terms of customer discovery.

7:01Not only that, there is another angle to it. if the team isn't really well connected, and you can clearly see that this team has very little overlapping nodes with the buyer and seller experience, even though a platform is a horizontal layer and they should be, ideally they should be intertwined with each other. You can clearly see that there is some instances of silos happening, which means that teams should talk more with each other. And two, I would also raise the question, of the stakeholder management. Like our stakeholders really being involved in the processes within the teams and so on.

7:44And that's kind of my core tool that I'm using because it shows me the quality of an entire, of the output of the entire organization. This is like the coolest visualization I've ever seen. So what tool is this? How are these metrics working? How are they measuring things? So you can use tools like Obsidian for this. I personally created this one from scratch using Fable because it was like a two-liner prompt, basically. The important question that you asked is a super important one, which is how do we measure this? and we have a very specific prompt that asks an AI what percentage of knowledge around industry and you have to be very specific, what do you imply by industry?

8:43In our case, we have five verticals. We have real estate, we have outdoor, we have good sales, we have services and jobs. What percentage of knowledge around the business, meaning the business model, the actual P &L, the drivers, and what percentage around customers, like different buyer, seller, customer segments, even on a cohort level, marketing, and so on. Which percentage of knowledge do you have at the current point in time taking all of your memories loaded into you directly? And what we've seen is that AI has been quite accurate in digitizing this abstract request into a specific number as an output.

9:26And what I'm also observing is that the more active product managers are, and they typically come with, I would say, transcripts or research studies or like RFD documents, and they load it directly into our agent, into its memory. And I'm seeing that the metric is actually improving over time. It doesn't mean that it's a single source of truth. And I think each organization should come up with a specific context metric that would be highly specific and relevant to them. But this is something that I found directionally works. Here's a quick word from our sponsors. I need to say something that most AI coding companies don't want to hear.

10:08Their tools are not built for enterprise. Think about what happens when 20 people in your org start building with AI. Marketing makes a dashboard. Product prototypes a feature. Here, sales builds a demo tool. Each one looks different. Different components, different styles, no shared design system. And none of the code is pullable. Engineering can't integrate any of it. You've got 20 AI-powered prototypes and zero production-grade output. Bolt.new solved this with something called Design System Agents. You upload your actual design system, your NPM packages, your CSS files, your component library.

10:41The DS agent builds a custom system from your real code. Now every person building in Bolt.new across your entire org is using your components. Your buttons, your tokens, your brand. And when engineering pulls the code, it matches what they already ship with. That's the difference between an AI tool and an AI platform. Check it out at bolt.new slash akash. I used to think I had a retention problem. Turns out I had a messaging problem. I was sending the same onboarding emails to every new use, whether they activated on day one or never logged in again. I had no idea who was slipping or why. Customer.io changed that.

11:12Every message I send is now based on what users actually do in the product. Someone hits a key activation moment, they get nudged to the next one. Someone goes quiet, they get a different path entirely. Their AI agent makes it fast. I describe the campaign I want and it builds the full journey for me. Triggers, timing, copy, even branching logic. And when I want to know how something is performing, I just ask the agent directly and it tells me what to do next. They also have an MCP server, which means AI tools like Claude can see directly what's happening in your customer.io workspace. Your segments, your customer data, your attribution, all of it.

11:46So instead of explaining your business context every time you need help, Claude already knows it. Notion used Customer.io to personalize their onboarding and hit nearly 50 % open rate. Improved conversion by 6-7 % with localized campaigns. And pushed open rates up another 20 % through A-B testing. The idea is simple. Customer.io helps you deliver more impact from every message you send. If you're a PM or founder and your onboarding is still one-size-fits-all, Try Customer.io at Customer.io. Okay, so this is a totally different way to view a product team. Most product teams, most PMs aren't used to this.

12:23If they want to build this, it sounds like there's two things they do. They define the metric, and then they create the visualization with Fable. Let's take a step back here. What is the role of a CPO in an organization like this? What is the new AgentX CPO doing here? That's a great question, Akash. I think what has changed significantly is that in the past, the whole team was responsible for outcomes and outputs. Outcomes were measured in how much your metric has moved and outputs, how much RFD documents, how much code, how much prototypes, Figma mockups your team has been able to produce depending on the function.

13:11Right now, what has changed and changed pretty significantly is that the cost of this output is negligible. But what becomes more important is the quality and the token consumption. So what do I mean by quality? Quality for me is the amount of AI outputs that have significantly moved or moved the metric. So it's essentially outcomes, but outcomes that were driven by AI outputs. And this is how I measure it in terms of the metrics. In terms of the actual tooling, I think previously what CPO was responsible for is building a scaffold, you know, procedural scaffolding on top of the organization. What it means is that you have certain like hiring principles, right, your vision.

14:06Then if you go a level lower, you have certain processes like a planning cadence, like a yearly strategy, then a quarterly strategy, then a sprint planning. And then you have like different reviews like design reviews, product reviews, and so on. And this allowed the organization to function. But right now, what is more important, it's not the process itself. it's creating an operating system within which AI could be a collaborator of product managers and where AI could not only help to come up or challenge ideas but it could autonomously make decisions. And the best way to achieve this is by owning the entire agentic scaffolding architecture and creating specific rituals, creating specific processes, and enabling the team to be in touch with those agents all the time, training the team to use it, actually.

15:08Let me know if you'd like to go deeper into any aspects of this. How does this manifest in the PM process? Where are the savings had for teams? Why should a CPO suddenly be spending so much time as an agentic orchestrator? The biggest reason for this, like if you break down the time that product managers on average used to spend, probably, not probably actually, I estimated this based on my previous experiences, around 50 % of the time of a typical product manager was spent on processes and rituals. You have different business reports, weekly reports, stakeholder management reports, different demos, and so on.

16:01And this is a repetitive type of task that doesn't really require much cognitive input in many cases. It's just something that you have to do manually. And if we abstract all of this and start delegating this part of work to AI, then essentially what you end up with is you have a product manager that is solely focused on discovery, like where the biggest leverage actually is. and a single product manager could actually do the work of two product managers because he doesn't have the burden of all the processes that a typical corporate environment imposes on him. Wow. Okay. So we've seen the knowledge graph.

16:52We understand the role of the CPO and what it does for a PM. Can you show us what this agent looks like in practice, how a stakeholder might interact with it? There are multiple buckets of tools that this agent can solve and we constantly improve it. So the first bucket is status reporting. Like you can ask it, like, what's the status of a specific project? Let's say, what's the status of auto spare parts, catalog, integration, respond in English because he might get finicky. Okay. And while we're waiting for this, there are multiple other things. So status can be, you can get a response in whichever shape or form you want.

17:44Directly in Slack, in Google Doc, in Confluence, whichever format is more convenient for you. Second thing is, it is integrated into all of the workspaces, right? Google Workspace, meaning Calendar, Gmail, and so on. I don't read my email anymore. Like an agent does it for me and pings me in case if it's urgent. I don't manage my calendar anymore. And it's especially difficult when we live in this interconnected world where everybody is remote in different parts of the globe. And just setting a single meeting, you can clearly see it's all there. Scope, listings automated. Yeah, you can clearly see like what's the status.

18:26It's quite straightforward. Nice. And did you like, have you been tweaking the prompts on how it gives status updates? Because this is like a pretty good one. Yeah, yeah. So it has a massive library of rules and imperatives. I think like 700 lines or so to give you the cleanest and the most factual output without hallucinations. Very cool. So we'll go into how people build that. What other use cases should they know about within Slack that they can accomplish here? Like one angle to this, let's say you're a stakeholder, right? And you have a certain feature that you just assume is important to have by whatever reason.

19:08Like a typical route for a stakeholder in this case is to go directly to the product manager and harass a product manager with feature requests, right? But now, like we have a gatekeeper, which is our agent running under the hood. And we train our stakeholders to interact with an agent before reaching out to a product manager. Let's say, yeah, I just woke up and I have a feature request. I urgently want to have a video submission for Autos. A story like an Instagram.

19:56Mikhail Shcheglov: Uh, cause it's fancy.

20:01Cause I saw somebody else doing it, right? That's always the reason. All competitors are doing it. We are behind. Um, yeah, uh, for, for sellers. Yes. Uh, and the, this is just like a typical feature request, which doesn't have a problem framing, it doesn't have any impact estimates. It doesn't have any rationale behind, and it doesn't respect the actual priorities that have already been defined for the organization. And we'll just give it a little bit of time to think. It will respond with a list of clarifying questions. And in case if after answering all the clarifying questions, an agent decides that this feature isn't worth building, then it will politely say so.

20:55If it is worth building then this feature will be added to the backlog and escalated to the responsible product manager. An agent has all the organizational structure mapped out internally. So he knows exactly who the owner of a specific domain is. Wow. And do you consider yourself the owner of the Saul Goodman agent? Yes. A lot of people, they want to outsource that like an AI ops role or something like that. I think it's a great point and it's a dangerous way to go. I think you should own this for two reasons. Like one reason is the speed of iteration. Like if you own this and you're constantly improving and I'm getting feedback every single day, what works, what doesn't work and so on.

21:42And I immediately like open my IDE or my cloud code and I make the changes right off the bat as we go. It's super fast from feedback to deployment. Uh, secondly, um, this actually has an impact on the organization because, uh, one, it saves time, uh, and two, it stirs decision-making, uh, which means it might have an impact on the entire business. And I think, uh, uh, CPO should own this. If you delegate this to an engineering team, or if you delegate it to another person who doesn't have the skin in the game, then you're not going to get the same level of speed and impact. Okay, so CPOs should be building this.

22:22Now they want to know how. Can we open up the covers and show people how they build something like this? Absolutely, absolutely. Let me open up my IDE. Actually, I will show you two ways. Yep, this is just to give an overview. I will kind of walk you through the architecture, the brain, the memory, the tools, the skills, and I'll explain it all as we go. So for the architecture here, what we're using is a combination of OpenClaw and Hermes. Why those two? Because OpenClaw is just a great scaffolding. It has everything available out of the box and great engineering support. But Hermes has a unique feature to it, which is an automated skill generation.

23:13And I've actually tested it across five core topics that my team works. And I've seen an improvement in a recall metric, which means that it was way more accurate in responding using those skills by 31%, which was a dramatic improvement. So I decided to blend both of those scaffoldings together in a combination. And it's jacked. in terms of the capabilities, because we've been improving it for like five months continuously. When it comes to memory, it has three layers of memory. So the first layer is what you've seen is a knowledge graph. It's like an entire universe of interconnected knowledge and everything.

23:59And you can see the code here and so on. But what is plugged in there, it's a second layer, which is a vector database. And the vector database is essentially every single piece of knowledge that an agent receives gets immediately converted into vector. And you need this because most of the requests are fuzzy. And in order for you to get good retrieval, you do need to have vectors because you might ask some random stuff, right, that is not related to any specific keyword. And an agent needs to be able to match your request to a specific point, to a specific number in your knowledge graph. And the third layer, and what I've seen makes agents the most robust, and they don't lose context, is every single conversation is stored in transcripts.

24:55You can clearly see that every single day the agent stores everything into MD files. And not only this encompasses Granola transcripts, because all of our meetings are transcribed, but also all of the conversations that I had with agents and all of the reflection that an agent has. So those three layers of memories, persistent memories, they create this robust knowledge base, which improves every single day. Wow. How does that exactly, how do you architect that properly? So obviously you get a Granola Enterprise license, you have that coming in and recording every single meeting. But I guess you don't want to just save all the meeting details.

25:36You want to strip out some of the personal conversation out of meetings. And then you want to write these more condensed synthesized MD files. How do you go from raw transcripts that might have personal information to useful synthesized output? That's a great point as well. And that was my assumption that you actually have to summarize and synthesize the output to be useful. But we tested this in multiple ways, and it turns out that summarization actually hurts a retrieval. Yeah, for two reasons. Like one is when you summarize something, you lose granular details. And the devil is always in the nuance and the details.

26:20And another reason is that when you summarize, you impose a certain template onto whichever conversation that you had. For instance, like the tasks that were solved, the tasks that remain, what is important, what is not important. And then on top of this, on top of this template, you try to kind of shove your transcript into this template. And we've noticed like a huge fidelity loss. And I think it was like 20-25 % worse recall. That's why my personal architectural decision here was just store every single transcript, every single conversation that we had in a raw form without any summarization because it costs nothing.

27:06Quick thought experiment for you. Is there anything in this video you should be trying on your own? If there is, try it, take a screenshot, post it on LinkedIn or X and tag me. I'd love to see what you're learning. Now a quick word from our sponsors before we get into the back half of the pod. Do you know how to take an AI product from idea, to development, to evaluation, to deployment, and eventually to scale? That's exactly what Product Faculty's AIPM certification helps you do. I even took the course myself. You'll learn directly from Rohan Varma, the product lead working on codecs at OpenAI.

27:39You'll go deep into AI prototyping, evaluations, agents, AI native workflows, cloud code, open claw, latency, cost, guardrails, rag, routing, fine tuning, and production systems. You'll even build your own AI product as your capstone with unlimited one-on-one support. So if you want to stop just learning AI and actually build AI products that work, join product faculty's AIPM certification on Maven. 5 ,000 plus students have graduated and they have 1 ,000 plus reviews. Use code AKASH550 to get$550 off your enrollment. I used to live in Report Purgatory. Every team had a different number. Every weekly review started with someone reconciling spreadsheets.

28:21We stopped hiring more analysts and gave the reconciliation to an AI employee instead. Victor is an AI employee that lives on Slack and Microsoft Teams. It connects to 3000 plus tools your team already uses, ships real deliverables, and every action goes through your team for approval first. It's the closest thing I've seen to a small team running like a much larger one. Let me share three things that Victor does that changed how my team operates. First, Ask Victor has replaced our Monday metric scramble. Someone types flag any customer whose usage dropped 40 % week over week and draft an outreach loom for the account owner to approve.

29:0190 seconds later, the next step is ready. Second, scheduled tasks run the work that nobody wants to remember. Every morning, Victor checks overnight support tickets, drafts replies for the on-call to approve, and escalates anything mentioning churn. Nobody had to ask for it. Finally, spaces ship internal tools in minutes. Ask for a renewals dashboard, Victor builds it, with Authentic Database, and post the link to your channel. The team can stop opening four tabs to get the same view. So stop chatting with AI and start working with it. Get started at victor.com. There's$100 in free credits with no card required at viktor.com slash akashgupta3.

29:39You can find that link in the description. I want to take a second to talk to you about the fourth cohort of Land PM Job. I trained 30 students in cohort one, 50 students in cohort two, and 75 students in cohort three. And I am bringing back a program for cohort four. It starts in August and it lasts three months where you're going to have intense sessions. A Monday morning session where I go over your resume, behavioral interviews, LinkedIn. On top of that, Bart Jaworski is going to be teaching you the PM fundamentals from 2026. How to write AI PRDs, how to AI prototype with cloud code. All of the key skills you need to freshen up your knowledge for this market.

30:15And Ankit Vermani is going to be teaching you AI product management. He is an AI product manager at Uber, and he is going to teach you how to build AI features that actually work successfully. On top of that, Prasad Ready is going to be doing one-on-ones with you for market reviews, LinkedIn review, candidate market fit review. So it is a full package. It is three courses in one for one loaf. So join at landpmjob.com. Wow. And Hermes and OpenClaw on their own can figure out how to get into which meeting and which meetings are relevant context and they won't just fill up their context window with random meetings?

30:50Yeah, absolutely. Because like what happens is that when you ask a request, what it does, it does a search query. And it's a blend. It's a hybrid search query. It tries to do exact keyword matching. If it's not successful at this, which I think 75 % it's not because it's hard. It's a very ambiguous raw context. Then it does the vector search. And vector search allows you to do this exactly fuzzy retrieval based on probabilities. And that pretty much handles the remainder 75 % of use cases. So it only retrieves the relevant pieces of data that is highly specific to your request. And it doesn't really overload the prompt with tokens there.

31:43Very cool. So that's the memory component. What else do people need to know to build this? I think the thing that after deciding on the architectural aspects of this, it is hugely important to create a list of imperatives. And as you know, LLMs are very biased because of how they are trained. And more specifically, they are focused on the resemblance of good output rather than the actual results. You know, and you can do workarounds to solve this. And the best way is to have this list of imperatives. For instance, what for me is super critical is that there's no fabrications, right? There is, if you look here, it's just a huge list of different imperatives that we've created over time.

32:44It's who you are, it's your voice, it's think before you act, because that's a terrible thing I've noticed. LLMs do a lot. They provide you the output that looks plausible, but there wasn't that much thinking behind it, because it's full of contradictions. Then there's always facts over guesswork and anti-patterns. Another angle which super pissed me off a lot, I call this fake helpful. For instance, if you ask your agent to book you a meeting, and suddenly your tokens in the Google workspace has expired, and then the agent says, well, I'm sorry, I cannot do this, but it's super easy for you to do.

33:26Just open up a calendar, type in Google Calendar, name your meeting, choose your time, choose... I mean, it's useless. I mean, it's an obvious advice. You don't even have to waste tokens explaining this. And I call this fake helpful. and if you are able to put it as an imperative it will save you a lot of time as we go but you can clearly see we've made a lot of tweaks all the time so for people who don't know we showed clawed md and soul md what's the difference between those files what should be in what soul md i believe is a part of open claw yeah it's a part of open claw it gives uh effectively like it's a specific context that is being loaded into every single prompt um and clawed md has just the highest priority of them all.

34:13And SolMD is the second in order of priority for open claw. So your CloudMD, it seemed like you kept that under 100 lines, which I think is the advice that Boris Journey, creator of CloudCode, gave. The SolMD, though, is like 800 lines. So that can be longer. Yeah, it's longer. And you might argue that we are overloading the context. And some of those lines might even be contradictory. But what I've noticed was that that's the most robust way that gives me the most accurate output with the best recall. And we test every single imperative. We actually test on the basic queries to see across the most important topics that we raise in the organization, like how good it is or how bad it is.

34:59All right. What else do people need to know to set this up? So another angle is you need to have tools, right? And it has a lot of tools. So since it's interconnected with all of your workspace, it is deeply integrated into Google. So it's a separate tool. It's integrated into Atlassian. It's a separate tool. It has automations. And automation, like the one that we currently have, it just consumes all of the mobile reviews. It writes a digest. Just like a typical supportability team would do. And it even tags the people that are responsible. And it can even raise red flags in case it's necessary.

35:37And these are all Python files. You just prompt these with natural language in the IDE to Claude and Claude writes these? Or how does that work? Previously, I used IDE. But now I'm doing this for showcase purposes mostly. I'm using Claude app because it allows me to do this on the go right here. It is accessible from my mobile phone in case an urgent thing appears and I want to change anything within the agent. So do you basically create a GitHub repo that Cloud Code can access? Yes, exactly. So every single change is immediately committed into GitHub repo. Awesome. So you do everything via GitHub via Cloud app.

36:20That's so powerful. So you can configure people kind of have this mistaken thing that they need to use the open Cloud gateway. but you can actually do everything through Claude and it can manage it on top of Hermes and OpenClaw. Absolutely. So the only benefit IDE gives you is that you can clearly see the structure of the project. Yes. And you went through two IDEs actually. So you showed us anti-gravity and cursor. Talk to us about when you should be using the Claude app versus anti-gravity versus cursor. My use case are the following. So everything that lives in the cloud and doesn't require, I would say, access to my computer, I'm using, I'm interacting with it through CloudApp.

37:06But any specific use cases that I have that require my computer use, and mostly these are around browser, browser use via different MCPs, or Excel use if I want to load a file and get some immediate feedback or data research. If I don't really want to load it into agent and I want to do something ad hoc, I use an ID for this. And why do I have two IDs? Simply because it allows me to have two autonomous separate instances of agents running at the same time. Okay. I think I understand the open clause setup part of this. It's mainly through the sole MD plus open clause, what's giving you that gateway to the Saul Goodman agent in Slack.

37:51Explain the Hermes part of this. You had said it's related to the skills and the recall. So the beauty of Hermes as a scaffolding is that it comes with a unique advantage over Oak and Claw, which is it automatically generates skills based on the tasks that you most frequently request the system to perform. And what we've noticed in testing was that the recall has improved significantly. Well, I mentioned 30 % with those automatically generated tasks, skills, sorry, versus without them. So we have it running under the hood all the time. Like, for instance, team hiring evaluation, immigration, like case building, because we have like different people who are relocating locally.

38:39so I have a lot of questions around it, then it made a decision that we should have a skill for this. The same team hiring evaluation also made a decision, why do I continue doing repetitive tasks? Why not just create a skill and offload all of this onto an agent? So it even kind of takes this meta part of the understanding whether or not you need a skill and makes a decision for you. That's an amazing part of it. Awesome. And how did it help with the recall? Oh, you mentioned it also helped there. Plus 31%. So what does it actually mean? So we have five core topics. Those topics are market, it's a business model, it's product, key growth levers, selection, price, buyer activation, trust.

39:32Then we have more tactical things like funnel and so on. and we have a set of different questions that we ask. And we ask like, I think like 10 questions for each of those areas. And we compared the non-skill response versus the skill response. And we also looked at how accurate the response in each of those categories for each of those questions was. And what we've seen was that having those skills in place gave us plus 31 % more accuracy, which was a deal breaker. How did someone set up like the metrics to measure something like recall? You can do it basically yourself. I'm doing this as a CPO because I think outside of knowledge graph and percentage of context digitization, so to say, you should also be able to track more tactical metrics, which is recall.

40:36and you can ask Claude or whichever system you're working with or your agent directly, tell me which areas are the most frequently asked or which I most frequently interact with and come up with 10 questions per each of those areas which are the most frequently asked. And then let's do a comparison. basically like one control group versus the treatment group with the skills and see what's the difference there. That's pretty much it. To use sort of a derogatory word but not in a derogatory way, you're vibing the metric. You're in natural language, you're explaining exactly how you want it to work and then it's creating the metric and it can go off and measure it for you.

41:26That is correct. But I mean, for the first pass, you have to at least visually look at the results. But once you have a methodology set up, then you can delegate the evals fully to the system. All right. Amazing. So now people know how to set this all up. Can you show us some more use cases? What are the best, most powerful things people should be using this for? One cool thing is this is also around skills, but I think this would be highly practical for people, especially in C-level positions. So I have a board of directors that I'm in constant touch with, and they make investment decisions. How much money we as a company should receive at which point in time, what would be the payback and so on and so forth.

42:17And there are multiple people in the board, and they have different perspectives. But the beauty is the agent was able to abstract their mental models into a set of principles. And we called it the board skill. So in case if you have a pitch deck and you want to defend a strategy for the next year, Then the first thing I do is I run this pitch deck through the board skill to kind of poke holes and give me brutal feedback on what could go wrong with the defense of this strategy. Fascinating. So your Granola meeting transcript is automatically recording your board meetings. Hermes has automatically created a skill around the profile of those board members.

43:07and so when you're creating a board deck and Claude let's say you use Claude design and you would put in your input towards the end of that process you're going to hit it with a prompt like use the board skill to see how the board would respond so that we can refine this is that right that is exactly right fascinating that brings up an interesting point for me we as CPOs we may not want to let our PM see what's going on in a board meeting? How do we make sure that information is locked down so that the right groups get access to the right information? Well, that's a great question. So, and two things.

43:44So the first one, it all boils down to scope of ownership. And the agent has different access rights depending on the person reaching out. And that Access rights will define the context which this person or employee is able to retrieve and the tools which this person would be able to work with. For instance, board skill is available only to me and to the executive committee within the company. And the second thing is the privacy aspect, right? Because not all people really appreciate that their personal meetings are transcribed or recorded. so unless you willingly provide this information to granola we will not train the model we will not create like the skills on top of this so every every employee has a say in this of course i'm personally a part of the experiment that's why i'm completely transparent and i don't care about my privacy at all okay so if you slip in that you uh you know we're taking care of a sick during the weekend, you're okay with that, hitting a transcript.

44:55But if a particular employee doesn't want it, they can elect to take it out. Absolutely, absolutely, yes. From the get-go, we don't really store the transcripts, the personal transcripts of people's meetings because I think that violates privacy. So unless you willingly provide us this information, we will not do this. So you can select, like, this is a product trio meeting. This obviously makes it in. This is a product review, but this is just a one-on-one between me and my designer. This is not going to make it in. Exactly, exactly. Got it. So Bordex is one really cool use case. What else should people be using this for?

45:33Let's see. I'd respond it in a different language. But the thing is, so we asked it, I want a specific feature, and instead he created a prototype. Whoa. Let's see, yeah. That's quite agentic and autonomous. It's logistic and autonomous. And it sometimes gets finicky depending on which language we interact with. And we use different languages. Like, so that's why it might get frustrated at times and it requires an imperative. But here, I think it created a prototype, which is very close to our actual design system. Okay. which is, I mean, it's not great in terms of UX, but as a one-shot pass, I think that's an okay thing.

46:23As a one-shot? I mean, it's just so much stuff, it's done. Yeah, pretty much so. And it's very compliant with the actual Nexus design system, which is an OLX design system at play. And do we know what underlying model it hit? Did it hit Fable, or do you have intelligent model routing? How does that work? So the thing is, there is an agentic orchestrator which makes a decision on which model to use. If it's a complex request or if it's a specific domain area which has high sensitivity or error blast radius, most likely it will assign fable onto it. if it's just an execution type of work or a status report writing I think it will assign opus 4.8 most likely for low level work that doesn't require super accuracy it will assign sonnet so token optimization becomes important here so mostly built on the anthropic stack you're not throwing in a codex or a GLM yet so it's also a great point our engineers actually do because if we look at the typical triad, so engineers have their own agentic stack, product has their own agentic stack.

47:42Of course, they communicate with each other via the same RAG and MCPs. But when it comes to product management, we use quote-quote by subscription, pretty much so. And we don't really have that much of token spend at the moment. That's why it's not a critical metric for us to track. task. While for engineers, it is critical. And they actually have a totally different scaffolding, and they have a blend of different models running under the hood. For the breakdown of tasks, which is typically considered the work of highest complexity, how do you break down an abstract EPIC into tasks? They use the highest cost models like ChatGPT 5.6 or Fable, but for lowest complexity items they use like smaller models or sometimes even locally deployed open source models okay how does this help you with things like your design system pretty significantly let me show you figma so if you look here right um the entire design system was built from from prompt wow um and it's very well documented and the components are robust and what is more important was that the components are exhaustive.

49:00So for instance, you have all of the sizes of the components, you have all of the states, all of the relevant tokens. And it is continuously supported by the agent himself. So in case if during the conversation between engineers and a product manager, there is a disconnect, right? Then because a certain component is missing or somebody tries to hard code a component, for instance, then an agent is informed about this and makes a decision to create this component here in the system. So it's automatically updating the design system based on the work different design teams and product teams are doing.

49:42Yeah, that is correct. And how do you make it auto-updating like that? So one is it has a certain tool on the product side that kind of collects all of the requests that are coming either directly via Slack or from a coding agent. And then it just fires up a development crown job to start building those components on time. Of course, there is a review process where a designer takes over and looks through the components, whether they are correct or not, like how well they are documented. Are there any conflicts? Are the states exhaustive and so on? But that's a review type of a job rather than execution job.

50:33And is the review, is the design system skill in Hermes auto improving? Like is the amount of review that the designer needs to do getting less over time? It's a great question. I'll have an answer by our next podcast. Amazing. So that's design system. The next use case you talked to me about, which I think it's fascinating and I really want to see, is on backlog and roadmap management. What do you use there? So, you know, the biggest problem with backlogs is that every single team has their own textbook. Every single team has their own Excel spreadsheet. There are many apps or tools that try to automate this, but I haven't seen a single successful case.

51:15That's why my understanding of a backlog is more of an abstraction on top of all of the Excel spreadsheets that team are working with. It's basically every single team has their own spreadsheet. I'm not going to open them right now because of the NDA things, but it's the typical type of a backlog. You have a project, you have a problem, you have a solution, different links, impact, effort, ROI, and so on and so forth. Every single team has their own backlog. And they're all interconnected. They're all connected into agent. And an agent can fetch all of this information and can contribute directly to the backlog.

51:54in case if a feature is definitely worth it and passes the ROI bar and passes the review of a responsible product manager. Fascinating. What about personal assistant? How can you use this as a personal assistant? That's my favorite part of the flow. I need to plug in the meeting with execs for one and a half hour next week. not too early, not too late. Suggest something. That's how you would typically talk to your personal assistant.

52:40So right now it does all the retrieval processes that we've talked about and the ideal response is he will come back with a list of different time slots. that would work for all of the execs, who all have busy calendars, who operate in different time zones all across the globe. And I would make a decision on which one to choose. If I'm too lazy, I can even delegate the decision making onto an agent. So while that's running, you mentioned you aren't even checking your email anymore. You weren't checking your calendar anymore. So how is it managing those for you? How is it surfacing the right emails, the right calendar events, getting that prep going?

53:21So every single day I get to digest what's important, what have slipped in terms of Slack messages or emails. I browse through it, or you can clearly see. Wednesday, July 22nd, yeah, clean a slot chief. Let's use it. Wow. So it's applying intelligence. It's figuring out when those execs are available. It's looking across time zones, and you can even schedule the meeting directly to the app. Yeah, I don't need to open the calendar. So I'm getting the digest. I review the digest and I make a decision on which emails are actually worth acting on versus the ones that are not or which tasks or messages I need to respond urgently versus the ones that are not.

54:04And if I'm not feeling like responding directly, I will ask the agent to come up with a template response. Amazing. I know the last area can be a bit of a headache for product leaders. On top of our own job, we need to keep the pipeline of really good PMs. How can this help with recruiting? Oh, actually quite a bit. So it has three tools integrated. So the first tool is LinkedIn recruiter. Let me try and see whether he will be able to respond. the number of product manager candidates we have in the pipeline. Just the number. Okay. LinkedIn recruiter is responsible for reach outs. So if you have a question, I'd say, I don't know, I want a specific candidate with a very specific domain expertise with this many years of experience located in this specific place, just go and find me.

55:11And if the candidate is found, then I will ask to do reach outs using like predefined templates that we've agreed on with an agent before. The second thing is, it is integrated into our CRM. Different companies use different CRMs. um i think greenhouse is probably one of the most uh popular and i don't really interact with the crm anymore because i do all of the work via the agent i just type in like um like what who is in the pipeline or i provide the feedback directly to an agent an agent posts this feedback or moves the candidate along the funnel uh or if needed i can even build build like the funnel statistics to see where the gaps are in terms of the hiring.

56:02And the third aspect to it, which is quite amazing, which is the interview process. Because all of the interviews are transcribed. You can get immediate third-person opinion on whether or not the candidate really lived up to your expectations. And in case if not, Of course, you are the final decision maker, not the AI. It just gives you an alternative point of view, which is sobering in many cases. Then you can ask AI or you can ask an agent to draft a rejection letter and reach out to the candidate. So this automates probably like 70-75 % of the recruiting workflow. And more importantly, it drafts beautiful, tailored rejection emails with really deep focus areas for improvement that were derived from the Granola transcript.

57:03None of the recruiters that I know actually do this well. So you deliver a much better candidate experience as well as making it faster and easier for you and potentially even making better decisions because you have the transcript. Exactly, exactly. Yes. This has been mind blowing. I want to talk to you about some hot topics now. Sure. Let's dive into them. What's on everybody's mind right now, just in the zeitgeist, is what's going to happen to my job. He just showed me how, you know, you might be able to do the work of two PMs with one PM. So what's going to happen to the future? Is product management going to exist in the future?

57:42What's your take? I have good news for you. I think not only product management is going to exist, I think it's going to thrive because I think product managers and product management in large organizations became like burdened with layers of hierarchy, with layers of corporate processes. And in many instances, product management is about compliance, performance, status reporting. So it's suboptimal product management, to be honest. And I think what AI really helps you to do, it doesn't remove away the decision making from the process. It gives you the juiciest bit. It removes a lot of the operational overhead from the work.

58:28And what product managers are now more focused on is the value discovery. Because what AI doesn't really know is what your customers want. Because AI doesn't really understand the customer journey well. Well, it has the funnel metrics, obviously, but it doesn't, it cannot really go offline and talk to a customer. You have to do this. And that's the funnest part in the job. Because you don't do all of this, you know, better anymore. You're only focused on delivering the real value. And I don't think it's going to go away anytime sooner. What actually will happen is that the teams will be smaller, leaner, and faster.

59:15So do you anticipate that the ratio of PM to engineer is going to change from what it historically was? Yes, definitely. I think not only it's going to change, the lines are going to blur between what a product manager does and an engineering team or an engineering manager does. there's likelihood that they could overstep and be a support system. Because at the end of the day, if a product manager is an orchestrator of, let's say, the product quality and token budget, right? An engineer is an orchestrator of an execution quality and token budget, right? The same applies for the designer. Designer is responsible for the consistency of a design system, which means that quality and token budget as well.

1:00:03So the boundaries between those roles would be very blurry. So you're a CPO. You're thinking about maybe I need to create a new product team in a particular area. When I was a VP of product at Apollo, we would typically think about, OK, if we're going to create a new front end product area, we probably want five to seven engineers. We want a PM. We want a designer. When you're thinking about a new product area now, how do you think about staffing that? It's a great question. So I typically divide them into different groups. So the first group is high complexity and high error blast radius. I think for those specific domains, nothing has pretty much changed.

1:00:45And it's all around monetization. it's all around algorithm like search, ML where a single tweak might have a dramatic impact on the conversion or on the outcome here you would need to have a dedicated owner but for the remainder of the domains like customer facing domains what I'm currently seeing is that teams can be easily scaled without adding people. You might have one product manager owning like three, four domains across many platforms at the same time. Okay. So the PMs you're hiring, they feel to me like they're truly AI native PMs. Even if they're not building AI features, you're probably hiring people who are really AI forward.

1:01:38What's your process these days to hire PMs? How do you find PMs that are at this level of AI native that they can actually succeed in this environment? Great point. So I typically look at three things. I think the fundamentals haven't changed, but I also added additional qualifiers into the interview. So the fundamentals meaning problem solving and systematic thinking. Like if you could have both, you will adapt in any situation. But additional qualifier that I've added is a level of craft when it comes to AI, right? How deep a product manager is in the AI. And there is a very simple way to test for this.

1:02:28I'm asking like, which of the use cases of your daily work you have automated using AI? And I get a spectrum of responses. If a response is on a level of, well, I'm talking with ChatGPT using, I don't know, terminal, using web interface. And so to me, that's kind of a low level of immersion or maturity when it comes to AI tool understanding. But on the other side, you might have a very advanced AI agent orchestrator, a product manager who has automated all of his professional work life, right? and build its entire using the eyes and so on. And then I can drill deeper, like asking, like, how do you evaluate this?

1:03:15How do you make decisions on tools, on the brain, on the memory and scaffolding and so on? But I got to be honest with you. It's in the market where I operate, it's a rare skill set. So product managers who are able to effectively answer those questions, they get way more points. You guys heard it from him, not from me, guys. He is a CPO at one of the greatest companies, I would say, to work on. And he is running it in an AI native way. And this is the skill set you need, the skill set we taught today. So if you haven't already, go play around with Hermes. Go play around with OpenClaw. And if you're a CPO, steal Mikhail's playbook.

1:03:56His team is operating at a level unlike many others. Mikhail, if they want to learn more about you, get in touch with you, find your content, where can they go? Well, I do have a Substack newsletter. It's called Corporate Waters. You can reach out and get immersed in my thinking and the actual use cases. Outside of this, you can reach out to me directly over LinkedIn. I highly recommend Corporate Waters. If you guys didn't know, Mikael and I did a really fun deep dive last year where he looked through all of the interviews he's done in his lengthy career, which, by the way, before he was a GPM at Bolt, he was also an ICPM for over a decade at companies like Yandex.

1:04:39So he collected all that data and we looked into what interviewers are looking for. So if you want to go see more information from Mikael, you can find it in my newsletter or his newsletter. Thank you so much for sharing all this amazing sauce today. It was a pleasure, Akash. Thank you for having me. The craziest thing is that Mikael agreed to open source the information that he used to build all of this. So go check the link in the description for the GitHub repo on his profile. You can fork that and you can begin building this company operating system for yourself. Until the next episode, we'll see you later.

1:05:38years we think we've gotten something pretty good going so let us know what we can do to make it even better who else we should interview and we will put on the best shows we possibly can finally don't forget my offer for the bundle you get an entire year of my paid newsletter plus my favorite ai tools bolt.new air table speechify descript magic patterns linear dovetail arise and mobbin That's$27 ,000 worth of value for just$150. So check that out at bundle.akashji.com if it interests you.

1:06:10Mikhail Shcheglov: And I can't wait to share our next episode soon.