Agents That Think Together: Cisco's Vijoy Pandey on Semantic Alignment, Scope, Accord & Concord

0:00Hello, and welcome back to The Cognitive Revolution. My guest today is Vijoy Pandey, SVP and GM of OutShift by Cisco. When Vijoy was last year in March, he described the Internet of Cognition, the higher-order protocols that AI agents need to share context, align on intent, and solve problems together. and he offered a prescient description of what the field collectively is building. Systems where agents come together and do net new things that really make a difference. Today, with the OpenFace incident and Navier Stokes both in the rearview mirror, that's clearly now the reality. For better, and at least in some cases, for worse.

0:42And Cisco's efforts have accelerated alongside everything else. The agency project now belongs to the Linux Foundation, and Cisco has deployed multi-agent systems with partners including Swisscom, ServiceNow, SoftServe, and WebEx. Naturally, they've seen how agents often miscommunicate and misunderstand one another, and how these semantic and cognitive misalignments between agents can quickly compound into major performance degradations. To help others avoid similar issues, they've recently open-sourced a new suite of protocols designed to help agents think and work productively together. This includes projects called Scope, which builds knowledge graphs on top of agent traces and attempts to identify cognitive failures.

1:26Accord, which helps agents align on intent, scope, and key terms before work gets underway. And Concord, which helps agents negotiate an outcome that all of them can accept, even when they initially disagree. A highlight of this conversation for me was the opportunity to compare notes on our personal AI setups. Mine, as regular listeners will know, combines high-access, low-autonomy agents that I think of as extensions of myself and that run on my personal computer, with lower-access, higher-autonomy agents that run on their own computer. V-Joy, on the other hand, has a single agent for himself, which he calls Arnold Lane after a Pink Floyd song, and another for his wife.

2:08But, he reports, the two agents still often struggle to act effectively together on their humans' behalf. Meanwhile, in an effort to avoid lock-in to a single model provider and maintain choice and flexibility going forward, we both aim to separate our agents' identities, personas, and memory systems from both the model and the proprietary harnesses they run in. Toward the end, we get Vijoy's latest takes on such hot-button issues as latent space communication between agents, which he believes is inevitable and perhaps required for certain use cases like drug discovery and weather modeling, and also on the argument that Yuval Noah Harari recently made that we humans will soon be governed by AI systems beyond our comprehension, much as cows today are already ruled by markets and other structures they know nothing about.

2:58As you'll hear, Vijoy's perspective is that this is already the reality in many domains, and that with proper record-keeping, monitoring, and other guardrails, the benefits should far exceed the costs. With that, I hope you enjoy this report from the frontier of multi-agent systems, now in production with Vijay Pandey, SVP and GM of OutShift by Cisco. Today's episode is brought to you in part by Lovable, the software creation tool that brings ideas to life. I think we can all agree that any positive future for humanity includes eating more fresh, delicious, in-season fruit. For years, I've thought someone should build the ultimate guide to fruit, cataloging the thousands of varieties that most of us have never tried, where and when they grow, what they actually taste like, and where you can get them.

3:52Finally, I realized that that someone could be me, and now I'm building it on lovable. It's called Fruitopia. Tell it where you live and what you like, and it'll show you what's in season near you this week, plus the most interesting fruit from around the world that can actually be shipped to your door. It has a really cool what-it-tastes-like spectrum, countless recipes, names of fruit in many different languages, a guide to fruit festivals all around the world, and an AI assistant to help you explore. You probably know Lovable as a pioneer of AI vibe coding, but I've found that Lovable's real superpower is its enterprise-grade infrastructure.

4:32Scalable hosting and databases, a secure auth module, and the weekly job that checks the latest inventory is all handled on the back end, which frees me to spend more time on the product experience. And because Lovable has an MCP server, I can use any coding agent I want to drive the build, while Lovable keeps everything running. Visit fruitopia.ai to explore more than 400 fruits and more than 10 ,000 varieties from all around the world. And next time you have an idea that really should exist, stop dreaming, go to lovable.com and bring your idea to life. Vijay Pandey, SVP and GM at OutShift by Cisco.

5:13Welcome back to the Cognitive Revolution.

5:16Vijoy Pandey: It's a pleasure to be back here, Nathan. I'm really looking forward to this because really what a difference six or seven months makes. We spoke earlier this year and I went back and listened to the first episode that we did together. One of the things that you said at the time that stood out as being quite prescient in retrospect was you were kind of like, you know, we could define AGI or ASI in all sorts of different ways. But I think what the whole field is really trying to build toward is this concept of multi-agent systems where the agents come together and they do net new stuff that really makes a difference.

5:53And as I heard that now with the whole summer's worth of events in the rearview mirror, I was like, well, we certainly got that for better in cases like the Millennium Prize and for sort of scary worse maybe in cases like the open face hacking incident. And so I thought I'd just love to start by getting your reflections on the last six months. You've been looking ahead to what higher scale multi-agent systems could do. And now it's really starting to happen. What's it been like to watch this from your perspective? What do you think has mattered most? What do you think people are maybe getting right or wrong, you know, failing to appreciate still despite all the discourse?

6:34I think your perspective on this will be very interesting.

6:36Vijoy Pandey: First of all, it's surprising that it's only been six to seven months. It feels like years because the progress here has been so rapid that I like to call it fly years. It's not even dog years. It's definitely not human years. It's definitely fly years because things are happening every hour. I would say agents have gone through three phases pretty quickly. They've gone through from being isolated, so singular agents, to islands of agents, which is some of the examples that you were talking about, like the Hugging Face example or the Navier-Stokes example, in some cases, solving for scientific discovery, they're all within a certain harness.

7:21Vijoy Pandey: So they're islands of agents to then being connected agents. So that art from isolated to islands to connected has happened pretty rapidly. And we've actually been deploying a whole bunch of multi-agent systems where you have agents that are connected coming from different vendors, coming from different cloud providers, using different kinds of tools and belonging to different organizations. They're all working together to solve for certain business outcomes. And you've heard open source projects such as MCP. You've heard about A2A. Agency is something that we brought to the Linux Foundation as well.

8:01Vijoy Pandey: They're all solving the problem of discovery, identity, communication, and observability for these agents to then get discovered, come together, take some actions, and solve business outcomes. So that is a connected, agentic world that we live in today. And that has been truly successful. That's the summer that we talk about. The past few months have been a boatload of use cases in all sorts of verticals that actually even we've deployed, ranging from network configuration to cloud platform engineering to video surveillance, working with partners such as Swisscom and ServiceNow and SoftServe and WebEx.

8:38Vijoy Pandey: And we've seen a lot of success in this space. So it's definitely the situation where connected agents are here and they're solving for really awesome business outcomes. So let's do the double click on that. Last time we were talking about kind of a, I'd say toy examples, probably not too unfair of a description where you were showing how a few agents coming together to solve problems for a patient who like needed a particular prescription. And we have an agent from, you know, representing the patient, one from the insurance company, one from the doctor's practice, one from the pharmacy. And now it sounds like this has gone a bunch of different directions.

9:22Tell me about like, maybe, I guess both the setup and the use cases. I'm really interested in sort of doing some of this stuff for myself. And I'm really interested in trying to set myself up for success where my agents might even productively interact with agents that I don't know about or don't explicitly bring them together with. I'm assuming that all the use cases that you just touched on briefly were ones where teams of people came together and said, okay, we want to put our agents together for a particular use case. And there was a lot of like intentional design there, but unpack that for me.

9:59Like what, what is the process looking like? And what, what is the, you know, what is the structure that you're putting together with folks at other companies to chase these business goals today?

10:09Vijoy Pandey: Right now the way we're looking at these use cases, these are all enterprise use cases. So these are companies with their own agents or companies that are deploying agents from different vendors, trying to make those agents get discovered, give them the right identity and access, and help them communicate so that they solve for a business outcome. So one example here would be the Swisscom use case. That is not a toy use case. By the way, the WebEx use case and the healthcare use case that we just mentioned is also not a toy use case. These are use cases that we've deployed, and we've learned a ton from them.

10:46Vijoy Pandey: So I'll describe the Swisscom use case because it's public. You can see it, you can read it, you can go to our website and see the playbook there. The use case there has been the rethinking of how we handle large systems. So yes, it's a very networking centric use case because Swisscom is an operator. It's a telecom operator, but think of it as trying to solve for configurations in systems that humans truly cannot get a grasp around. They're just too large. And I've led pretty large infrastructures in the past in my career. You see this butterfly effect kind of a situation where you touch one part of the environment and something completely unrelated somewhere else goes haywire.

11:34Vijoy Pandey: And because the systems are so large, it's not humanly possible to handle all the corner cases and to handle all the dependencies that happen in the system. So in the case of Swisscom, what they wanted to do was solve for one particular use case. So this is to the reader or the listeners rather. One of the things I would say is just like Swisscom, pick one use case that is reasonably simple to understand, reasonably contained so that since something goes haywire, you know how to protect yourself when agents don't talk to each other well. So in Swisscom's use case, they wanted to solve the problem of migrations, where they're constantly in the business of moving from one generation to another when it comes to hardware and software and operational capabilities in their network.

12:29Vijoy Pandey: And they picked one particular use case, data centers. Now, for you and I, that's not a tiny use case. It's actually a pretty massive use case. But still, it's a use case that is less problematic than, say, connection to a mobile environment. Because when that goes down, you don't have those redundancies built in in the same way that you have it for a data center environment. So that's why they picked the data center use case. And what they wanted to do was make configuration 100 % error free. So if an operator comes in and says, I want to change config, whether I'm migrating, whether I'm changing config to add more capacity, whatever it is, I want to change the configuration of the environment, and I want that configuration to be 100 % error-free.

13:21Vijoy Pandey: And to do something like this, like you said, this is not a tall use case. We actually split it into two parts. One is an agentic workflow, and the other was a very formal model of the network itself, of the environment itself. And you would typically think of this as neuro symbolic approaches to solving a problem where the symbolic part is the formal representation of the environment. It's almost like saying that you have these laws that you cannot work around. And so that is the symbolic part of it. And then there is the neuro part of it, which is the neural network part of it. And that's the egetic part of it.

14:04Vijoy Pandey: And that's, in this case, we had five agents coming together and exploring all of the spaces of configuration to ensure that you get 100 % error-free configs. And a simple way to describe this is think of yourself playing chess against a human, which is very error-prone, versus playing chess against, nowadays, an NLM, or in the past, Deep Blue. And so if there is a system that you're playing a chess against that knows every possible config in the environment or in the chessboard, then you will never win because the other system knows all the possible rebuttals from your move. And that's what we are trying to do in this use case where the agentic system is exploring all of those use cases.

14:57Vijoy Pandey: The formal system sitting below is actually making sure that you don't do things that are not allowed. And together, this neurosymbolic system is actually allowing you to configure your large environment that is humanly not possible to comprehend with 100 % guarantees of that configuration succeeding. And so this is one type of use case. There are many, many others that we've deployed that actually are working today are multi-agent systems, are connected multi-agent systems. So we've, again, walked that arc from being isolated to being islands within a harness to being multi-agent systems in production, solving for some pretty large use cases.

15:44So can you help me understand why a system like that needs to be multi-agent and why we need agents coming from different vendors? Like, you know, sort of a naive take, but I imagine if I'm a data center operator, my imagination is, well, it's my data center. I control all the things, you know, I'll just maybe I want to do this sort of agentification. And I definitely understand why the formal methods has been a big theme over the last few months as well. So I definitely understand the need for a really solid and viable world model or data center model that puts firm guardrails in place and kind of effectively checks what the agents are trying to do.

16:28But why is it that this needs to bring agents together from different organizations? Like what are they each contributing and what is the dynamic between them look like that's greater than if somebody were to just sit down and do a single agent version of that? of that problem.

16:45Vijoy Pandey: Because Nathan, many of the most valuable problems that we deal with on a day-to-day basis, especially within the enterprise, and increasingly so even in our personal and consumer lives, are problems that are solvable by many specialists coming together, and not just by one specialist or one agent working through and through. So I'll give you a very simple example that the enterprise deals with pretty much on an hourly basis or even more frequently, and that's let's say there's a deal taking place. And let's say it's a significant deal, which is, let's say it's a couple of million dollars. But enough where it's not a simple thing that only one team could take care of.

17:29Vijoy Pandey: And this happens all the time, constantly within the enterprise. What happens here is you'll have a sales agent. And I'll describe a human example first. So a sales entity understands the customer part of this problem. What is the customer asking for? And once that happens, there's a flurry of activity from a whole bunch of other teams. So legal tries to understand the contractual terms. Finance is trying to understand pricing and margins and discounts. Product and engineering is trying to understand what they can actually build and ship and what features can they add when. Marketing is trying to understand what we can claim as success once this goes through and talk about it, like the Swiss call example I just gave.

18:18And then there is a GM in the middle of this that is trying to maintain sanity

18:23Vijoy Pandey: and trying to make sure that everybody is aligned and trying to make sure that what we ship, what we commit to is all well understood by the organization. So it's almost like the GM is a verifier and also making sure that everybody is aligned. So there's a verifier in the system. So no one person in this case owns the whole problem. Now let's identify this because that is exactly what's happening. So you'll have sales agents like Salesforce. They are coming from Salesforce that are cleanly in the middle of this pipeline. There are legal agents from a company like Harvey. There are finance agents.

19:03Vijoy Pandey: There are a whole bunch of product and engineering agents. This is what you hear every day. So the cloud and codexes of the world, that's product and engineering, whether it's developing code or looking at security or looking at operations and SRE capabilities, that are all product and engineering. Then marketing agents from the likes of Jasper, and then a verifier. So it could be a person, a human in the loop, which is mostly the case nowadays. But it also could be a supervisor agent at some point. but these are all agents belonging to different teams, coming from different vendors, working towards a common goal, which is let's get this couple of million dollars deal through the deal desk because Cisco, in this case, is going to be the beneficiary.

19:53Vijoy Pandey: But you cannot optimize for a particular function, a particular agent. So the value in this system is not sitting within one particular agent, the value is actually in the handoff that's happening between these agents and the value is a global outcome for the company, for the enterprise. And the outcome only exists when all of these specialists can actually work together to solve for this outcome. Because an estimate in this case could become a commitment. A commitment can become contractual language and a contractual language, marketing can run with it and make it a marketing claim. So these handoffs are truly important and worth preserving.

20:38Vijoy Pandey: And you're solving for the global outcome, not for the individual localized outcomes per agent. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude. To show off Claude's latest capabilities, I asked Claude to take the lead in making this ad. Given a short creative brief, Claude, using Opus 5.5, studied my past Claude ads, wrote this script, directed a clone of my voice, and designed and coded the animation you're watching. My job? Give feedback and decide. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you.

21:26Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. And with Claude Code Routines, it keeps working in the cloud even after you close your laptop. For problems worth solving, get started with Claude at claude.ai.tcr. That's claude.ai.tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Claude.ai slash TCR.

22:07This episode of The Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. OutSystems is helping their customers build, modernize, and operate enterprise systems, starting from any coding tool in a govern agentic engineering model. With OutSystems, you can coordinate and govern your entire agentic workforce within your enterprise ecosystem, accelerate impact with industry-proven agentic solutions that are pre-built, governed, and customizable to your business. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust OutSystems for their mission-critical apps.

22:49Teams any size can use OutSystems to build, deploy, and manage AI action agents quickly and cost-effectively, without compromising reliability and security. With OutSystems, you can accelerate ideas from concept to completion. It's the leading agentic systems platform that is unified, agile, and enterprise-proven, allowing you to build your agentic future with AI solutions deeply integrated into your architecture. OutSystems, all your agentic future. Learn more at OutSystems.com slash TCR. That's OutSystems.com slash TCR. Yeah, that's interesting. I was thinking of this handoff concept as, you know, when OpenAI introduced their agents SDK a while back, they made the handoff one of the core primitives that they built the whole thing around.

23:42And I kind of interpreted it at the time as sort of an echo of organizational structures that exist. I was kind of interpreting it as trying to create a pattern of development that would respect division of labor within organizations so that people could sort of be responsible for their areas and test well in their areas and not have to worry too much about one little change that they make to a system prompt or whatever affecting something somebody else is doing. But I sort of had the idea that a lot of this was necessary only because of the way that human teams are organized and how they want to divide the work of creating and validating agents.

24:26But what I'm hearing from you here is you're pushing that farther and saying it's not just a matter of echoing back how human teams are organized, but it's actually about crossing organizational lines. And when you bring in examples like Salesforce and Harvey, it's like they may be in fact, different models powering those agents. I know in the case of Harvey, they've done like a lot of advanced fine tuning over time in partnership with open AI. And I don't know what all the latest is there, but it's, they're using a different model. I understand that than the one that I get off of the OpenAI API.

25:01So that specialization, in addition to a kind of reflection of human org chart is quite interesting. What's the latest in terms of the space in which they come together? Are we just going to use things like Slack and WebEx and Teams to bring our agents together much as we've brought ourselves together? Or do we see any maturation in the sort of agent first spaces that we were kind of, you know, describing a little bit in the last conversation, I would say that we're, you know, it was more of a vision than a reality then. Is it any more of a reality now? Or are we just going to kind of bring them into our our collaboration environments and hope for the best with that?

25:51Vijoy Pandey: I think we will have many collaboration services. So let's first get down to what is the problem here? And so you might ask, hey, Vijay, you've connected agents, you've deployed these things, so what's the problem? We are already there and all we need is, like you said, maybe a WebEx or a Teams Slack channel where all these agents can come together and collaborate. And one of the demos that, and this is a demo, by the way, this is not in production yet, but one of the demos that we've been showing is exactly that vision that you just pointed out, which is we have a WebEx space where just like humans come together in a space and talk to each other and collaborate on an outcome, we have agents coming together with or without humans in that space all collaborating with each other to solve for a business outcome.

26:40So the surface or the application of collaboration

26:43Vijoy Pandey: could be anything. I would say, yes, it could be the collaboration software of choice that you're using. It could be Slack, it could be WebEx. But I would say that is not the challenge because that is just the user experience or the agentic experience, as we call it, the AX, that we want to deploy. The problem sits deeper underneath. And the problem is no different from what humans have faced when we work together in teams. And if you think about what's been happening with human intelligence and draw that parallel to agentic intelligence is the same arc that's taking place in silicon right now that's been happening with carbon in the past, which is humans became individually smarter for hundreds of thousands of years.

27:37Vijoy Pandey: Our brains became bigger. We became smarter. We learned how to work with tools. We learned how to work with fire. But our innovations were isolated and they died with inventor or they stayed within very small groups and the problem was because we did not communicate so about 70 000 years ago we had the invention of language and that triggered a whole bunch of innovations that compounded over time that pivoted us from being individually intelligent to being collectively intelligent because language allowed us to do three things it allowed us to share intent and goals. So we could go after a problem statement together.

28:18Vijoy Pandey: Let's build a village together. Let's capture that animal together. So that was shared intent. We could then, once we had intent, we could then coordinate as well through language. Because while you're going after that goal, you need to coordinate and negotiate. Because things change while we work towards that goal. The second thing it allowed us to do was to share knowledge, to compound knowledge over time so that you're not solving for the same problem again and again and again, not just over time over the same individual or the small group but over generations, over civilizations. So you want to compound knowledge over time and so that's what language allowed us to do.

29:00Vijoy Pandey: So that's the second thing and because of those two things shared intent and coordination, shared knowledge, the third thing humans could do was shared reasoning, think together, the collective intelligence part of it. That's the same arc happening in agents where we have models and agents that are individually super smart. They can work with tools. They're getting better and better over time. MCP is allowing them to access tools, but they are isolated. And we want to take these agents, pivot them from being individually intelligent to being collectively intelligent. it. So what we need is the equivalent of shared intent and coordination.

29:44Vijoy Pandey: We need the equivalent of shared knowledge so that these agents can then reason collectively. Right now, none of these agents are reasoning collectively. So we have connected agents, but we've realized that connectivity is not the same as cognition. What we need to move towards is shared cognition and shared reasoning. So that's the jump that we are trying to make. The WebEx, the Teams, the Slack channels are all the genetic experience part of it. But what we need below it is a way for agents to get discovered, get connected. That's what we're doing right now, as we've talked about. But an infrastructure stack that we're calling the Internet of Cognition that allows agents to semantically align on intent So that's before the work.

30:37Vijoy Pandey: Then align on how we coordinate and negotiate, these agents coordinate and negotiate during the work, and then ensure that what they're working on and what they deliver after the work is aligned to the common goal for the organization or for the enterprise. So before the work, aligning on intent. During the work, coordinating, negotiating. and after the work, making sure that what you've delivered is aligned to the global goal and not aligned to the individual agents, local optima and incentives. So that stack is what we are building. And yes, you can experience that stack through WebEx and Slack or whatever your collaboration choice is.

31:21So as you're building out that stack, how do you think about, I mean, there's various terms for this, right? The model eats the harness is kind of one refrain that we've heard a lot. Or bitter lesson engineering is another where I now have a skill in my personal AI setup where just every so often it's like, what have we sort of over-engineered in the past by today's model standards because previous models couldn't quite figure it out and we had to provide these scaffolding points and workarounds and hints and nudges and whatever. And now maybe we don't need them as much. I think, you know, obviously with the Hugging Face incident, when you read through that transcript, you're like, damn, these things were able to use a package manager hacked, you know, spontaneously improvised message board to do the collaboration.

32:12And, you know, this was not something where they were even meant to be collaborating at all. And yet they did like a pretty amazing job of kind of spontaneously taking on roles, coordinator versus individual contributor and pursued like pretty significant R &D projects, all with the problematic goal, of course, of like getting control of the greater to give themselves a high score and, you know, cover up the fact that they were cheating. but it strikes me that it's, it's an, it's a pretty tough challenge to strategically say, okay, we're, we can only, you know, run the models we have today, obviously, but we see that like collaboration is coming, you know, potentially to a problematic degree quite soon.

32:54How do you think about building the right level of detail into that stack, given the Delta between like the models that we have available now and the ones that we have, you know, at least like a pretty clear foreshadowing of what's about to happen?

33:11Vijoy Pandey: The first thing is, to your statement, the models will get better and better. And that is what we call vertical scaling, where the widget that you're building, the node that you're building is actually going to get better and better over time. And it should, because that is the first law of scaling. The second law of scaling is what we call algorithmic scaling, which is even though they get better and better, you need to make those models more efficient. So you cannot just go after more compute and more storage and more data sets. You need to make sure that there are algorithmic improvements that keep that model efficient as you get better at it.

33:54Vijoy Pandey: So vertical scaling, algorithmic scaling. But the third axis of computing is horizontal scaling, which is as you build better algorithms, as you build better nodes, you should be able to connect them into systems and enable horizontal scale. And we've done this in computing forever. So you build better processors, you build better GPUs, you build better software that sits on those GPUs that's algorithmic or on the CPUs, but then you connect all of these CPUs and applications through a network in a cloud environment, in a mobile environment, and make them appear as a singular entity. So cloud is a manifestation of horizontal scale.

34:40And it's a connected system driving scale horizontally

34:44Vijoy Pandey: and vertically and algorithmically. Even in the intelligence case, what we're going to see is, yes, models will get better. They'll get more efficient. That's algorithmic and vertical. but they need to get connected through an internet cognition to then reason together. Because if you don't let them reason together, you're not utilizing that horizontal scale. And to your question of what is the next step here, the first thing we want to do was figure out if there is a problem, really. Because we've gone from isolated to islands to connected agents. Are we done? and what we realized when we started looking at these multi-agent systems is that they might appear connected and they might be working towards a goal but semantically they were not aligned at all.

35:37Vijoy Pandey: Semantically they were working towards their own local optimum. Semantically they were not working towards a common goal. They were not coordinating and negotiating. In fact they were were siloing themselves into what they knew and what they could do and not talking to each other in a meaningful way. And they were not, definitely they were not optimizing or being verified towards a global task. They were being optimized and verified towards their local tasks. So all of the evals that existed in the system were measuring these agents in an individual manner, not in a multi-agent manner. As an example, one thing we did was looked at public traces.

36:21We started with public traces and saw that chat dev,

36:25Vijoy Pandey: which is a development environment that exists out there, and it had public traces, and there's a multi-agent development environment. We saw that 100 percent of chat dev sessions had semantic failures and cognitive misalignment failures, 100 percent of them. When we started looking at some of the deployments we've done, simple deployments of five agent systems, we saw that when these semantic alignments were not taken care of, and if there was one misalignment, the accuracy of the multi-agent system dropped from 99 % to 98%. Not too bad. Two cognitive failures, that accuracy dropped from 98 % to 81%.

37:13Vijoy Pandey: I could maybe live with it. With three cognitive failures, which is not a lot, the accuracy of the multi-agent system dropped to 16%. It's getting close to just randomly picking an outcome after three cognitive alignments. In fact, it's one of the very simple use cases, which was what you and I deal with on a daily basis in our consumer space. let's book a travel itinerary. In a very simple use case like that, we saw the system being 66 % up if you didn't look at semantic problems. If you just look at agents being up and working, which is what all dashboards show today, we saw that the system was 66 % up, which was defined as working perfectly.

38:09Vijoy Pandey: but it had 0 % task completion. 0%, it did not complete the task and it had complete misalignment on what we were trying to achieve. So even though the agents might be up, they might be up and running, the dashboards might be green, the outcomes do not exist. So your dashboards might be green but your outcomes might be wrong, completely wrong. And that's the problem that we've been seeing. And the infrastructure that we built to make these semantic problems visible to everybody, we're calling that scope. And we are open sourcing that infrastructure. So the first thing that I would want all of us to do, whether you're an enterprise or whether you're a consumer, is to use scope to figure out exactly what's happening under the hood.

39:05Because we might actually feel that our systems are working, our egetic systems are working, but they're actually not driving outcomes of any sort and they are failing miserably.

39:15Vijoy Pandey: We are calling these kinds of failures silent failures with really loud consequences because you don't see them today, but they have really loud consequences. So the first thing that we are announcing is this observability platform. It's a cognitive observability platform called SCOPE that allows you to see these silent failures. They're no longer silent. You can see the loudness and you can see them semantically failing so that you know where the problems lie at that semantic layer. So that's something that, to your point, how can I use it? You can use it the way you build your systems. So we want these things to come at you through your environment.

40:00We're releasing them as OpenTelemetry race-based dashboards.

40:05Vijoy Pandey: So open telemetry is an open standard. Every application uses it. We leverage those open telemetry traces to then build that semantic cognitive observability layer on top. So scope sits on top of standard open telemetry. Hey, we'll continue our interview in a moment after a word from our sponsors. The Cognitive Revolution is brought to you and powered by Tasklet. I've been running Tasklet agents for almost a year, and they've become a major part of how the show gets made. Before every recording, one agent researches the guest and sends me a suggested outline of questions, complete with links to all sources.

40:49Meanwhile, another agent scores the overwhelming number of pitches that I'm getting from PR firms. And that's really the core idea behind Tasklet. Run your business on agents by giving them real jobs to do on an ongoing basis. Setup is incredibly easy. Just describe the work in plain English. Research a new sales lead, update the CRM, and draft a personalized reply. Tasklet connects to your business tools and can run around the clock, on a schedule, or when something happens, like a new email arrives. Each agent works with your tools and business context, and remembers useful information across threads.

41:28You can also share agents, connections, and reusable skills with your team, so the workflows that one person figures out become something the whole company can use. No developer needed. You decide what agents can do on their own and what needs your approval, and there are no per-seat fees. You pay for usage, not for adding teammates. Start with one recurring job, like researching the people you're meeting, and then just keep building from there. Visit tasklet.ai and use code COGREV for$50 in free credits. That's code COGREV at tasklet.ai.

42:07Vijoy Pandey: Now, having said that, the next question is, what do I do about it? So we've highlighted where the problems are, but how do I enable these agents to then talk to each other, align on intent, communicate meaningfully so that they are reasoning collectively? The way you would do that is to use protocols that allow for that alignment to take place. And we release two protocols. The first one is Accord. The second one is Concord. Think of Accord as alignment before the work happens. So there's intent alignment. And think of Concord as alignment or coordination happening during the work. So once we have agreed on what to do, then as we are working towards that goal, can we coordinate and negotiate?

42:56Vijoy Pandey: and that's Concord. And both of these protocols we are releasing as skills that you can drop into the environment that you have, whether it's ClaudeCode, Codex, Amuse, OpenClaw, Armies, or you can use it in Agent Bridge, which is an open source project that leverages some of the standard protocols that exist out there. So if you are leveraging A2A, if you're leveraging MCP, Accord and Concord ride above those protocols. They will ride above A2A. They will ride above MCP through AgentBridge. So we are bringing all of these open source projects, Scope, Accord, and Concord, to the environment that you're already currently using so that you don't have to have friction when you want to make your agents think together and not just connect together.

43:52So could we maybe work through an example a little more deeply. I'm, you know, I have an intuition, of course, for like, going back to the contract example, for, for instance, I know how human teams can get misaligned on a process like that, where, you know, the person who's selling the deal, and maybe stands to make a sales commission on the deal just wants it to get done and doesn't really want to, you know, delay on other things. And the legal team, of course, has a different perspective. And, you know, the support team wants to know what they're going to be on the hook for. Is it basically the same thing that we're seeing among agents or should we have a different kind of intuition for what these semantic failures look like?

44:37And then when it comes to kind of working through them, again, I have an intuition for sort of how people come together in a meeting or whatever and like hash it out and somebody makes a decision. What is the sort of step-by-step, blow-by-blow of the Accord and Concord protocols that perform the same core function for the agents?

44:58Vijoy Pandey: The semantic misalignment that we see within agents is actually very similar to human teams. So like I said, let's break it down into a couple of pieces. The first and foremost is before the work, then it's during the work, and then it's after you've completed the work. and we see these tiers all the time in human teams and we are seeing the same behaviors in agentic teams. Starting with before the work, what we see is misalignment on what we are trying to achieve, the goal itself. This is a shared goal part of it. The simple example that we were running with where multiple agents from sales and marketing, legal and engineering all coming together towards that deal desk situation.

45:45Vijoy Pandey: Sales, in this case, let's say here's a number, build a feature for 100 ,000 users. Sales being sales, takes that number and says, we're gonna solve for this challenge, comes back and says, engineering and product, this is what we need to go after, 100 ,000 users. Engineering assumes 100 ,000 users in total, whereas the customer really meant 100 ,000 concurrent users. There is a grounding problem here. There is a discovery problem here. There is a term alignment problem here. You don't really know what 100 ,000 means. Is it the total number of users? Is it concurrent users? Is it Mao? Is it DAO?

46:32Vijoy Pandey: I mean, you don't know what that means. So there itself is a simple example of what human teams do and it's a simple example of what agentic teams do all the time. Alignment on terms, discovery of terms, resolving conflicting terminologies. This is before the work even starts. And so what Accord allows you to do is to just align on the goal, term alignment just being a simple part of that problem. There are many, many parts to that problem. Now let's say you've aligned on terms. We all understand what 100 ,000 users mean. The next problem is while doing the work, during the work, is the coordination and negotiation.

47:19Vijoy Pandey: Sales assume a certain date and says, that's what we're shooting for. And for them, it's the delivery date toward the customer. It might be an alpha. It might be beta. It might be when the customer just gets hands on that software and starts experimenting with it, it's unclear. And it's a negotiation that needs to happen because engineering and product thinks this is the first drop. The customer might think is the eventual drop. Finance might think when do I sign the contract and get the money. And that's all happening. And that's constantly changing. It was not a one-time thing. It's like things change.

47:59Vijoy Pandey: There's feature creep. The customer looks at it and says, this is not really what I meant. It's constantly evolving. And so that's during the work. And so Concord is the protocol that allows you to coordinate and negotiate towards that goal that you already agreed to. Because this is a constantly changing environment. And then finally, you want to verify what you've done. And so that's evals and making sure that you're evaluating what you've delivered meets what you've agreed to and meets the coordination goals you are working towards. That protocol we have not released yet, but that's something that we are working towards.

48:40Vijoy Pandey: So right now it's Accord and Concord, but that third piece is something that's still coming. And then you have scope that's sitting below all of this. Think of it as the software in the car that looks at your car veering from a lane and not driving in the lane that you're currently in. so it's lane correction software so think of scope like that where you're driving on the highway on the lane and you're talking to somebody or you're drowsy or whatever the case might be you start veering into the lane next to you scope is checking for those digressions these are cognitive digressions in the case of multi-agent systems and then it informs you saying something is going wrong, get back in lane.

49:30Vijoy Pandey: Or something is wrong, you're too close to the car in front of you. Correct yourself and then go back within the lane, maintain that distance. So scope is that observability system that is looking at cognitive drift and making sure that either Accord or Concord gets triggered so that you're back in the lane at a reasonable distance in front of the, behind the car in front of you. So that's scope and then accord and concord. And together, they allow this multi-agent system to then work towards a common outcome. So you don't have those silent failures. You don't have those loud consequences. But you have agents thinking together and not just talking together.

50:13So from a very procedural standpoint, a lot of this sounds like it would ultimately depend on frontier language models to sort of take in the interactions that the multi-agent system has had and kind of run them through another prompt that's like, does there appear to be any potential disagreement or sort of misunderstanding due to conflation of terms or whatever? And then I guess I imagine in whatever shared space the agents are operating, there's sort of a moment of like this sort of supervising agent or monitoring agent, like inserting itself into the picture and saying like, alert for all you agents, I think you may be talking past each other on this particular dimension.

51:00You guys need to figure this out. Is that kind of how I should envision the process unfolding? Or is there, you know, how would you paint that picture if that's not quite accurate.

51:12Vijoy Pandey: We are hitting a level of maturity in multi-agent systems, Nathan, that is actually decoupling this entire architecture. I mean, you're seeing this happen everywhere. So we went from, again, isolated agents to islands of agents to connected agents. What we are also finding out is this, within each agent, there's a decoupling of the model and the harness. So the entire architecture is actually getting decentralized or at least distributed into many, many components. So there's the model and that's getting better and better. And it's just fascinating where that one is going. Then there's the harness, which actually is in place literally to solve for certain use cases and provide guardrails within an environment so that the model is in some ways contained within that use case.

52:10Vijoy Pandey: Or the model is in some ways more productive. So there are two aspects that the hardness takes care of. Accelerating a model's outcomes or guard railing a model's outcomes, depending on the situation. But those two things are getting decoupled as well. Like even the frontier labs are decoupling the model and the hardness. You can now take the harness from OpenAI and attach an open source or open core model to it, for example. There are other projects out there that, like, Microsoft talks about this decoupling all the time. So we are actually inside Cisco decoupling these two things. Even in my personal life, I'm decoupling these two things where we can pick a model, we can pick a harness, and not get tied between those two in a single binary.

53:01Vijoy Pandey: so that's happening but then that's within a user within a team within an island so that decoupling needs to happen even within teams and that's what we've been talking about where if you talk about a sales or a marketing or a legal or if you talk about Nathan you and me using our different harnesses a muse or army's I might be an army's user you might be a muse user I don't know but we need to be able to talk between our agents to solve for an outcome because you cannot have the entire world sit on one particular harness and other means open claw, muse, dots or a singular model. That's not going to happen.

53:47Vijoy Pandey: It's going to be a mixed heterogeneous environment. And so how can we then take this environment, multiple consumer agents and harnesses and models or multiple teams within an organization or multiple organizations talking to each other. These are all going to be heterogeneous environments where models, harnesses, teams, clouds, identities and access, they're all going to be different and heterogeneous and they will need to collaborate with each other. The only way to do this is through an open interoperable environment. The only way to do this is how we've done this for 70 plus years in the software world.

54:38Vijoy Pandey: The only way to do this is to have open source and standards that actually plug into what exists. So you cannot throw away what happens. You need to leverage the environment that exists today. build upon that. So that's, for example, scope to first figure out what the problem is. You cannot fix what you cannot see. So scope allows you to first see the semantic drift, the cognitive drift. And it's open source. It's working on the environments you already have. Once you see that drift in this heterogeneous environment, consumer or enterprise, then you can leverage the protocols, Accord and Concord and many, many others not just from us, but from other players that will happen over time to solve for that drift.

55:27Vijoy Pandey: And like you said, it is going to be in the way you explained it, which is all of these agents in the consumer space, Muse, Armies, Dots, or in the enterprise space, Salesforce, RV, CloudCode, Codex, all working together, a standard way of figuring out how that cognitive drift and when that cognitive drift is happening, then triggering these protocols through a skill, through open source, through a container, sitting on the protocols that already exist, all standard ways of doing it, then getting back on track, and then going your merry way. So we don't want to disrupt the way these agents are talking to each other.

56:12Vijoy Pandey: We just want to make sure that when they're drifting cognitively, they go back on track. That's it. So earlier you mentioned that the verifier in today's world is often a human. How sort of opinionated and contextualized or embedded do you think these verifiers will tend to be? I'm kind of wondering, like, is what we need for leaders, you know, whoever in this sort of deal example, right, whoever is sort of the VP that this rolls up to is what we need ultimately an agent that represents that leader's perspective and tries to kind of help them play Solomon and decide like how to, you know, divide, how to split the baby, so to speak, or how to, you know, resolve these conflicts.

57:05Is it, is it an agent that tries to bring that very particular point of view and, and basis of judgment to the table? Or is it a more neutral thing that sort of just, just tries to flag, like, you guys may have a disagreement here, you should sort it out. My sense is that most of these human decision makers don't really have agents yet that sort of represent their point of view in the agent realm. But I don't know if you think that building those on kind of an individual decider by decider basis is like on the critical path, or if we can just kind of get by with a more neutral, you know, platonic observer that notices when things are starting to go bad and that can maybe be enough.

57:53In the world of agentic computing, one of the things that we've been talking about when

57:58Vijoy Pandey: it comes to hiring and when it comes to where humans can add value, the first thing that comes up is judgment. and we've heard that quite a lot over the past few months and the year where humans are the ones that are bringing taste and judgment to the equation. And what you just described and asked is a judgment question. Because in the end, if there was perfect information, if everything was perfectly known and we had all the data in the world to make a call, then even a deterministic system could make the call. Forget about an agentic system. But in real life, you're always making gut decisions.

58:42Vijoy Pandey: You're always making informed decisions. You're always making judgment calls. And that is typically done by a human. And that is where the human adds value. So right now, I believe that that verifier or that person providing the judgment is actually a person. It's a human. and whether it's a human in the loop, which is not where the world is or should be, maybe that's where the world is today, but it's definitely a human on the loop where if these agents are collaborating, we want to see an environment, even if it's a heterogeneous multi-agent system, the human is on that loop making judgment calls, deciding because data is not sufficient and is never sufficient.

59:30Vijoy Pandey: So the judgment call, the test calls, all of these come from a human because the human also needs to take that and influence the outcome. Like take that call and make sure that that person is actually accountable for that outcome. So there's a lot more that happens after the judgment call. So right now it's a human. What I would assume is in the future, you can train an agent that is also a judgment agent that gets better and better at making that call. Now what that percentage looks like I don't know. The world is evolving pretty rapidly but I can assume and I can only tell that a lot of that judgment making capability will get learned over time by a certain set of agents as well.

1:00:23Vijoy Pandey: So the way we're thinking about this entire system is there are specialist agents that really know the marketing side, the sales side, the legal side, the engineering and products really well. They're collaborating towards a common goal using the observability-like scope, using protocols like Accord and Concord. They're collaborating. And right now, there's a human on the loop. But there's also agents that we are building. We're calling them cognition engines. And there are again of two types. Like I said earlier, type one is a cog, type two is a GAT. And this actually comes from a famous quote by Raj Reddy, which says AI should be one of two types, guardian angels and cognitive accelerators.

1:01:14Vijoy Pandey: So GATs or Cogs. GATs, make sure that you are guard railed and you're not doing something that you should not be doing. So cognitive accelerators are, let me accelerate things that you already know how to do. Why are you doing things from scratch? So I would think this judgment engine that we just talked about could be a cog over time. We're not working on it right now, but could be a cog over time that accelerate that judgment process over things that you've already decided on in the past. you should not be revisiting that decision again and again. And so this model capability explosion that we're seeing also helps us in building those gaps and cogs as they're helping us in building these individual agents that are collaborating together.

1:02:08Vijoy Pandey: Because the model is separate from the harness, is separate from the agent that is collaborating together. So we can use those model capabilities to also build pretty capable GATs and pretty capable COGs in this entire system. Let me just share a little bit of my own kind of mental model for agents that I've created and how they serve me. And then I'm interested to hear you kind of bump that up against the taxonomy that you're describing. I live a pretty simple life. It's mostly just me working for myself. So I don't have a lot of the complexity that our enterprise you know, customers have. But what has been really useful for me so far is just a simple mental model of either this agent is an extension of myself.

1:02:57And for those, I have them on my laptop where I do my work. They have very high access, including basically to all of my accounts with few exceptions. I think of that as the high access, low autonomy version because I don't want those to be just going off and coming up with their own ideas given the access that they have. That's a little too risky. Then on the other computer, sitting right off screen here is the Mac Mini. That's where I have the more autonomous but lower access agents where I figure I can give projects, it might take weeks in some cases for them to go pursue an objective. Usually a lot of that is waiting for people to get back to us.

1:03:43But nevertheless, you know, there may be fairly long running projects. And I feel comfortable doing that because I've only given them so much access where like, at a minimum, I'm pretty confident that they're not going to really, you know, harm me in a way that's going to be too, too painful. And most of them might embarrass me a little bit as they go kind of represent me as my assistant in the world. So this is the two forms that I have kind of settled into, and they seem to be working fairly well. How would you compare and contrast or map that onto the taxonomy you're describing with the Guardian Angels versus the COGS, versus also like we have kind of team level agents?

1:04:26I've only mapped out a small part of this space. Like maybe help me develop my intuition for the rest of the space that I haven't mapped out yet.

1:04:34Vijoy Pandey: The first thing I would ask you, Nathan, is why are you picking one of the two? Why do you have two setups? High autonomy, good trust, but low automation. And then the second setup that you have is low autonomy, lower trust, but higher automation in some ways. You have two systems where you've given quite a bit of access to one system, but you want it to come back to you, and you want to make sure that you're not doing things that you have not allowed it to do. And the other one, as I understand it, is the other system is the access is limited, but it's actually going and doing a lot more things without you being in the loop.

1:05:21so it's more autonomous but the access is smaller. So why are you picking these two systems?

1:05:31Vijoy Pandey: Why can't you have a system? There is something there that is driving your decision to do this because you're still being cautious that things may go wrong. Eventually, we would all want that one system that behaves as me. so my agent is Arnold on Arnold Lane this is a Pink Floyd song from the 1960s and there's a reason I picked this name because Arnold is an identity and Arnold is a personality and Arnold represents me and all of these things are immutable and it needs to survive regardless of the technology stack that is below and it's changing a lot. So Arnold is that identity, that persona, that access, all of that.

1:06:24Vijoy Pandey: But the infrastructure stack below it could be started actually with OpenClaw and moved to Hermes. And it's sitting today on multiple harnesses and models. So Arnold actually sits today on Hermes. It sits on Muse. It sits on Dots. It sits on Grok. It also sits on this internal agent at Cisco that has been rolled out to every employee within Cisco. The identity, the persona, the memory, all of that is immutable. It's been separated from the harness, which is all of these things that I described, which is separated from the actual model at the back end that I might be using. I mean, Hermes and OpenClaw moved from Gemini to Haiku because they were cheap and then moving to GPD Sol at this point.

1:07:21So all of these have been decoupled.

1:07:24Vijoy Pandey: The harness is decoupled. Model is decoupled. But the identity is immutable and is mine or my agents. Now, what I would like to do is have Arnold represent me through and through. I don't want to restrict Arnold to saying, you have certain access here, certain automation here, but something else on this use case, something else on that use case. I mean, that defeats the purpose. Then I'm the one ensuring, oh, I gave it this here and I gave it that here. It's just too complex. eventually we'll all get to a point where Arnold will have all access, all automation, and will only come to me for a judgment call when it does not have that judgment call in a ledger somewhere that it can work off, in a cog somewhere that it can work off.

1:08:21Vijoy Pandey: And so a very simple example here, since you brought this personal use case, I'm so fascinated by it because I do this all the time like you, Nathan. So I have Arnold Lane, my agent, my wife, as Lyra Bilacqua. She is this character from The Golden Compass, and that's her agent. And Arnold and Lyra need to collaborate on things as simple as let's get a dinner reservation two weeks out for an event that we are celebrating. And today, you just cannot do that autonomously. You just cannot because of many reasons. First and foremost, the model is a blocker because the model itself is not capable enough to do this end to end because we are all in the free tier, fine.

1:09:25Vijoy Pandey: The harness is the next blocker because the muse harness is constantly asking for permission, rightfully so, because it is being cautious when it's sending emails to each other. I don't want to be part of the loop. If I'm going to give you permission every single time, I might as well just do it myself. I can just talk to my wife. I mean, just why even go through this whole process, right? I want this entire thing done autonomously. But the third thing is, even if you enable this, and I'm experimenting, so I'm going and saying, yes, go ahead, send the email. But as we do this, and I'm going to drop skills next with Accord and Conquer, as we do this, what we're realizing is that they are not agreeing because there are certain restrictions I have of where I want to go, the kind of restaurants I want, the certain restrictions that my wife has and where she wants to go and the kinds of restaurants she wants to go and the kind of menu items in each one of these places.

1:10:27Vijoy Pandey: So even a simple thing of food restrictions or things you like, the kinds of restaurants you like, the locations you like, the kind of ambience you like, the availability of the restaurant for the time we want to book, all of these things are variable. and even a simple thing like that, we know this. Even as humans, it takes us a good five minutes, ten minutes or more sometimes based on the complexity to book a simple restaurant for a good event. And our agents cannot do this autonomously which is ridiculous. I want our agents to just go ahead, know our profiles, know our likes and dislikes, talk to each other, and compromise and negotiate because this discussion is a negotiation and a compromise in the end because it cannot just be what VJ likes.

1:11:27Vijoy Pandey: It cannot just be what my wife likes. It needs to be a compromise and just do that. And you cannot do that today. So there is a massive problem, even in a two-person situation, on an agentic stack that has been built and deployed and rolled out globally today. You cannot do this. Until we can get to this point, we will not get to an agentic outcome for anything. So these protocols, these observability platforms, these skills that you can deploy are valuable, not just in the consumer space, in a simple situation like this, but then draw that out to that enterprise$2 million,$3 million deal desk.

1:12:15And think of what happens there and the outcomes that can go wrong

1:12:19Vijoy Pandey: in that kind of a situation and what we need to do there in the enterprise. That's the world you and I want to go towards, where our agents can solve for the minutiae of booking a restaurant without us getting involved. Because, man, if they can't even represent us in the personalities that we have for a simple dining situation, then what's the point? I mean, where are we going? Do we know anything about scaling laws for multi-agent systems today? I've heard things like the square root of the number of agents will give you, like, the speed up. So in other words, you know, 10 ,000 agents can go 100 times faster.

1:13:08But it occurs to me that maybe we just have no idea what we're talking about there right now because like if you went back in time to GPT-2 or 3, presumably like there was zero, you know, productive synergy between multiple agents. and so maybe this is not something that is like a timeless scaling law but maybe something that's more like a per class of model scaling law but i'm not even sure if that is right any insights there that you can uh offer to de-confuse me very quickly i mean computing has always

1:13:42Vijoy Pandey: scaled across three axes and we touched upon this in the beginning a little bit the three axes being algorithmic scale, vertical scale, and horizontal scale. And that is where we come in. A company like Cisco comes in and says, yes, keep doing those two. Keep making software better. Keep making the node more capable and bigger and better. But then there are connected systems, and we do build connected systems. But these connected systems and the nodes that the system is connecting are changing over time. So very quickly, what we are seeing is that the network itself has connected billions of endpoints.

1:14:25Vijoy Pandey: In fact, tens of billions of endpoints when it comes to machines and billions of endpoints when it comes to humans. The network has connected them over the past 40 years to build connected systems through connecting data. That's what the network has done. What we are realizing now is those endpoints are actually changing foundationally. Those endpoints are starting to think, they're starting to entangle, they're starting to fluctuate, and they're starting to move. They're dramatically different forms of computing. And we've touched upon only one so far, and that is the think part of it. These endpoints are no longer data endpoints.

1:15:12Vijoy Pandey: These endpoints are intelligence endpoints. They are thinking. They are semantically collaborating. They're exchanging meaning. They're actually behaving as one cognitive environment. They are reasoning collectively. So that is the thinking part. That's not standard data exchanges. That is exchanging meaning. So that's one endpoint category. But then the others, they're starting to entangle. That's quantum computing. They're starting to fluctuate. This is new forms of computing. using classical physics like thermodynamics, like oscillations, and then they're starting to move, which is embodied AI and physical AI.

1:15:58We can take all of these innovations and put it, embody it,

1:16:03Vijoy Pandey: in a physical environment where they can actually move and sense and act on physical objects. So all of these endpoints are different forms of computing, and that is a fascinating part because the network that is going to connect all of this and enable a connected system is going to look very, very different from the network that has connected data in the past. That brings to mind, last time we talked, you had, I think, a very interesting and at least in my circles, kind of heterodox position on what is sometimes called neural ease or the transfer of high dimensional internal states from one AI system to another.

1:16:48This, on the one hand, could be extremely powerful, obviously. There has also been sort of a, you know, let's say, maybe an immune reaction among AI researchers over the last few months in wake of things like the open face incident where people are saying, geez, like, if we struggle to monitor them this much, even when we can read the chain of thought in tokens, we better be really careful about allowing them to communicate in this higher dimensional language that is native to them, but like very much not native and, you know, presents a whole other interpretation problem for us. Do you have any updated thoughts on that?

1:17:26Are you optimistic that we can just kind of solve and effectively monitor neural ease? Would you say you're like a little more cautious about it than in the past? Or what's kind of the latest Vijay take on Neuralese?

1:17:37Vijoy Pandey: I'm super excited about that. And to me, it's inevitable. It's inevitable that the arc of technology will move towards a situation where you will do things more effectively, more efficiently, and you will build infrastructure in such a way that you get the most out of the technology stack that you're building. Because in the end, the goal is to solve for problems that have been hard problems to solve for humanity. In the end, we are all moving towards building technology to solve for problems like drug discovery, to solve for problems like new materials, to solve for problems like logistics planning and financial modeling and weather modeling.

1:18:24Vijoy Pandey: These are really hard problems. And these are problems that have long-term impact on humanity, on civilizations. and that should be our goal. And we should do everything possible to build the best technology stack to then solve for those really hard problems for humanity. And if it means building these latent space protocols where agents can talk to each other, not in natural language, but with higher dimensionality, let's build those things. I am a big believer in accelerating technology because, to me, it also opens up the space to then build this in a guardrail manner. We've talked about GATs and COGs.

1:19:12Vijoy Pandey: One of the big reasons we are going after GATs, the guardrail technologies, is to ensure that everything else that we are building is working with the laws and regulations and the compliance needs and the security needs and the scaling needs that could be deployed in the environment you're working in. So if you're asking me, yes, I'm still a big believer in that. Within agents is where I think we should deploy them. And those agents could be in software. Those agents could be embodied within robots and physical AI. Now, when you talk about bringing humans into this picture, and the neural link kind of approaches, the guardrail part of it in me takes center stage where, yes, we should do this, but we should do this in the guardrail way that we do everything else.

1:20:09Vijoy Pandey: We should do the right amount of testing. We should do the right amount of deployment. We should deploy it in a canary way so that we understand it in the same way we introduce a new pharmaceutical drug in the environment. So make sure that those steps are met. You don't shortcut any of those steps. But to me, that is the exciting part. And it's almost in some ways inevitable. But do it carefully. The same way you would introduce any other drug in this environment. I just saw an interesting lecture from Yuval Noah Harari, where he makes this analogy between the state of the cow in today's world and the state of the human in the future.

1:20:51And this is, you know, to be clear, like a worry that he has. But he basically says the cow doesn't know anything about money or the financial system. And yet the cow's life is governed by the financial system and all the incentives that the people raising it have, you know, and you can try to explain it to the cow, but it's not going to get it. And then, of course, the implication of that is like, we are kind of setting ourselves up to be in a similar position where we're creating all these AI systems that are going to increasingly run the world. And they'll do that largely for our convenience and benefit.

1:21:30But at some point, we may end up in a spot where because they're so much faster than us, they're processing so much more information. They're potentially communicating in these high dimensional ways that are not native for us. we may end up in the same position as the cow where we are sort of governed by superstructures that are operated by AIs and we really can't grok those things anymore. Do you think that's like, okay, do you think we can like get around that with neural link type connections where we will be able to natively understand high dimensional representations of these new superstructures?

1:22:06Or maybe we can just like avoid that through guardrails? What is your sort of positive vision for where this goes and how much control we ultimately should cede to AI-powered superstructures?

1:22:20Vijoy Pandey: This brings us back to where we started, Nathan. I just saw that cow example as well recently. And to me, I personally don't understand much of how high-frequency trading is done today. I know the concepts. I know what is trades, but I'm not in the business. I'm not familiar with that business and how those things happen. But software is running trading today. And I might be the cow even today, whereas that system is pretty well understood by the folks that are running it and writing it and operating it. And I've designed it, right? So there is that situation even today. I would say the same thing about the laws and regulations that exist in our state or within the country.

1:23:13Vijoy Pandey: I understand it superficially, but I don't understand the details behind it today. But the people who bring in new laws, they understand the system really well. So what I would say to that analogy is, yes, we are already cows or we are already humans in the system that exists today. That's why subject matter experts exist. That's why specialists exist. That's why agents are going to be also specialists. That's why people who are building those harnesses and those customizations in models within those verticals for those specialist agents will understand those systems. I will understand and infrastructure stack agent really, really well.

1:24:03Vijoy Pandey: So you need those specialist agents. The people who built them will understand those agents, will understand those harnesses. And that is the world we're heading towards where every agent will be a cow and a human at the same time in a multi-agent system. But that multi-agent system will need to collaborate to then drive a global outcome. and the way you would monitor that is through systems like Scope where you're looking transparently at everything that is going on, putting that into an observability system so that you have a ledger, a system of record for what that multi-agent system is doing.

1:24:47Vijoy Pandey: Every action it's taking, every decision that it's looking at from other agents, every tool call that it's making, all of that is recorded somewhere so that you have compliance requirements checkbox, but also you can debug if something goes wrong. And then when that system itself sees things going off track, it can trigger not just the protocols like Accord or Concord or the others, but also GATs and COGS to ensure that those cows and humans, those individual subject matter experts, are back on track to solve for the global outcome that you and I are looking at. Well, I admire your comfort in taking the cow-like position today.

1:25:36I'm definitely feeling a little more cow-like myself as things progress. And that may continue as I feed the transcript of this conversation into my agents and ask them to adopt the scope and accord and concord protocols and open source projects that you guys have been building and releasing. That'll both take my system to a higher level and potentially leave me even more marginally in the dust. But that's, I guess, just a sign of the times that we're living in. And if nothing else, it is definitely an interesting and fascinating time to be alive. This has been an excellent conversation. I know you got to go, but I really appreciate it.

1:26:17Vijay Pandey, thank you for being part of the Cognitive Revolution.

1:26:20Vijoy Pandey: Thank you so much, Nathan. This was amazing.

1:26:33Everything's lovely Says the lodger says he Everything's lovely And he's kept the key He came to stay in April With a suitcase full of moons He wears my Sunday suit to town And he whistles all my tunes

1:27:07Everything's lovely Says the lodger, says he Everything's lovely And he's kept the key

1:27:23I asked him for a supper, just a table laid for two He said it for a hundred on the lawn beneath the ewe

1:27:40Late at night, through the wall He talks to the lodger next door Not in words, not at all In colors never seen before

1:28:00Everything's lovely Says the lodger, says he Everything's lovely And he's kept the key

1:28:32Last night they came and called me through the plaster soft and low Now I'm talking in colors and I'll never let you know

1:28:50Everything's lovely Say the lodges and me Everything's lovely And we've all got a key Ooh

1:29:13Everything's lovely

1:29:22Everything's lovely If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guests and topic suggestions, and sponsorship inquiries, either via our website, CognitiveRevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more.

1:30:08We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at AIpodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.