Building the Internet of Agents with Vijoy Pandey - #737

0:00This is like no different from humans, where you see a bunch of SMEs come together and solve for a business problem. You replace SMEs with agents. That's the world we're heading into, where a bunch of these subject matter expert agents come together and solve for a particular problem. And they need to collaborate when they come together. To us, the Internet of Agents is that platform. It's an open, interoperable, quantum-safe platform for agent-to-agent collaboration.

0:41All right, everyone. Welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. Today, I'm joined by Vijoy Pandey. Vijoy is a senior vice president and general manager of OutShift by Cisco. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Vijoy, welcome to the podcast. It's amazing to be here. Thank you, Sam. I'm super excited for our conversation. We're going to be digging into one of my favorite topics nowadays, agents, and in particular, how agents work best together in an enterprise. Before we dive in, I'd love to have you share a little bit about your background.

1:20I am the SVP GM of OutShift by Cisco. And think of OutShift as an internal incubator within Cisco. So we are building businesses for Cisco in adjacent spaces. So if you think about Cisco, you think of a networking giant, you think of security, you think of collaboration and observability, especially after an acquisition of Splunk. And if you think about OutShift, we are looking at what adjacencies can Cisco get into, both in terms of space and time. So space is, yes, we are going after, let's say, a persona X, let's say the network engineer. Is there an adjacent persona that we can build products around?

2:01and then go after that market. In time, it's more around handling risk. So if you think about technologies that might have risk involved with it, like quantum, you might have markets where the tech is proven, but the market is still forming. Agents is a good example. So we look at some of those problems where there is a little bit of technology risk, a little bit of market risk, and then we enter that space And we see ourselves as reducing risk for Cisco when they want to enter a particular market. And the way we are structured is actually pretty interesting as well. We are structured as a bunch of startups inside of Cisco.

2:39So we have everything from ideation to engineering to product to marketing, customer support, and business development all rolled into one. Because when you enter a new market, it's not just about building product. It's the go-to-market situation that easily complicates itself. And that is the hardest challenge to solve. So talk a little bit about Cisco's interest in agents. Why is this an interesting opportunity? It's actually a multifold problem statement. We all know that the world is going to be very agent forward, where you'll have a collection or an ensemble of agents coming together and solving a big business need.

3:23And so this is like no different from humans. where you see a bunch of SMEs come together and solve for a business problem. And you replace SMEs with agents. That's the world we're heading into where a bunch of these subject matter expert agents come together and solve for a particular problem. And they need to collaborate when they come together and solve this problem. So where is that collaboration platform? And so to us, the Internet of Agents is that platform. It's an open, interoperable, quantum-safe platform for agent-to-agent collaboration. So that's the problem that we're going after through Internet of Agents.

4:07And it's a forward-looking problem statement. And it helps in the products that we build within Cisco, but it also helps as a collaboration platform for all agents that exist out there. And when you describe it as a problem statement, that's a bit abstract, but you've announced an open source project called Agency that is designed to make these ideas more concrete. Can you talk a little bit about Agency and what you're trying to do there? Think of Agency as the open source version for Internet of Agents. And so if you think about Internet of Agents, again, the problem that we're trying to solve is how do you bring disparate agents from different vendors with different capabilities, with different subject matter expertise to come together to solve for a business need.

5:02So let's start with a concrete problem. Two years ago, if you were to stand up a sales funnel pipeline software stack for the the entirety of a sales organization, you would bring in Salesforce. You would bring in, let's say, identity for Microsoft. You would bring in cloud native security and security platforms from Cisco. I'm being biased here. You would bring in employee databases from Workday. And that'll give you an idea of which identity gets authorized to do what. And you would go through a bunch of clicks connect all of these platforms together and voila, out would pop a portal for you to look at sales funnels and sales pipelines in your large enterprise.

5:55And I'm not going into the complexities of now setting it up, deploying it, scaling it, managing it, but there's an entire pipeline. I'm not minimizing the work, the pain. There was some magic in there somewhere. There is a lot of magic. There's a lot of toil, I would say, our IT operations teams actually go through. But it was all front-ended through a SaaS or multiple SaaS platforms and a ton of code that somebody would write within the operations team to make it all happen, to make the magic happen. What's happening now, for good or bad, and I believe it's all good, but we can have a discussion, is that all of these platforms are becoming agentic in nature.

6:43So Salesforce, Agent Force, Workday is becoming agentic in nature. They have a bunch of agents. ServiceNow, which is a ticketing platform that is actually driving the workforce behind this is completely agentic in nature or moving towards that. Microsoft Identity, that's becoming agentic. Cisco, security is becoming agentic in nature. All of the cloud native platforms are becoming agentic in nature. So you have all of these platforms that were point and click, write some blue code, walk away, you're done. They're all becoming, here's an NLP interface, here's an agentic backend. How do you connect these things to deploy the same thing, which is a simple sales funnel application for your large enterprise?

7:30So you have all of these disparate agentic components, which are subject matter experts in their own domains, but they all need to come together. They're all by different vendors. They speak different languages. I mean, one thing may not seem the same to the other. So semantically, they're different. They have all been built on different frameworks. So some might have used Langchain as a framework. Some might have used Lama Index as a framework. Some might have been built on Bedrock or Q on Amazon. They might reside on different clouds. and they all need to be stitched together in an identity and security and guardrail framework that is very enterprise-centric.

8:15That is, in our case, when we deploy this, it needs to fit into the Cisco security paradigm and the Cisco workflows and the Cisco scaling and cloud paradigm. And so how do you do this? The platform for that is exactly what Internet of Agents is trying to accomplish. And so if you think about the workflow, we break it down into four steps. Before we even go there, I want to interject. You started this by saying, for better or for worse, this is the direction that we're going. And as you described this use case of got a bunch of SaaS systems and you're trying to integrate them, the way that you would typically do that, you mentioned glue code integration tools and thinking about replacing that with agentic systems.

9:02you know, in many ways we're taking kind of well-defined interfaces and replacing them with undefined, noisy, and probabilistic interfaces, which doesn't necessarily seem like it's a pure win. The wins, I guess, are potentially a degree of agility. And if you can eliminate the need to build that glue code, you know, that's a win. But given the current state of agentic systems, you know, there are some drawbacks for sure. So, you know, talk a little bit about how you see all this playing out and like how organizations will trade the determinism of traditional integration versus the, you know, the potential noisy messiness of agentic style integration.

9:53You hit the problem right on the head here, Sam. So I think as with every step function change, and I believe we are at a step function change, where we're going from deterministic computing to probabilistic computing or a combination of deterministic and probabilistic computing. So that's the big step function change, which means that we are moving from an abstraction that was cloud computing to an abstraction that is agentic computing. And it's a higher level of abstraction. And every time a higher layer of abstraction comes in, there is pain associated with it. We've seen this time and again.

10:32It doesn't seem like that now that we're looking at cloud in the quote, rear view mirror, right? That's right. But there was a lot of pain. I mean, imagine what people did when we started out, right? It was like, I like this thing in my environment and I'm just going to replicate this thing on the cloud. Yeah. Because this is what I'm feeling. And we used to talk about server huggers, folks who didn't even want to let their stuff go to the cloud. That's right. And it's like, I like this pet that I have, and I like really, I've named it, I've groomed it. Pets versus cattle analogy. Yes, exactly.

11:06So you have to go through that pain. And we haven't, we are not done with that either. I mean, we are very, I mean, even with the cloud transformation, it's like how do you reorganize, restructure your code so that it's cloud native in nature? That's not done yet. In fact, we've given up on some areas and said, this is going to be a monolith. Don't touch it. I'm good. Whereas in some other cases, we're like, okay, let's break it down and make it more scale out and scalable. So that pain we're still in the middle of, and we're starting on this new pain, which is take everything deterministic and make it agentic.

11:45The step function here is you are dealing with deterministic APIs, and then you threw in human toil on top of that to make things all work. Because in the end, a human is making decisions. A human is taking actions. A human is reading and making sense of some data or dashboards. and all of those things, what we are saying is we have an agentic workflow and we have LLMs and foundation models and code around it that actually can help you as humans make some of those offload, some of those decisions and data parsing and code generation and document generation to an agentic system so that you don't have to do it.

12:40Because what we are doing is actually moving that level of abstraction one level higher, where humans don't have to do a bunch of work that they are somehow used to doing and somehow have believed that it's the thing that we call work and we enjoy. But we need to actually, agents are good enough that they can offload some of that work from you as a human. And that has not been possible in deterministic systems. So yes, whereas there's going to be pain to make it all happen, what will allow humans to do is offload some of that work to agents and that humans can do higher abstraction work and higher value work.

13:24And that's the pain that we have to go through. So I'll give you a quick example here if you're okay with it. So we built this multi-agent system called Jarvis. And Jarvis, there are many, many projects called Jarvis. everybody loves iron man and everybody loves the marvel characters so in our case jarvis is an sre jarvis so it's sre jarvis so let's call call it sre jarvis sorry as in site reliability engineer thank you yes so devops kind of devops kind of engineer so it's a site reliability engineer Jarvis in the cloud native space. And so what it's trying to do is it's trying to look at everything that OutShift does.

14:12So that's how we started out. Everything that OutShift does and where humans were involved and can we start replacing some of those outcomes with Jarvis. And so on the surface, Jarvis is one interface and when I say it's one interface, it's actually five plus user interfaces. You can access Jarvis through a WebEx chat. In due time, it'll become Slack as well, but some text interface. You can access it through Backstage, which is a pretty well-known interface orchestration engine in the cloud data world, which is open source, by the way. You can access Jarvis through a VS Code plugin, so a codified plugin.

15:00You can access it through Jira or CLI. So it's got a bunch of user interface, but you interface with Jarvis. But Jarvis itself is 20 agents behind the scenes. And the 20 agents are doing all sorts of tool and data calling. So they're doing RAG tooling. They're doing live calling to Kubernetes. They're doing doing tool calling to PagerDuty. They're doing tool calling to AWS Backstage or Commodore, which is another open source project in the cloud-native world. So they're doing a whole bunch of tool calling on the backend. They're interacting with Jira. They're doing CICD pipelines. They're setting up EC2 dev instances.

15:45They're doing plumbing to access LLMs. They're writing code. So they're writing Kubernetes code. They're writing sandbox code for VMs as well as Docker images and so on. And the outcome for all of this. So whoop-de-doo. What's the big deal? Are these just engineers gone wild? I mean, is this? So here's what we've managed to achieve. We used to have a support desk where three engineers or SREs used to support that desk continuously. And all of that is now being handled by Jarvis. So three engineers worth of daily toil is now handled by Jarvis. And this is at outshift scale, which is not Cisco scale.

16:36So we start looking at Cisco scale, which we are doing right now. We're deploying it in various Cisco teams. I'll have better numbers for you, but it's going to be 10x, 100x larger than this. Now you're looking at, so now think about, so that's one outcome. The other thing we managed to do is almost 30 % of tasks that engineers would do, not just SRE, but consumers of the software SRE platform. We've taken that from a day to minutes. And the kinds of tasks that we're talking about here are engineer needs to deploy something to a Kubernetes cluster or... Or set up a new LLM for usage or create an EC2 instance, for example, or like plug in a whole bunch of RAC databases into the pipeline that we have.

17:28So any cloud task that you can think of, set up a new VM or set up new identities, or any new task you can think of, you would need SRE engineers and software developers to come together, go through a workflow cycle, and it would take a day or more than a day to set these things up. and now it's literally minutes for 30 % of the tasks. So not everything, but it's 30 % of the tasks. And we're walking through those workflows. The query responses, because engineers might have queries or questions on how does this work or is this running? Those things have gone to seconds because those things truly should not take any time.

18:14Tell me what Docker images are running in this cloud instance. that should be like, voila. I mean, I should not even look at a dashboard and try to parse that dashboard. It's like, here's an NLP request. I should just spit that out for you. So that seconds, the support desk has gone from three engineers to Jarvis and 30 % of tasks have been eliminated from human interaction and is completely identified. So this is the, so what? And we are trying to do this, again, in a pretty different environment with Swisscom, who's basically one of our biggest service providers that we work with. And so we're trying to figure out how we can fast track their network config deployment and debugging pipeline using a multi-agent workflow.

19:16So these are some of the examples that we've been working on. There are many, many, many more. Where the Internet of Agents is the platform that we're leveraging, but these are real-world use cases that you all come across every day. So when you talk about this as a platform, what is the platform providing? Or conversely, what is needed when you're deploying out an agentic Internet of Agents, let's call it? what do those agents need that needs to be provided by some platform? So if you think about the Internet of Agents as the way, the platform where all these agents come together and collaborate, and the agency is the open source manifestation of Internet of Agents, we think about the entire problem of agent-to-agent collaboration in four phases.

20:08The first one is discovery and identity, where you need to discover the agents that you need to bring together, their capabilities, their reputation, their uptime, the languages that they speak, how long have they been available. You need to look at all of those things to then narrow down on the agents that you want to bring in to solve for your business need. Along with that, you need to assign identity to those agents because agents are like humans. So they have human capabilities and characteristics, but they operate at machine speed and scale. And that makes them a pretty dangerous object from the identity perspective because you are giving them agency to act on their behalf, to do things on your behalf.

21:04So ensuring that identity is taken care of is also a big problem. So discovery and identity is phase one. And once you've settled on those things, you want to compose them in a workflow. So all of these 20 agents, like we talked about in Jarvis, they need to come together in a workflow to solve for, let's say, deploying a workflow in EC2. And so how do you connect them, stitch them together in a workflow? That composability is phase two. Once you have that composability done, you need to deploy it on that cloud-native infrastructure or on-prem, doesn't matter. So deployment is phase three, which is where you're also scaling out these things.

21:48And then finally, you have to observe and evaluate. So in the cloud-native world, we're stuck with observability. In the energetic world, we need to also figure out whether agents are doing what they said they were about to do. So are you really doing what you said you were about to do, not just on a poor agent basis, but especially when you have 20 of these things from different vendors coming together and collaborating? Because believe me, there's going to be finger pointing. So this is a governance layer in an enterprise context that would live within that phase. There'll be governance in that phase as well.

22:26You're right. But there's also evaluation where you say, you said you had 90 % accuracy. But what I'm seeing is only 60 % accuracy. And that's per agent. But if you have 10 of these, we thought that 10 of these would give me 90 % accuracy. Each one of them is giving me 95 % accuracy. But when you stitch them together, there's a dramatic drop-off, and you might not get your 90 % accuracy in a collection or an ensemble of agents, because they're all probabilistic in nature. So evaluation of the outcomes from agentic workflow is a big deal. So that's the fourth phase. And so tying all of these things together is what will give you the outcomes that you desire.

23:20Okay. So a couple of questions to help me narrow in on, you know, what exactly we're talking about here. So the first is you said that agency or this internet of agency, internet of agents is where the agents come to collaborate. I guess I'm forming a visual picture of that. Like if I have Salesforce, the Salesforce agent is, you know, on Salesforce's infrastructure somewhere, the Workday agent is on Workday's infrastructure somewhere. I've got some agents running on my infrastructure, what does it mean for them to come to a central place to collaborate? Is this like, you know, should I be thinking more like a communications hub or should I be thinking of it in a different way?

24:12Yeah, so that is, I mean, that is exactly the problem statement that the agenda of agents and agencies solving, which is all of these agents that you just described and with the use case that we were referring to earlier, setting up a sales funnel pipeline in a large enterprise. yes, there'll be a Salesforce agent running on their cloud with their code, with the frameworks that they've used to develop. Similarly with ServiceNow, Workday, Cisco, Microsoft. But the poor IT admin and the poor SRE within large enterprises is trying to stitch all these things together in a cohesive way. And so how do you do this?

24:54So the analogy I would draw is, Think about the original internet. And so when you would want all of these servers and all of these web servers scattered across the globe, belonging to different entities, to come together and talk to each other, how would you do that? The first thing you would do in the discovery phase is you would do a DNS request and say, you said pets.com. Where is that pets.com? And identity or verification is, it's signed by VeriSign. And so I trust that pets.com is a valid endpoint to talk to. And so the equivalent of that in agency would be there's an agent directory. And you would go there and say, I really need an agent that does sales funnel setups.

25:55and who is it from? Oh, it's from Salesforce. Where does it reside? Oh, the endpoint is sitting on Salesforce's cloud. Is it reputable? Yes, it is because it's Salesforce and I trust Salesforce. So, and you would do that with ServiceNow and you would do that with Workday. But then there are other agents that you might need in the picture. agents that Cisco, if you're deploying it within Cisco as an enterprise, like our IT organization or our engineers might be writing agents as well. So where do I find them? Maybe they're code and they're sitting within Team X or Team Y within Cisco. Or you might be trying out a new startup, which might not be as reputable.

26:41So this startup might come in and say, hey, here, these are guardrails that are the next best things since sliced bread for your agentic workflow. But I'm net new and I'm a 10 % company. So how do you discover those things, experiment with those things, bring them in this workflow? So the DNS equivalent in the internet of agents and agency is the agent directory. and there is identity associated with it as well to ensure that there is trust behind those agents. So that's step one. And you leave all of these things sitting where they are, right? So you're not really bringing them together physically.

27:23They're endpoints sitting somewhere. And then you get into the next few phases, which is composability and deployment and evaluation. Yeah, and presumably there's a service associated with each of those as well. So one or more. That's right. So to your question earlier, now you've connected all of these things in a workflow for a sales funnel application. ServiceNow is speaking language X because even though they might all be natural language, English, for example, or French, doesn't matter. All of these agents may not mean the same things when they say, this is my outcome, or this is the accuracy with which I'm giving you this outcome.

28:11So the one thing that has happened when we move from deterministic computing to probabilistic computing is that instead of your APIs, quote, unquote, being deterministic in nature, your APIs are now all probabilistic in nature, both from the intent perspective, which is show me the best X. What is best? It is up to interpretation. And the response, the outcome also is, this is the best X with 90 % confidence. 90 % for Salesforce or Cisco or Microsoft could be very different. So how do we build a framework where all of these outcomes and inputs in probabilistic manner can have a common understanding.

29:03And so there is a communication layer that needs to be built to handle this. And so that's where the communication protocols come in, both from the semantic perspective, which is what does best mean, what does 90 % mean, but also from the syntactic perspective, which is where the world is going crazy right now, which is A2A and MCP and ACP. That's the syntactic layer, which is, does the noun come before the verb or the verb come before the noun? That's what we're dealing with right now. But nobody is talking about the semantic layer, which is, what does best mean? What does 90 % mean? So that's the communication layer.

29:43And then think about the actual transport. That changes dramatically as well. And this is where Cisco steps in and is really awesome at it because we are a networking company, which is all of these agents are now communicating with each other with massive state exchanges. They're all multimodal in nature. It's video, it's audio, and you expect real-time behavior because they, guess what? Agents are supposed to be like humans, right? Operating on machine speed and scale. So how can you have audio, video, text, images at machine speed and scale being done efficiently at real time. That is an unsolved problem that we're trying to solve through agency and internal regions as well.

30:33So are you suggesting with that, and now I've got like three questions behind this, but are you suggesting with that that you are envisioning or exploring a post-HGTP protocol layer or transport for these kinds of agentic applications. I think most of us think, okay, video, images, audio, we just need bigger pipes. But bigger pipes is a relatively solved problem. You just pay Cisco or whoever more money, get better cards, blah, blah, blah. But it sounds like you're talking about more like protocol, transport level things. You're reading my mind here, Sam. You're right. I mean, typically what you would think is bigger pipes is better outcomes.